Nyxene-v2-11B: A Merged Language Model
Nyxene-v2-11B is a 10.7 billion parameter model developed by beberik, representing an advanced iteration of the Nyxene series. This model is constructed through a sophisticated merging process using mergekit, combining four distinct 7B base models to achieve enhanced capabilities.
Key Capabilities & Architecture
The model's unique architecture is derived from a multi-stage merging strategy:
- Initial Merges: It first combines
fblgit/una-cybertron-7b-v2 with chargoddard/loyal-piano-m7-cdpo to form "loyal-piano-cybertron-11B", and berkeley-nest/Starling-LM-7B-alpha with openaccess-ai-collective/DPOpenHermes-7B to form "Starling-DPOHermes-11B". - Final Merge: These two intermediate 11B models are then merged using a slerp method, with specific
t parameter weightings applied to different tensor types (e.g., lm_head, embed_tokens, self_attn, mlp, layernorm) to fine-tune the final model's characteristics. - Prompt Template: The recommended prompt template for optimal performance is a standard instruction-following format, utilizing
<|system|>, <|user|>, and <|assistant|> tags.
Performance & Benchmarks
Evaluated on the Open LLM Leaderboard, Nyxene-v2-11B achieves an average score of 67.84. Notable scores include:
- AI2 Reasoning Challenge (25-Shot): 67.41
- HellaSwag (10-Shot): 84.54
- MMLU (5-Shot): 65.26
- TruthfulQA (0-shot): 55.62
- Winogrande (5-shot): 79.56
- GSM8k (5-shot): 54.66
These results indicate its proficiency across various reasoning, common sense, and language understanding tasks, making it suitable for general-purpose applications requiring robust instruction following and knowledge recall.