Model Overview
nlpguy/AlloyIngot is a 7 billion parameter language model resulting from a strategic merge of two pre-trained models: eren23/dpo-binarized-NeutrixOmnibe-7B and Gille/StrangeMerges_21-7B-slerp. This merge was performed using the SLERP (Spherical Linear Interpolation) method, a technique often employed to combine the strengths of different models while maintaining performance.
Performance Highlights
Evaluated on the Open LLM Leaderboard, AlloyIngot demonstrates strong general capabilities with an average score of 76.20. Key benchmark results include:
- AI2 Reasoning Challenge (25-Shot): 73.98
- HellaSwag (10-Shot): 89.05
- MMLU (5-Shot): 64.83
- TruthfulQA (0-shot): 75.12
- Winogrande (5-shot): 85.08
- GSM8k (5-shot): 69.14
These scores indicate proficiency in common sense reasoning, factual recall, and mathematical problem-solving.
Key Differentiator
AlloyIngot's primary distinction lies in its origin as a SLERP merge, combining specific strengths from its constituent models. This method allows for fine-grained control over how different layers and components of the base models contribute to the final merged model, as evidenced by the detailed t parameter configuration for self-attention and MLP layers.
Recommended Use Cases
- General-purpose text generation and understanding: Its balanced performance across various benchmarks makes it suitable for a wide range of NLP tasks.
- Reasoning and question answering: Strong scores in ARC, MMLU, and TruthfulQA suggest good capabilities in these areas.
- Exploration of merged model performance: Developers interested in the practical application and results of model merging techniques may find this model particularly useful.