Gille/StrangeMerges_46-7B-dare_ties
Gille/StrangeMerges_46-7B-dare_ties is a 7 billion parameter language model created by Gille, formed by merging Gille/StrangeMerges_45-7B-dare_ties, kettleguts/zephyr-7b-beta_sparse05, and chihoonlee10/T3Q-Mistral-Orca-Math-DPO using the dare_ties method. This model demonstrates an average performance of 69.96 on the Open LLM Leaderboard, with notable scores in HellaSwag (86.40) and Winogrande (79.48). It is designed for general language tasks, leveraging its merged architecture to balance various capabilities.
Loading preview...
Overview
Gille/StrangeMerges_46-7B-dare_ties is a 7 billion parameter language model developed by Gille. It is a product of a merge operation using LazyMergekit and the dare_ties method. The model integrates components from three distinct base models:
- Gille/StrangeMerges_45-7B-dare_ties
- kettleguts/zephyr-7b-beta_sparse05
- chihoonlee10/T3Q-Mistral-Orca-Math-DPO
This merging strategy aims to combine the strengths of its constituent models, with a base model of liminerity/M7-7b and bfloat16 dtype for its operations.
Performance Highlights
The model's performance has been evaluated on the Open LLM Leaderboard, achieving an average score of 69.96. Key benchmark results include:
- HellaSwag (10-Shot): 86.40
- Winogrande (5-shot): 79.48
- AI2 Reasoning Challenge (25-Shot): 67.24
- TruthfulQA (0-shot): 65.17
- MMLU (5-Shot): 62.17
- GSM8k (5-shot): 59.29
These scores indicate a balanced capability across various reasoning, common sense, and language understanding tasks, with particular strength in HellaSwag and Winogrande.
Usage
Developers can easily integrate this model using the Hugging Face transformers library. The provided Python snippet demonstrates how to load the model and tokenizer, apply a chat template, and generate text, making it accessible for various natural language generation tasks.