vicgalle/CarbonBeagle-11B is a 10.7 billion parameter language model created by vicgalle, resulting from a linear merge of vicgalle/NeuralBeagle-11B and jeonsworld/CarbonVillain-en-10.7B-v4. This model, with a 4096-token context length, is an experimental merge of different architectures and sizes, demonstrating strong performance in its size class on the Open LLM Leaderboard. It is optimized for general language understanding and reasoning tasks, achieving an average score of 74.64 on the Open LLM Leaderboard at its creation.
Loading preview...
Model Overview
vicgalle/CarbonBeagle-11B is a 10.7 billion parameter language model developed by vicgalle, created through an experimental linear merge process. This model combines vicgalle/NeuralBeagle-11B and jeonsworld/CarbonVillain-en-10.7B-v4, aiming to leverage the strengths of different architectures and sizes. The merging process involved upscaling mlabonne/NeuralBeagle14-7B to vicgalle/franken-Beagle-11B, DPO-tuning it to vicgalle/NeuralBeagle-11B, and then linearly merging it with jeonsworld/CarbonVillain-en-10.7B-v4.
Performance and Capabilities
At its release on January 21, 2024, CarbonBeagle-11B demonstrated competitive performance, ranking as a top model in its 10.7B-11B size class and even against 13B models on the Open LLM Leaderboard. It achieved an average score of 74.64 across various benchmarks, including:
- AI2 Reasoning Challenge (25-Shot): 71.84
- HellaSwag (10-Shot): 88.93
- MMLU (5-Shot): 66.62
- TruthfulQA (0-shot): 69.43
- Winogrande (5-shot): 84.06
- GSM8k (5-shot): 66.94
Further evaluations on the Open LLM Leaderboard show an average of 22.36 for more advanced reasoning tasks, with specific scores like IFEval (0-Shot) at 54.15 and BBH (3-Shot) at 33.06.
Usage Considerations
This model is licensed under CC-BY-NC-SA 4.0, meaning it cannot be used for commercial purposes. Redistribution must also adhere to the same license. Users should consider these licensing terms before deployment.