Hugging Face Model Overview
vicgalle/CarbonBeagle-11B is a 10.7 billion parameter language model developed by vicgalle, created through an experimental linear merge process. This model combines vicgalle/NeuralBeagle-11B and jeonsworld/CarbonVillain-en-10.7B-v4, aiming to leverage the strengths of different architectures and sizes. The merging process involved upscaling mlabonne/NeuralBeagle14-7B to vicgalle/franken-Beagle-11B, DPO-tuning it to vicgalle/NeuralBeagle-11B, and then linearly merging it with jeonsworld/CarbonVillain-en-10.7B-v4.
Performance and Capabilities
At its release on January 21, 2024, CarbonBeagle-11B demonstrated competitive performance, ranking as a top model in its 10.7B-11B size class and even against 13B models on the Open LLM Leaderboard. It achieved an average score of 74.64 across various benchmarks, including:
- AI2 Reasoning Challenge (25-Shot): 71.84
- HellaSwag (10-Shot): 88.93
- MMLU (5-Shot): 66.62
- TruthfulQA (0-shot): 69.43
- Winogrande (5-shot): 84.06
- GSM8k (5-shot): 66.94
Further evaluations on the Open LLM Leaderboard show an average of 22.36 for more advanced reasoning tasks, with specific scores like IFEval (0-Shot) at 54.15 and BBH (3-Shot) at 33.06.
Usage Considerations
This model is licensed under CC-BY-NC-SA 4.0, meaning it cannot be used for commercial purposes. Redistribution must also adhere to the same license. Users should consider these licensing terms before deployment.