Model Overview

vicgalle/CarbonBeagle-11B is a 10.7 billion parameter language model developed by vicgalle, created through an experimental linear merge process. This model combines vicgalle/NeuralBeagle-11B and jeonsworld/CarbonVillain-en-10.7B-v4, aiming to leverage the strengths of different architectures and sizes. The merging process involved upscaling mlabonne/NeuralBeagle14-7B to vicgalle/franken-Beagle-11B, DPO-tuning it to vicgalle/NeuralBeagle-11B, and then linearly merging it with jeonsworld/CarbonVillain-en-10.7B-v4.

Performance and Capabilities

At its release on January 21, 2024, CarbonBeagle-11B demonstrated competitive performance, ranking as a top model in its 10.7B-11B size class and even against 13B models on the Open LLM Leaderboard. It achieved an average score of 74.64 across various benchmarks, including:

• AI2 Reasoning Challenge (25-Shot): 71.84
• HellaSwag (10-Shot): 88.93
• MMLU (5-Shot): 66.62
• TruthfulQA (0-shot): 69.43
• Winogrande (5-shot): 84.06
• GSM8k (5-shot): 66.94

Further evaluations on the Open LLM Leaderboard show an average of 22.36 for more advanced reasoning tasks, with specific scores like IFEval (0-Shot) at 54.15 and BBH (3-Shot) at 33.06.

Usage Considerations

This model is licensed under CC-BY-NC-SA 4.0, meaning it cannot be used for commercial purposes. Redistribution must also adhere to the same license. Users should consider these licensing terms before deployment.