GarrulusMarcoro-7B-v0.1 Overview

GarrulusMarcoro-7B-v0.1 is a 7 billion parameter language model developed by dfurman, created through a strategic merge of two distinct models: udkai/Garrulus and mlabonne/NeuralMarcoro14-7B. This merge was performed using the LazyMergekit tool, combining the strengths of its constituent models.

Key Capabilities & Performance

This model exhibits strong performance across a range of benchmarks, as evaluated on the Open LLM Leaderboard. It achieves an impressive average score of 74.20, indicating robust general reasoning and language understanding. Specific benchmark results include:

• AI2 Reasoning Challenge (25-Shot): 72.35
• HellaSwag (10-Shot): 88.00
• MMLU (5-Shot): 64.65
• TruthfulQA (0-shot): 67.05
• Winogrande (5-shot): 87.21
• GSM8k (5-shot): 65.96

Technical Details

The merge process utilized a slerp method, with specific t parameters applied to different layers (self_attn and mlp) to fine-tune the combination. The model operates with a 4096-token context length and is configured for bfloat16 dtype, making it efficient for deployment and inference. This model is well-suited for applications requiring a balanced performance across various general-purpose language tasks.