louisbrulenaudet/Pearl-34B-ties
Pearl-34B-ties by louisbrulenaudet is a 34.39 billion parameter model created through a TIES-merging process of jondurbin/bagel-dpo-34b-v0.2 and abacusai/MetaMath-Bagel-DPO-34B. This model excels in general language understanding and reasoning, achieving a 75.48 average score on the Open LLM Leaderboard, making it a strong performer among models around 30B parameters. It is particularly optimized for tasks requiring robust evaluation metrics, demonstrating high scores in MMLU and Winogrande.
Loading preview...
Pearl-34B-ties: A Merged 34B Model
Pearl-34B-ties, developed by louisbrulenaudet, is a 34.39 billion parameter language model that stands out for its performance on the Open LLM Leaderboard. As of March 22, 2024, it was recognized as the "Best base merges and moerges model of around 30B" with an average score of 75.48.
Key Capabilities & Merging Process
This model is a result of a TIES-merging process, combining:
TIES-Merging is a method designed to efficiently merge multiple task-specific models into a consolidated multitask model. It addresses redundancy and conflicts in model parameters by:
- Trim: Reducing redundancy by retaining only the most significant parameters.
- Elect Sign: Resolving sign conflicts by creating a unified sign vector.
- Disjoint Merge: Averaging parameter values aligned with the unified sign vector.
Performance Highlights
Pearl-34B-ties demonstrates strong performance across various benchmarks, including:
- MMLU: 76.63
- Winogrande: 82.64
- HellaSwag: 84.83
Its balanced performance across these metrics indicates a robust capability for general language understanding and reasoning tasks. The model's configuration also specifies a bfloat16 dtype for efficient processing.