louisbrulenaudet/Pearl-7B-0211-ties
Pearl-7B-0211-ties by louisbrulenaudet is a 7.24 billion parameter language model created using the TIES-Merging method, combining several 7B models including Pearl-7B-slerp, WizardMath-7B-V1.1, WestLake-7B-v2-laser, and NeuralTrix-7B-dpo. This merge technique aims to create a consolidated multitask model by addressing parameter redundancy and resolving sign conflicts across its constituent models. It demonstrates strong performance across various benchmarks, particularly excelling in HellaSwag, TruthfulQA, and Winogrande, making it suitable for diverse general-purpose language tasks.
Loading preview...
Pearl-7B-0211-ties: A Merged 7B Model
Pearl-7B-0211-ties is a 7.24 billion parameter model developed by louisbrulenaudet, created through the TIES-Merging method. This technique efficiently combines multiple task-specific models into a consolidated multitask model by addressing parameter redundancy and resolving conflicts in parameter signs. It integrates components from several 7B models, including Pearl-7B-slerp, WizardMath-7B-V1.1, WestLake-7B-v2-laser, and NeuralTrix-7B-dpo.
Key Capabilities & Performance
Evaluated on the HuggingFace Open LLM Leaderboard, Pearl-7B-0211-ties demonstrates competitive performance, often surpassing other models in its size class and even some larger models in specific benchmarks. Notably, it achieves:
- 75.11 Average score on the Open LLM Leaderboard.
- 88.86 on HellaSwag, indicating strong common-sense reasoning.
- 71.46 on TruthfulQA, suggesting good factual accuracy.
- 84.37 on Winogrande, showcasing robust pronoun resolution abilities.
TIES-Merging Explained
The TIES-Merging process involves three steps:
- Trim: Reduces redundancy by retaining only the most significant parameters from fine-tuned models.
- Elect Sign: Resolves conflicts in parameter signs across models to establish a unified direction of change.
- Disjoint Merge: Averages parameter values based on the unified sign vector.
This method allows for the creation of a powerful multitask model from specialized components, offering a balanced performance across various linguistic tasks.