Pearl-7B-0211-ties: A Merged 7B Model

Pearl-7B-0211-ties is a 7.24 billion parameter model developed by louisbrulenaudet, created through the TIES-Merging method. This technique efficiently combines multiple task-specific models into a consolidated multitask model by addressing parameter redundancy and resolving conflicts in parameter signs. It integrates components from several 7B models, including Pearl-7B-slerp, WizardMath-7B-V1.1, WestLake-7B-v2-laser, and NeuralTrix-7B-dpo.

Key Capabilities & Performance

Evaluated on the HuggingFace Open LLM Leaderboard, Pearl-7B-0211-ties demonstrates competitive performance, often surpassing other models in its size class and even some larger models in specific benchmarks. Notably, it achieves:

• 75.11 Average score on the Open LLM Leaderboard.
• 88.86 on HellaSwag, indicating strong common-sense reasoning.
• 71.46 on TruthfulQA, suggesting good factual accuracy.
• 84.37 on Winogrande, showcasing robust pronoun resolution abilities.

TIES-Merging Explained

The TIES-Merging process involves three steps:

• Trim: Reduces redundancy by retaining only the most significant parameters from fine-tuned models.
• Elect Sign: Resolves conflicts in parameter signs across models to establish a unified direction of change.
• Disjoint Merge: Averages parameter values based on the unified sign vector.

This method allows for the creation of a powerful multitask model from specialized components, offering a balanced performance across various linguistic tasks.