Phigment6-slerp is a 3 billion parameter language model developed by liminerity, built upon the Phi-2 architecture. It was created using the Divergent Knowledge Enhancement through Retrograde Merging Strategies (DKERS) methodology, which involves spherically interpolating and merging multiple Phi-2 based models. This approach aims to combine the strengths of its constituent models, resulting in enhanced performance across various linguistic tasks. It is particularly optimized for general language understanding and generation, demonstrating strong benchmark results for its size.
Loading preview...
Phigment6-slerp: A Merged 3B Parameter LLM
Phigment6-slerp is a 3 billion parameter large language model (LLM) developed by liminerity, based on the Phi-2 architecture. It distinguishes itself through its unique Divergent Knowledge Enhancement through Retrograde Merging Strategies (DKERS) methodology. This process involves the strategic merging of several pre-trained Phi-2 based models, specifically amu/dpo-phi2, g-ronimo/phi-2-OpenHermes-2.5, vince62s/phi-2-psy, and mobiuslabsgmbh/aanaphi2-v0.1, using spherical linear interpolation (SLERP).
Key Capabilities & Innovations
- Model Fusion: Utilizes a novel DKERS methodology to combine the strengths of multiple Phi-2 variants, aiming for superior performance without training from scratch.
- Enhanced Performance: Demonstrates significant improvements in performance metrics such as perplexity, F1-score, and ROUGE scores compared to its predecessors.
- Robustness: Exhibits enhanced generalization capabilities and increased resistance to adversarial attacks, indicating a more robust understanding of language nuances.
- Compact Size: Achieves strong performance within a 3 billion parameter footprint, making it efficient for various applications.
Benchmark Performance
Phigment6-slerp has been evaluated on the Open LLM Leaderboard, achieving an average score of 63.58. Notable scores include:
- AI2 Reasoning Challenge (25-Shot): 62.63
- HellaSwag (10-Shot): 77.25
- MMLU (5-Shot): 58.65
- Winogrande (5-Shot): 73.88
Good For
- Applications requiring a capable yet efficient 3B parameter model.
- General language understanding and generation tasks.
- Scenarios where combining knowledge from diverse models is beneficial.
- Developers seeking a robust Phi-2 based model with improved generalization.