FuseChat-Llama-3.1-8B-Instruct: Implicit Model Fusion

FuseChat-Llama-3.1-8B-Instruct is an 8 billion parameter model developed by FuseAI, leveraging an innovative implicit model fusion (IMF) technique. This approach enhances the performance of smaller target LLMs by transferring capabilities from multiple robust source LLMs, including Gemma-2-27B-It, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct.

Key Capabilities & Training

The model's development involved a two-stage training pipeline:

• Supervised Fine-Tuning (SFT): Utilized best responses from source models to mitigate distribution discrepancies and enhance base capabilities.
• Direct Preference Optimization (DPO): Optimized using preference pairs (best and worst responses) from source models, with a length-normalized DPO for Llama-3.1 series models.

The training dataset was meticulously constructed from diverse open-source community datasets, focusing on instruction following, general conversation, mathematics, coding, and Chinese-language capabilities. Responses were sampled from the four powerful source LLMs and annotated using an external reward model (ArmoRM) for preference pair creation.

Performance Highlights

FuseChat-Llama-3.1-8B-Instruct demonstrated substantial performance gains across 14 benchmarks, achieving an average improvement of 6.8 points. Notably, it showed significant improvements of 37.1 points on AlpacaEval-2 and 30.1 points on Arena-Hard for instruction-following tasks. It also outperformed AllenAI's Llama-3.1-Tulu-3-8B on most benchmarks, highlighting the effectiveness of the IMF approach in creating a highly capable compact model.