Overview

EmbeddedLLM/Mistral-7B-Merge-14-v0.5 is a 7 billion parameter experimental language model derived from the Mistral-7B-v0.1 base. It represents a sophisticated merging effort, initially combining 14 distinct models, and subsequently undergoing a second merge using the DARE TIES method. This iterative merging process integrates contributions from models such as Weyaxi/OpenHermes-2.5-neural-chat-v3-3-openchat-3.5-1210-Slerp, openchat/openchat-3.5-0106, and mlabonne/NeuralMarcoro14-7B.

Key Capabilities

• Advanced Merging Technique: Utilizes the DARE TIES method for combining multiple models, aiming to leverage the strengths of diverse architectures.
• Broad Benchmark Performance: Achieves an average score of 71.96 on the Open LLM Leaderboard, with specific scores including 68.69 on ARC, 86.45 on HellaSwag, 65.65 on MMLU, and 71.19 on GSM8K.
• Flexible Chat Templates: Supports both ChatML and Llama-2 chat templates for interaction.

Good For

• Research into Model Merging: Ideal for developers and researchers interested in the practical application and performance outcomes of advanced model merging techniques like DARE TIES.
• General Purpose Language Tasks: Its balanced performance across various benchmarks suggests suitability for a wide range of common language understanding and generation tasks.
• Experimentation with Merged Models: Provides a robust base for further fine-tuning or integration into applications requiring a capable 7B parameter model.