Model Overview

EmbeddedLLM/Mistral-7B-Merge-14-v0.4 is a 7 billion parameter experimental language model developed by EmbeddedLLM. It is the result of a sophisticated, multi-stage merging process designed to combine the strengths of multiple base models. Initially, 14 different models were merged using the DARE TIES method to create an intermediate version. This intermediate model was then further merged with Weyaxi/OpenHermes-2.5-neural-chat-v3-3-openchat-3.5-1210-Slerp using Gradient SLERP.

Key Capabilities & Performance

This merged model demonstrates strong general performance, achieving an average score of 71.19 on the Open LLM Leaderboard. Specific benchmark results include:

• ARC: 66.81
• HellaSwag: 86.15
• MMLU: 65.10
• TruthfulQA: 58.25
• Winogrande: 80.03
• GSM8K: 70.81

Usage & Characteristics

The model supports both ChatML and Llama-2 chat templates, offering flexibility for integration into various applications. While it performs well across a range of tasks, the developers note that it "may require further instruction fine-tuning" for optimal performance in specific instruction-following scenarios. The merging configuration details, including the use of slerp for specific tensor filters, are publicly available, providing transparency into its construction.