Model Overview

EmbeddedLLM/Mistral-7B-Merge-14-v0.1 is an experimental 7 billion parameter language model built upon the Mistral architecture. Developed by EmbeddedLLM, this model explores advanced merging techniques to combine the strengths of multiple specialized Mistral-7B variants.

Key Merging Process

This model's unique characteristic lies in its two-stage merging methodology:

• DARE TIES Merge: Initially, 14 distinct Mistral-7B models were merged using the DARE TIES method. This process aimed to consolidate diverse capabilities from models like mistralai/Mistral-7B-Instruct-v0.2, ehartford/dolphin-2.2.1-mistral-7b, SciPhi/SciPhi-Mistral-7B-32k, and several others focusing on instruction, coding, and mathematical reasoning.
• Gradient SLERP Merge: The resulting merged model was then further combined with janai-hq/trinity-v1 using Gradient SLERP. This second merge step was intended to enhance overall performance and create a more versatile base model.

Purpose and Use

This model is presented as a strong base model, indicating its suitability for a wide range of general-purpose language tasks. While it performs well out-of-the-box, the developers note that it is designed to benefit significantly from subsequent instruction fine-tuning, allowing users to tailor its behavior for specific applications and achieve optimal performance.