Model Overview

EmbeddedLLM/Mistral-7B-Merge-14-v0.2 is a 7 billion parameter experimental language model built upon the Mistral architecture. It was created by merging 14 distinct models using the DARE TIES method, and then further merged with janai-hq/trinity-v1 using Gradient SLERP. This process aimed to produce a robust base model suitable for subsequent instruction fine-tuning.

Key Characteristics & Performance

• Architecture: Mistral-7B base.
• Merging Technique: Utilizes DARE TIES for the initial 14-model merge, followed by Gradient SLERP with janai-hq/trinity-v1.
• Open LLM Leaderboard Score: Achieved an average score of 72.88.
  • ARC: 68.86
  • HellaSwag: 87.01
  • MMLU: 65.05
  • TruthfulQA: 64.19
  • Winogrande: 81.69
  • GSM8K: 70.51

Intended Use

This model is presented as a strong base model that performs well across various benchmarks. It is specifically designed to be a foundation that requires further instruction fine-tuning to optimize its performance for specific conversational or task-oriented applications. Its experimental merging approach makes it a unique candidate for developers exploring advanced model combination strategies.