Konstanta-Alpha-V2-7B Overview

Konstanta-Alpha-V2-7B is a 7 billion parameter language model developed by Inv, constructed through a sophisticated merging process of multiple pre-trained models. This model integrates components from SanjiWatsuki/Kunoichi-DPO-v2-7B, maywell/PiVoT-0.1-Evil-a, mlabonne/ArchBeagle-7B, and LakoMoor/Silicon-Alice-7B.

Merge Methodology

The model was created using a two-stage merging approach. Initially, DARE TIES was applied to merge Kunoichi with PiVoT Evil, and separately, ArchBeagle with Silicon Alice. These two resulting models were then combined using the gradient SLERP merge method. The configuration indicates a base model of mistralai/Mistral-7B-v0.1 for the initial DARE TIES steps, with ChatML identified as the most effective chat format.

Performance Highlights

Evaluations on the Open LLM Leaderboard show Konstanta-Alpha-V2-7B achieving an average score of 72.35. Specific benchmark results include:

• AI2 Reasoning Challenge (25-Shot): 69.62
• HellaSwag (10-Shot): 87.14
• MMLU (5-Shot): 65.11
• TruthfulQA (0-shot): 61.08
• Winogrande (5-shot): 81.22
• GSM8k (5-shot): 69.90

These scores indicate a balanced performance across various reasoning, common sense, and language understanding tasks, making it suitable for general-purpose applications requiring robust language capabilities.