InnerILLM-7B-slerp Overview
InnerILLM-7B-slerp is a 7 billion parameter language model developed by InnerI, created through a spherical linear interpolation (slerp) merge of two base models: OpenPipe/mistral-ft-optimized-1218 and mlabonne/NeuralHermes-2.5-Mistral-7B. This merging technique aims to combine the strengths of its constituent models.
Key Capabilities & Performance
The model demonstrates competitive performance on the Open LLM Leaderboard, achieving an average score of 71.09. Specific benchmark results include:
- AI2 Reasoning Challenge (25-Shot): 67.58
- HellaSwag (10-Shot): 86.19
- MMLU (5-Shot): 64.15
- TruthfulQA (0-shot): 59.84
- Winogrande (5-shot): 80.11
- GSM8k (5-shot): 68.69
These scores indicate proficiency in common sense reasoning, language understanding, multi-task language understanding, truthfulness, and mathematical problem-solving. The model was evaluated with an average loss of 0.8070214592665433 using a custom testing script.
Merge Configuration
The slerp merge method was applied to layers 0-32 of both source models. The t parameter for the slerp merge was configured differently for self_attn and mlp components, with a general value of 0.5 for other parameters, suggesting a balanced blend of the parent models' characteristics.
Good for
- General-purpose text generation and understanding tasks.
- Applications requiring a capable 7B parameter model with a 4096 token context window.
- Developers looking for a merged model with balanced performance across various benchmarks.