Model Overview

Gille/StrangeMerges_4-7B-slerp is a 7 billion parameter language model resulting from a strategic merge of two previous models: Gille/StrangeMerges_3-7B-slerp and Gille/StrangeMerges_2-7B-slerp. This merging process utilized the slerp (spherical linear interpolation) method, specifically configured with varying interpolation values across different layers, including self-attention and MLP blocks, to achieve a balanced integration of their respective strengths.

Key Capabilities & Performance

This model has been evaluated on the Open LLM Leaderboard, achieving a competitive average score of 72.63. Its performance highlights include:

• HellaSwag (10-Shot): 87.01
• Winogrande (5-Shot): 82.95
• AI2 Reasoning Challenge (25-Shot): 69.45
• GSM8k (5-Shot): 68.61
• MMLU (5-Shot): 65.33
• TruthfulQA (0-shot): 62.40

These scores indicate a solid capability across various reasoning, common sense, and language understanding tasks. The model supports a context length of 4096 tokens and is designed for general text generation applications.

Usage

Developers can easily integrate StrangeMerges_4-7B-slerp into their projects using the Hugging Face transformers library. The model is compatible with bfloat16 data types and can be run on GPU-accelerated environments with device_map="auto" for efficient resource utilization.