ichigoberry/MonarchPipe-7B-slerp
MonarchPipe-7B-slerp is a 7 billion parameter language model created by ichigoberry, formed by merging OpenPipe/mistral-ft-optimized-1227 and mlabonne/AlphaMonarch-7B using a slerp method. This model leverages the Mistral architecture and offers an 8192-token context length. It is designed to combine the strengths of its base models, showing competitive performance in general language understanding and reasoning tasks.
Loading preview...
MonarchPipe-7B-slerp Overview
MonarchPipe-7B-slerp is a 7 billion parameter language model developed by ichigoberry. It is a merged model, combining the capabilities of OpenPipe/mistral-ft-optimized-1227 and mlabonne/AlphaMonarch-7B using a slerp (spherical linear interpolation) merge method via LazyMergekit. This merging technique aims to blend the strengths of its constituent models, which are both based on the Mistral architecture.
Key Capabilities & Performance
The model demonstrates competitive performance across various benchmarks, as evaluated by the Nous benchmark suite. While its overall average score is slightly below AlphaMonarch-7B, it shows strong results in specific areas:
- AGIEval: Achieves 46.12, outperforming AlphaMonarch-7B in this specific metric.
- General Reasoning: Designed to handle a range of general language understanding and reasoning tasks, benefiting from the fine-tuning of its base models.
Configuration & Usage
The merge configuration utilizes a slerp method with specific t parameters applied to different layers (self_attn and mlp) to fine-tune the blending process. The model supports a context length of 8192 tokens and is intended for use with bfloat16 data types. Developers can easily integrate MonarchPipe-7B-slerp into their projects using the Hugging Face transformers library, with provided Python code examples for text generation.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.