MonarchPipe-7B-slerp Overview

MonarchPipe-7B-slerp is a 7 billion parameter language model developed by ichigoberry. It is a merged model, combining the capabilities of OpenPipe/mistral-ft-optimized-1227 and mlabonne/AlphaMonarch-7B using a slerp (spherical linear interpolation) merge method via LazyMergekit. This merging technique aims to blend the strengths of its constituent models, which are both based on the Mistral architecture.

Key Capabilities & Performance

The model demonstrates competitive performance across various benchmarks, as evaluated by the Nous benchmark suite. While its overall average score is slightly below AlphaMonarch-7B, it shows strong results in specific areas:

• AGIEval: Achieves 46.12, outperforming AlphaMonarch-7B in this specific metric.
• General Reasoning: Designed to handle a range of general language understanding and reasoning tasks, benefiting from the fine-tuning of its base models.

Configuration & Usage

The merge configuration utilizes a slerp method with specific t parameters applied to different layers (self_attn and mlp) to fine-tune the blending process. The model supports a context length of 8192 tokens and is intended for use with bfloat16 data types. Developers can easily integrate MonarchPipe-7B-slerp into their projects using the Hugging Face transformers library, with provided Python code examples for text generation.