arcee-ai/gemma-7b-it-zaphyr-slerp
arcee-ai/gemma-7b-it-zaphyr-slerp is an 8.5 billion parameter language model created by arcee-ai, built by merging Google's Gemma-7b-it and HuggingFaceH4's Zephyr-7b-gemma-v0.1. This model leverages a SLERP merge method to combine the strengths of both base models, aiming for enhanced instruction-following capabilities. With an 8192 token context length, it is designed for general-purpose conversational AI and instruction-tuned applications.
Loading preview...
Model Overview
The arcee-ai/gemma-7b-it-zaphyr-slerp is an 8.5 billion parameter language model developed by arcee-ai. It is a product of merging two distinct models: google/gemma-7b-it and HuggingFaceH4/zephyr-7b-gemma-v0.1. This merge was performed using the mergekit tool, specifically employing the SLERP (Spherical Linear Interpolation) method.
Merge Configuration
The model's unique characteristics stem from its specific merge configuration. The SLERP method was applied across all layers (0 to 28) of both source models. A key aspect of this merge is the differential weighting applied to attention (self_attn) and feed-forward (mlp) layers:
- Self-Attention Layers: Weights were varied across layers, with values like
[0, 0.5, 0.3, 0.7, 1]indicating a blend that shifts influence fromgemma-7b-ittowardszephyr-7b-gemma-v0.1in certain attention components. - MLP Layers: Weights were also varied, using
[1, 0.5, 0.7, 0.3, 0], suggesting a blend that emphasizesgemma-7b-itin some MLP components andzephyr-7b-gemma-v0.1in others. - General Layers: A default
0.5weight was applied to other parameters, indicating an equal blend.
The base model for this merge was google/gemma-7b-it, and the model uses bfloat16 for its data type. This intricate merging strategy aims to combine the robust base capabilities of Gemma with the instruction-following prowess of Zephyr, resulting in a model optimized for interactive and instruction-based tasks.