arcee-ai/Gemma-Openchat-SauerkrautLM
arcee-ai/Gemma-Openchat-SauerkrautLM is an 8.5 billion parameter language model, merged from Openchat-3.5-0106-Gemma and SauerkrautLM-Gemma-7b using a slerp merge method. This model combines the strengths of its base components, offering a versatile foundation for various generative AI tasks. It is designed to leverage the capabilities of both Gemma-based models, providing a balanced performance profile.
Loading preview...
Model Overview
arcee-ai/Gemma-Openchat-SauerkrautLM is an 8.5 billion parameter language model created by merging two distinct Gemma-based models: openchat/openchat-3.5-0106-gemma and VAGOsolutions/SauerkrautLM-Gemma-7b. This merge was performed using the mergekit tool, specifically employing a slerp (spherical linear interpolation) merge method.
Key Characteristics
- Architecture: Based on the Gemma family, combining two instruction-tuned variants.
- Parameter Count: 8.5 billion parameters, offering a balance between performance and computational efficiency.
- Context Length: Supports an 8192-token context window.
- Merge Method: Utilizes
slerpfor combining model weights, with specifictparameters applied to self-attention and MLP layers to fine-tune the merge outcome. - Dtype: Configured for
bfloat16precision.
Intended Use Cases
This merged model is suitable for applications requiring a robust generative AI foundation, benefiting from the combined capabilities of its constituent models. It can be applied to a range of natural language processing tasks where the strengths of both Openchat's instruction following and SauerkrautLM's general language understanding are advantageous.