Model Overview

arcee-ai/Gemma-Openchat-SauerkrautLM is an 8.5 billion parameter language model created by merging two distinct Gemma-based models: openchat/openchat-3.5-0106-gemma and VAGOsolutions/SauerkrautLM-Gemma-7b. This merge was performed using the mergekit tool, specifically employing a slerp (spherical linear interpolation) merge method.

Key Characteristics

• Architecture: Based on the Gemma family, combining two instruction-tuned variants.
• Parameter Count: 8.5 billion parameters, offering a balance between performance and computational efficiency.
• Context Length: Supports an 8192-token context window.
• Merge Method: Utilizes slerp for combining model weights, with specific t parameters applied to self-attention and MLP layers to fine-tune the merge outcome.
• Dtype: Configured for bfloat16 precision.

Intended Use Cases

This merged model is suitable for applications requiring a robust generative AI foundation, benefiting from the combined capabilities of its constituent models. It can be applied to a range of natural language processing tasks where the strengths of both Openchat's instruction following and SauerkrautLM's general language understanding are advantageous.