Model Overview

Weyaxi/openchat-3.5-1210-Seraph-Slerp is a 7 billion parameter language model developed by Weyaxi. It is a product of model merging, specifically utilizing the slerp (spherical linear interpolation) method via mergekit to combine two distinct base models: openchat/openchat-3.5-1210 and Weyaxi/Seraph-7B. The foundational architecture for this merged model is mistralai/Mistral-7B-v0.1.

Key Characteristics

• Merged Architecture: Combines openchat-3.5-1210 and Seraph-7B using slerp interpolation.
• Parameter-Specific Merging: The slerp method applies different interpolation ratios (t values) to self-attention and MLP layers, allowing for fine-grained control over how the characteristics of the source models are blended.
• Tokenizer: Uses a union tokenizer, ensuring compatibility with the vocabularies of its constituent models.
• Precision: Trained with bfloat16 dtype for efficient computation.

Potential Use Cases

• General Conversational AI: Suitable for chatbots and interactive applications, inheriting capabilities from its openchat base.
• Research in Model Merging: Provides an example of advanced slerp merging techniques for combining LLMs.