Weyaxi/openchat-3.5-1210-Seraph-Slerp

TEXT GENERATIONConcurrency Cost:1Model Size:7BQuant:FP8Ctx Length:4kPublished:Dec 27, 2023License:apache-2.0Architecture:Transformer0.0K Open Weights Cold

Weyaxi/openchat-3.5-1210-Seraph-Slerp is a 7 billion parameter language model created by Weyaxi, merged using the slerp method from openchat/openchat-3.5-1210 and Weyaxi/Seraph-7B, based on Mistral-7B-v0.1. This model leverages a specific slerp merging strategy to combine the strengths of its base models, making it suitable for general conversational AI tasks. Its 4096-token context length supports moderate-length interactions.

Loading preview...

Model Overview

Weyaxi/openchat-3.5-1210-Seraph-Slerp is a 7 billion parameter language model developed by Weyaxi. It is a product of model merging, specifically utilizing the slerp (spherical linear interpolation) method via mergekit to combine two distinct base models: openchat/openchat-3.5-1210 and Weyaxi/Seraph-7B. The foundational architecture for this merged model is mistralai/Mistral-7B-v0.1.

Key Characteristics

  • Merged Architecture: Combines openchat-3.5-1210 and Seraph-7B using slerp interpolation.
  • Parameter-Specific Merging: The slerp method applies different interpolation ratios (t values) to self-attention and MLP layers, allowing for fine-grained control over how the characteristics of the source models are blended.
  • Tokenizer: Uses a union tokenizer, ensuring compatibility with the vocabularies of its constituent models.
  • Precision: Trained with bfloat16 dtype for efficient computation.

Potential Use Cases

  • General Conversational AI: Suitable for chatbots and interactive applications, inheriting capabilities from its openchat base.
  • Research in Model Merging: Provides an example of advanced slerp merging techniques for combining LLMs.

Finetunes
2 models