MSL7/INEX12-7b

TEXT GENERATIONConcurrency Cost:1Model Size:7BQuant:FP8Ctx Length:4kPublished:Mar 3, 2024License:apache-2.0Architecture:Transformer0.0K Open Weights Cold

INEX12-7b is a 7 billion parameter language model created by MSL7, formed by merging liminerity/merge2 and yam-peleg/Experiment26-7B using mergekit. This model leverages a slerp merge method across its constituent models, offering a unique blend of their capabilities. It is designed for general language tasks, providing a balanced performance profile derived from its merged architecture.

Loading preview...

INEX12-7b Model Overview

INEX12-7b is a 7 billion parameter language model developed by MSL7. This model is a product of an advanced merging technique, combining two distinct base models: liminerity/merge2 and yam-peleg/Experiment26-7B. The merge was executed using mergekit, specifically employing the slerp (spherical linear interpolation) method.

Key Characteristics

  • Merged Architecture: Built from two established models, aiming to synthesize their strengths.
  • Parameter Count: Features 7 billion parameters, suitable for a range of language generation and understanding tasks.
  • Merge Method: Utilizes slerp for layer-wise interpolation, with specific t values applied differently to self-attention and MLP layers, indicating a fine-tuned blending strategy.
  • Data Type: Configured to use bfloat16 for efficient computation.

Intended Use Cases

INEX12-7b is a versatile model, well-suited for general-purpose applications where a balanced performance from a merged architecture is beneficial. Its design suggests potential for:

  • Text generation and completion.
  • Conversational AI and chatbots.
  • Content creation and summarization.

Finetunes
2 models