Vasanth/Beast-Soul

TEXT GENERATIONConcurrency Cost:1Model Size:7BQuant:FP8Ctx Length:4kPublished:Jan 21, 2024License:apache-2.0Architecture:Transformer Open Weights Cold

Beast-Soul is a 7 billion parameter language model developed by Vasanth, created by merging udkai/Turdus and decruz07/kellemar-DPO-Orca-Distilled-7B-SLERP using LazyMergekit. This model leverages a slerp merge method based on OpenPipe/mistral-ft-optimized-1218, offering a 4096-token context length. It is designed for general language generation tasks, combining the strengths of its constituent models.

Loading preview...

Beast-Soul: A Merged 7B Language Model

Beast-Soul is a 7 billion parameter language model developed by Vasanth, constructed through a merge of two distinct models: udkai/Turdus and decruz07/kellemar-DPO-Orca-Distilled-7B-SLERP. This merge was performed using the LazyMergekit tool, employing a slerp (spherical linear interpolation) method.

Key Characteristics

  • Architecture: Based on the OpenPipe/mistral-ft-optimized-1218 model as its foundation.
  • Merge Method: Utilizes slerp for combining the weights of the constituent models, with specific parameter adjustments for self-attention and MLP layers.
  • Context Length: Supports a context window of 4096 tokens.
  • Precision: Configured to use bfloat16 data type for efficient computation.

Usage and Application

Beast-Soul is suitable for a variety of natural language processing tasks, leveraging the combined capabilities of its merged components. Developers can integrate it into their projects using the Hugging Face transformers library, as demonstrated in the provided Python usage example. This model offers a balanced approach to performance and resource utilization for general text generation and understanding.

Finetunes
2 models