Beast-Soul: A Merged 7B Language Model

Beast-Soul is a 7 billion parameter language model developed by Vasanth, constructed through a merge of two distinct models: udkai/Turdus and decruz07/kellemar-DPO-Orca-Distilled-7B-SLERP. This merge was performed using the LazyMergekit tool, employing a slerp (spherical linear interpolation) method.

Key Characteristics

• Architecture: Based on the OpenPipe/mistral-ft-optimized-1218 model as its foundation.
• Merge Method: Utilizes slerp for combining the weights of the constituent models, with specific parameter adjustments for self-attention and MLP layers.
• Context Length: Supports a context window of 4096 tokens.
• Precision: Configured to use bfloat16 data type for efficient computation.

Usage and Application

Beast-Soul is suitable for a variety of natural language processing tasks, leveraging the combined capabilities of its merged components. Developers can integrate it into their projects using the Hugging Face transformers library, as demonstrated in the provided Python usage example. This model offers a balanced approach to performance and resource utilization for general text generation and understanding.