OmniBeagleMBX-v3-7B Overview

OmniBeagleMBX-v3-7B is a 7 billion parameter language model developed by paulml. It is constructed through a slerp merge of two distinct base models: mlabonne/OmniBeagle-7B and flemmingmiguel/MBX-7B-v3. This merging technique, facilitated by LazyMergekit, combines the strengths of its components to create a new model with a 4096 token context length.

Key Characteristics

• Merged Architecture: Combines OmniBeagle-7B and MBX-7B-v3 using a spherical linear interpolation (slerp) method.
• Configurable Merge Parameters: The merge configuration specifies distinct weighting for self-attention (self_attn) and multi-layer perceptron (mlp) layers, allowing for fine-grained control over the merged model's characteristics.
• 7 Billion Parameters: Offers a substantial capacity for various language understanding and generation tasks.

Intended Use Cases

This model is suitable for developers and researchers looking for a merged 7B model that balances the capabilities of its base components. It can be used for:

• General text generation and completion.
• Experimentation with merged model architectures and their performance.
• Applications requiring a 7B parameter model with a 4096 token context window.