paulml/OmniBeagleMBX-v3-7B

TEXT GENERATIONConcurrency Cost:1Model Size:7BQuant:FP8Ctx Length:4kPublished:Feb 4, 2024License:cc-by-nc-4.0Architecture:Transformer0.0K Open Weights Cold

OmniBeagleMBX-v3-7B is a 7 billion parameter language model created by paulml, resulting from a slerp merge of mlabonne/OmniBeagle-7B and flemmingmiguel/MBX-7B-v3. This model leverages the strengths of its constituent models, offering a balanced performance profile for general text generation tasks within a 4096 token context window. It is designed for developers seeking a merged model with specific parameter weighting for self-attention and MLP layers.

Loading preview...

OmniBeagleMBX-v3-7B Overview

OmniBeagleMBX-v3-7B is a 7 billion parameter language model developed by paulml. It is constructed through a slerp merge of two distinct base models: mlabonne/OmniBeagle-7B and flemmingmiguel/MBX-7B-v3. This merging technique, facilitated by LazyMergekit, combines the strengths of its components to create a new model with a 4096 token context length.

Key Characteristics

  • Merged Architecture: Combines OmniBeagle-7B and MBX-7B-v3 using a spherical linear interpolation (slerp) method.
  • Configurable Merge Parameters: The merge configuration specifies distinct weighting for self-attention (self_attn) and multi-layer perceptron (mlp) layers, allowing for fine-grained control over the merged model's characteristics.
  • 7 Billion Parameters: Offers a substantial capacity for various language understanding and generation tasks.

Intended Use Cases

This model is suitable for developers and researchers looking for a merged 7B model that balances the capabilities of its base components. It can be used for:

  • General text generation and completion.
  • Experimentation with merged model architectures and their performance.
  • Applications requiring a 7B parameter model with a 4096 token context window.