Mergerix-7b-v0.4 Overview

Mergerix-7b-v0.4 is a 7 billion parameter language model developed by MiniMoog. This model is a product of merging two distinct base models: liminerity/M7-7b and automerger/YamshadowExperiment28-7B. The merge was performed using a slerp (spherical linear interpolation) method via LazyMergekit, aiming to combine and enhance the capabilities of its source models.

Key Characteristics

• Merged Architecture: Built from liminerity/M7-7b and automerger/YamshadowExperiment28-7B to leverage their respective strengths.
• Merge Method: Utilizes slerp for combining model weights, with specific parameter configurations applied to self-attention and MLP layers.
• Parameter Count: A 7 billion parameter model, offering a balance between performance and computational efficiency.
• Context Length: Supports an 8192-token context window, allowing for processing and generating longer sequences of text.

Intended Use Cases

This model is suitable for a variety of general-purpose language generation tasks, benefiting from the combined knowledge and capabilities of its merged components. Developers can integrate it into applications requiring robust text generation, summarization, or conversational AI, particularly where the specific strengths of the merged base models are advantageous.