Model Overview

This model, grimjim/Mistral-7B-Instruct-demi-merge-v0.2-7B, is a 7 billion parameter language model created by grimjim through a strategic merge of two Mistral v0.2 models: alpindale/Mistral-7B-v0.2-hf and mistralai/Mistral-7B-Instruct-v0.2. The merge was performed using the SLERP method via mergekit.

Key Characteristics

• Architecture: Based on the Mistral v0.2 family.
• Parameter Count: 7 billion parameters.
• Extended Context Length: Supports a 32K token context window without a sliding window, significantly exceeding the 8K context of the original Mistral-7B-0.1 release. This feature has been confirmed through light testing.
• Purpose of Merge: The primary goal of this merge was to create a balanced model that combines the robust training of the base Mistral v0.2 with the instruction-following capabilities of the Instruct version. This design makes it a suitable foundation for subsequent fine-tuning or further merging operations, aiming to "thaw out" the Instruct model's strengths while maintaining flexibility.

Use Cases

This model is particularly well-suited for developers and researchers looking for:

• A base model for further fine-tuning on specific tasks or datasets.
• Applications requiring long context understanding and generation, thanks to its 32K token window.
• Scenarios where a balance between a raw base model's flexibility and an instruction-tuned model's direct applicability is desired.