grimjim/kunoichi-lemon-royale-v2-32K-7B

Model Overview

kunoichi-lemon-royale-v2-32K-7B is a 7 billion parameter language model developed by grimjim, created through a merge of models derived from Mistral v0.1 and Mistral v0.2 architectures. This merge specifically grafts a Mistral v0.1-derived model (with a 4K sliding window context, up to 8K practical context) onto a Mistral v0.2-derived model (with a 32K context length). While the rope_theta was adjusted, the model appears to function effectively with a context length up to 16K tokens.

Key Capabilities

• Extended Context Handling: Designed to manage longer contexts, appearing to work well up to 16K tokens.
• Formatting Adherence: Demonstrates strong ability to follow formatting instructions, particularly with ChatML prompts, even though its base model claims Alpaca prompt compatibility.
• Merge Method: Created using the task arithmetic merge method, leveraging grimjim/Mistral-7B-Instruct-demi-merge-v0.2-7B as the base model.

Good For

• Applications requiring precise output formatting.
• Use cases benefiting from an extended context window, up to 16K tokens.
• Developers experimenting with merged Mistral-based models for specific instruction-following tasks.