grimjim/kukulemon-32K-7B
grimjim/kukulemon-32K-7B is a 7 billion parameter language model, created by grimjim, that demonstrates a functional 32K context length. This proof-of-concept model is a SLERP merge of kukulemon-7B and Mistral 7B v0.2 models, specifically engineered to extend context capabilities. It natively supports Alpaca prompting and has been successfully tested with ChatML prompts, making it suitable for applications requiring extended conversational memory.
Loading preview...
kukulemon-32K-7B: Extended Context Language Model
This model, developed by grimjim, is a 7 billion parameter proof-of-concept demonstrating a functional 32K context length. It was created by merging existing Mistral 7B v0.2 models with grimjim/kukulemon-7B using the SLERP merge method via mergekit.
Key Capabilities
- Extended Context Window: Achieves a functional 32K context length, significantly beyond the typical 4096 tokens of its base models.
- Merge Method: Utilizes the SLERP (Spherical Linear Interpolation) merge method, which proved viable for integrating the extended context capabilities.
- Prompt Format Flexibility: Natively supports Alpaca prompt format and has been successfully tested with ChatML prompts.
- Recommended Inference Settings: For ChatML prompts, a medium temperature (around 1) and low
minP(e.g., 0.01) are suggested for optimal results.
Good For
- Long-form Conversations: Ideal for applications requiring the model to maintain context over extensive dialogues or documents.
- Experimental Context Extension: Useful for developers and researchers exploring the practical application of extended context windows in merged models.
- Alpaca and ChatML Workflows: Compatible with common prompting formats, facilitating integration into existing pipelines.
Note: The DARE-TIES merge method was considered but deemed too risky for this model due to potential damage.