MaziyarPanahi/Tess-XS-v1-3-yarn-128K-Mistral-7B-Instruct-v0.1
MaziyarPanahi/Tess-XS-v1-3-yarn-128K-Mistral-7B-Instruct-v0.1 is a 7 billion parameter instruction-tuned language model, created by MaziyarPanahi, that merges Mistral-7B-Instruct-v0.1 with Tess-XS-v1-3-yarn-128K. This model is designed to leverage the strengths of both base models, particularly enhancing long context understanding up to 128K tokens through the YARN extension. It is suitable for general instruction-following tasks requiring extended context processing.
Loading preview...
Model Overview
MaziyarPanahi/Tess-XS-v1-3-yarn-128K-Mistral-7B-Instruct-v0.1 is a 7 billion parameter instruction-tuned language model. It is a merge of two distinct models:
- Mistral-7B-Instruct-v0.1: A foundational instruction-following model from Mistral AI.
- Tess-XS-v1-3-yarn-128K: A model known for its extended context capabilities, specifically incorporating the YARN (Yet Another RoPE Normalization) technique to handle context lengths up to 128,000 tokens.
Key Capabilities
- Extended Context Window: Inherits the 128K token context length from
Tess-XS-v1-3-yarn-128K, making it highly effective for tasks requiring processing very long documents or conversations. - Instruction Following: Benefits from the instruction-tuning of
Mistral-7B-Instruct-v0.1, providing robust performance on a wide range of prompts. - Merged Architecture: Utilizes a
slerpmerge method with specific parameter weighting for self-attention and MLP layers, aiming to combine the strengths of both base models.
Usage
This model is well-suited for applications that demand both strong instruction adherence and the ability to process and generate content based on extensive input contexts. Developers can integrate it using the Hugging Face transformers library, as demonstrated in the provided Python example.