adrianxu9778/Qwen2.5-7B-Instruct
Qwen2.5-7B-Instruct is a 7.61 billion parameter instruction-tuned causal language model developed by Qwen, part of the Qwen2.5 series. This model features a transformer architecture with RoPE, SwiGLU, and RMSNorm, supporting a full context length of 131,072 tokens and generating up to 8,192 tokens. It offers significantly improved capabilities in coding, mathematics, instruction following, and generating long texts, with multilingual support for over 29 languages.
Loading preview...
Qwen2.5-7B-Instruct Overview
Qwen2.5-7B-Instruct is an instruction-tuned causal language model from the Qwen2.5 series, developed by Qwen. This 7.61 billion parameter model builds upon its predecessor, Qwen2, with substantial enhancements across several key areas. It utilizes a transformer architecture incorporating RoPE, SwiGLU, RMSNorm, and Attention QKV bias.
Key Capabilities & Improvements
- Enhanced Knowledge & Reasoning: Significantly improved capabilities in coding and mathematics due to specialized expert models.
- Instruction Following: Demonstrates significant improvements in adhering to instructions and generating structured outputs, particularly JSON.
- Long-Context Handling: Supports a full context length of up to 131,072 tokens and can generate texts up to 8,192 tokens.
- Multilingual Support: Offers robust support for over 29 languages, including Chinese, English, French, Spanish, German, and Japanese.
- System Prompt Resilience: More resilient to diverse system prompts, enhancing role-play and chatbot condition-setting.
Technical Specifications
- Parameters: 7.61B (6.53B non-embedding)
- Layers: 28
- Attention Heads (GQA): 28 for Q, 4 for KV
Long Text Processing
The model's config.json is set for 32,768 tokens, but it can handle longer inputs up to 131,072 tokens using YaRN for length extrapolation. Users can enable YaRN by adding rope_scaling configuration to config.json when processing extensive contexts is required.