Jkkkbgsnzt/Qwen2.5-Coder-7B
Qwen2.5-Coder-7B is a 7.61 billion parameter causal language model from the Qwen2.5-Coder series, developed by Qwen. Optimized for code generation, code reasoning, and code fixing, it builds upon the Qwen2.5 architecture with 5.5 trillion training tokens, including extensive source code. This model supports a long context length of up to 131,072 tokens, making it suitable for complex coding tasks and real-world applications like Code Agents.
Loading preview...
Qwen2.5-Coder-7B Overview
Qwen2.5-Coder-7B is a 7.61 billion parameter causal language model, part of the Qwen2.5-Coder family developed by Qwen. This series, formerly known as CodeQwen, focuses on enhancing coding capabilities across various model sizes. The 7B model is built on the strong Qwen2.5 foundation and has been trained on an extensive 5.5 trillion tokens, including source code and text-code grounding data.
Key Capabilities & Improvements
- Enhanced Code Performance: Significant improvements in code generation, code reasoning, and code fixing compared to its predecessor, CodeQwen1.5.
- Long-Context Support: Features a full context length of 131,072 tokens, with
config.jsoninitially set for 32,768 tokens. It utilizes YaRN for effective long-text processing, allowing for scaling up to the full context. - Foundation for Code Agents: Designed to provide a comprehensive foundation for real-world applications such as Code Agents, while maintaining strong performance in mathematics and general competencies.
- Architecture: Employs transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias.
Usage Recommendations
This base model is not recommended for direct conversational use. Instead, it is intended for further post-training, such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), or continued pretraining. It is also suitable for fill-in-the-middle tasks. For deployment with long contexts, vLLM is recommended, with specific rope_scaling configurations for YaRN.