Wjwiiwi/Qwen2.5-Coder-0.5B
Wjwiiwi/Qwen2.5-Coder-0.5B is a 0.49 billion parameter causal language model from the Qwen2.5-Coder series, developed by Qwen. This model is specifically pretrained for code-related tasks, building upon the Qwen2.5 architecture with RoPE, SwiGLU, and RMSNorm. It features a substantial 32,768 token context length and is designed for applications requiring strong code generation, reasoning, and fixing capabilities, serving as a foundation for code agents.
Loading preview...
Qwen2.5-Coder-0.5B Overview
Qwen2.5-Coder-0.5B is part of the Qwen2.5-Coder series, a collection of code-specific large language models developed by Qwen. This particular model has 0.49 billion parameters (0.36 billion non-embedding parameters) and is built on a transformer architecture incorporating RoPE, SwiGLU, and RMSNorm. It supports a substantial context length of 32,768 tokens.
Key Capabilities and Features
- Code-Specific Pretraining: Significantly improved for code generation, code reasoning, and code fixing compared to its predecessor, CodeQwen1.5.
- Extensive Training Data: Trained on 5.5 trillion tokens, including source code, text-code grounding, and synthetic data.
- Foundation for Code Agents: Designed to serve as a robust base for real-world applications like Code Agents, while also maintaining general and mathematical competencies.
- Architecture: Utilizes transformers with RoPE, SwiGLU, RMSNorm, Attention QKV bias, and tied word embeddings.
Recommended Use Cases
This base model is primarily intended for further development rather than direct conversational use. It is well-suited for:
- Post-training applications: Ideal for fine-tuning (SFT), reinforcement learning from human feedback (RLHF), or continued pretraining.
- Fill-in-the-middle tasks: Effective for code completion or similar tasks where context needs to be filled.
For detailed evaluation results and performance benchmarks, users are encouraged to consult the official blog and GitHub repository.