Qwen/Qwen2.5-0.5B
Qwen2.5-0.5B is a 0.49 billion parameter causal language model developed by Qwen, featuring a 32,768-token context length. This base model, part of the Qwen2.5 series, incorporates transformer architecture with RoPE, SwiGLU, and RMSNorm. It offers significant improvements in knowledge, coding, mathematics, and multilingual support across 29 languages, serving as a foundation for further fine-tuning.
Loading preview...
Qwen2.5-0.5B Model Summary
Qwen2.5-0.5B is a pre-trained, causal language model developed by Qwen, featuring 0.49 billion parameters and a substantial 32,768-token context length. It is built upon a transformer architecture incorporating RoPE, SwiGLU, RMSNorm, Attention QKV bias, and tied word embeddings. This model is part of the Qwen2.5 series, which enhances capabilities over its predecessor, Qwen2.
Key Capabilities & Improvements
- Enhanced Knowledge & Specialized Domains: Significantly improved knowledge base, particularly in coding and mathematics, leveraging specialized expert models.
- Instruction Following & Text Generation: Offers better instruction following, improved generation of long texts (over 8K tokens), and enhanced understanding of structured data like tables.
- Structured Output Generation: Excels at generating structured outputs, especially JSON, and demonstrates greater resilience to diverse system prompts for robust chatbot implementation.
- Multilingual Support: Provides comprehensive support for over 29 languages, including major global languages like Chinese, English, French, Spanish, and Japanese.
Intended Use
This 0.5B model is a base language model and is not recommended for direct conversational use. It is designed as a foundation for further post-training applications such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), or continued pretraining to adapt it for specific tasks and conversational agents.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.