unsloth/Qwen2.5-0.5B
unsloth/Qwen2.5-0.5B is a 0.49 billion parameter causal language model from the Qwen2.5 series, developed by Qwen. This base model features a 32,768 token context length and is designed for pretraining, offering significant improvements in knowledge, coding, mathematics, and multilingual support across 29 languages. It is optimized for further fine-tuning for specific applications rather than direct conversational use.
Loading preview...
Qwen2.5-0.5B Overview
This model is the 0.49 billion parameter base version of the Qwen2.5 series, developed by Qwen. It is a causal language model with a substantial 32,768 token context length. The Qwen2.5 series introduces significant enhancements over its predecessor, Qwen2, particularly in:
Key Capabilities & Improvements
- Expanded Knowledge: Incorporates significantly more knowledge, with greatly improved capabilities in coding and mathematics due to specialized expert models.
- Instruction Following: Enhanced instruction following, better generation of long texts (over 8K tokens), and improved understanding of structured data like tables.
- Structured Output: Excels at generating structured outputs, especially JSON, and is more resilient to diverse system prompts for role-play and chatbot conditioning.
- Long-Context Support: Supports contexts up to 128K tokens and can generate up to 8K tokens.
- Multilingual Support: Offers robust support for over 29 languages, including Chinese, English, French, Spanish, German, and more.
Technical Specifications
- Architecture: Transformer-based with RoPE, SwiGLU, RMSNorm, Attention QKV bias, and tied word embeddings.
- Parameters: 0.49 billion total parameters (0.36 billion non-embedding).
- Layers: 24 layers.
- Attention Heads: 14 for Q and 2 for KV in Grouped-query Attention (GQA).
Usage Recommendations
This is a base language model intended for pretraining. It is not recommended for direct conversational use. Developers should apply post-training techniques such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), or continued pretraining to adapt it for specific tasks and conversational agents. For more detailed information, refer to the Qwen2.5 blog and GitHub repository.