Qwen/Qwen1.5-0.5B
Qwen1.5-0.5B is a 0.6 billion parameter, decoder-only transformer language model developed by Qwen. As a beta version of Qwen2, it features stable 32K context length support and multilingual capabilities. This base model is designed for further fine-tuning, such as SFT or RLHF, rather than direct text generation.
Loading preview...
Qwen1.5-0.5B: A Foundation for Advanced LLM Development
Qwen1.5-0.5B is a 0.6 billion parameter, decoder-only transformer language model, serving as a beta release for the Qwen2 series. Developed by Qwen, this model is part of a family of eight different sizes, ranging from 0.5B to 72B parameters, and includes an MoE model. It builds upon the original Qwen architecture with significant improvements, particularly in multilingual support and chat model performance.
Key Capabilities & Features
- Architecture: Transformer-based, decoder-only model utilizing SwiGLU activation and attention QKV bias.
- Context Length: Offers stable support for a 32K token context window across all model sizes.
- Multilingual Support: Enhanced multilingual capabilities for both base and chat models.
- Ease of Use: Eliminates the need for
trust_remote_codefor simpler integration. - Tokenizer: Features an improved tokenizer designed to adapt to multiple natural languages and code.
Intended Use Cases
Qwen1.5-0.5B is primarily intended as a foundational model for further development and customization. It is not recommended for direct text generation out-of-the-box. Instead, developers should consider it for:
- Fine-tuning: Ideal for Supervised Fine-Tuning (SFT) to adapt to specific tasks or domains.
- Reinforcement Learning with Human Feedback (RLHF): Suitable for alignment and preference learning.
- Continued Pretraining: Can be used for further pretraining on specialized datasets to enhance domain-specific knowledge.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.