ou474747/Qwen2.5-1.5B-Instruct
Qwen2.5-1.5B-Instruct is a 1.54 billion parameter instruction-tuned causal language model developed by Qwen. This model, part of the Qwen2.5 series, features a 32,768-token context length and is significantly improved in coding, mathematics, instruction following, and long text generation. It also offers robust multilingual support for over 29 languages and enhanced structured data understanding, making it suitable for diverse conversational AI applications.
Loading preview...
Overview
Qwen2.5-1.5B-Instruct is an instruction-tuned causal language model from the Qwen2.5 series, featuring 1.54 billion parameters and a 32,768-token context window. It builds upon the Qwen2 architecture with improvements in several key areas, including enhanced knowledge, coding, and mathematical capabilities, achieved through specialized expert models.
Key Capabilities
- Improved Instruction Following: Demonstrates significant advancements in adhering to instructions and generating structured outputs, particularly JSON.
- Long Text Generation: Enhanced ability to generate texts exceeding 8,000 tokens.
- Structured Data Understanding: Better at interpreting and processing structured data like tables.
- Multilingual Support: Supports over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, and Arabic.
- System Prompt Resilience: More robust to varied system prompts, improving role-play and chatbot condition-setting.
Architecture Details
This model utilizes a transformer architecture with RoPE, SwiGLU, RMSNorm, Attention QKV bias, and tied word embeddings. It consists of 28 layers and 12 attention heads for Q with 2 for KV (GQA). The model supports a full context length of 32,768 tokens and can generate up to 8,192 tokens.