Overview

Qwen2.5-1.5B is a 1.54 billion parameter base causal language model from the Qwen2.5 series, developed by Qwen. It builds upon the Qwen2 architecture, incorporating improvements in knowledge, coding, and mathematics through specialized expert models. The model supports a substantial context length of 32,768 tokens and is designed for pretraining, not direct conversational use.

Key Capabilities

• Enhanced Knowledge & Reasoning: Significantly improved capabilities in coding and mathematics.
• Instruction Following: Better instruction adherence and resilience to diverse system prompts.
• Long Text Generation: Improved ability to generate texts over 8K tokens.
• Structured Data & Output: Enhanced understanding of structured data (e.g., tables) and generation of structured outputs, particularly JSON.
• Multilingual Support: Supports over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, and Arabic.

Architecture & Training

This model utilizes a transformer architecture featuring RoPE, SwiGLU, RMSNorm, Attention QKV bias, and tied word embeddings. It has 28 layers and 12 attention heads (GQA) for Q and 2 for KV. The model is a base language model, meaning it is intended for further post-training steps like SFT, RLHF, or continued pretraining, rather than direct conversational applications.