crazyman177607/Qwen3-0.6B-Base
Qwen3-0.6B-Base is a 0.6 billion parameter causal language model from the Qwen series, developed by the Qwen Team. It is pre-trained on 36 trillion tokens across 119 languages, featuring an expanded, high-quality corpus including coding, STEM, and reasoning data. This base model incorporates architectural refinements like qk layernorm and a three-stage pre-training process to enhance general knowledge, reasoning skills, and long-context comprehension up to 32,768 tokens. It is designed as a foundational model for various language understanding and generation tasks.
Loading preview...
Qwen3-0.6B-Base Overview
Qwen3-0.6B-Base is a foundational causal language model, part of the latest Qwen3 series developed by the Qwen Team. This 0.6 billion parameter model (0.44B non-embedding parameters) is pre-trained and features a substantial context length of 32,768 tokens. It builds upon previous Qwen iterations with significant advancements in its training methodology and data.
Key Enhancements and Features
- Expanded Pre-training Corpus: Trained on an extensive 36 trillion tokens across 119 languages, tripling the language coverage of its predecessor, Qwen2.5. The dataset includes a rich mix of high-quality data focusing on coding, STEM, reasoning, and multilingual content.
- Architectural Refinements: Incorporates advanced training techniques and architectural improvements, such as
qk layernorm, to enhance stability and overall performance across different model scales. - Three-Stage Pre-training: Utilizes a structured pre-training approach:
- Stage 1: Focuses on broad language modeling and general knowledge acquisition.
- Stage 2: Improves specialized reasoning skills, including STEM, coding, and logical reasoning.
- Stage 3: Enhances long-context comprehension by extending training sequence lengths.
- Scaling Law Guided Tuning: Hyperparameters are systematically tuned based on comprehensive scaling law studies across the three pre-training stages, optimizing training dynamics and final performance.
Model Specifications
- Type: Causal Language Model
- Parameters: 0.6 Billion (0.44B non-embedding)
- Layers: 28
- Attention Heads (GQA): 16 for Q, 8 for KV
- Context Length: 32,768 tokens
This model serves as a robust base for developers requiring a compact yet capable language model with strong multilingual and reasoning foundations, suitable for further fine-tuning or direct application in tasks benefiting from its extensive pre-training.