Qwen3-14B-Base Overview

Qwen3-14B-Base is a 14.8 billion parameter pre-trained causal language model from the Qwen series, building upon advancements in training data, architecture, and optimization. It features a substantial expansion in its pre-training corpus, now encompassing 36 trillion tokens across 119 languages, significantly tripling the language coverage of its predecessor, Qwen2.5. The dataset includes a rich mix of high-quality data, such as coding, STEM, reasoning, and multilingual content.

Key Improvements & Features

• Expanded Pre-training Corpus: Trained on 36 trillion tokens across 119 languages, with a focus on high-quality data for coding, STEM, and reasoning.
• Architectural Refinements: Incorporates training techniques like global-batch load balancing loss for MoE models and qk layernorm for all models, enhancing stability and performance.
• Three-stage Pre-training: A structured approach that first builds general language modeling, then refines reasoning skills (STEM, coding, logical reasoning), and finally extends long-context comprehension up to 32,768 tokens.
• Scaling Law Guided Tuning: Hyperparameters are systematically tuned using scaling law studies across the pre-training pipeline for optimal training dynamics and performance.

Model Specifications

• Parameters: 14.8 billion (13.2 billion non-embedding)
• Context Length: 32,768 tokens
• Layers: 40
• Attention Heads (GQA): 40 for Q, 8 for KV

For detailed evaluation results and further information, refer to the official Qwen3 blog and GitHub repository.