ArchiveStudio/Qwen3-1.7B-Base
ArchiveStudio/Qwen3-1.7B-Base is a 1.7 billion parameter causal language model from the Qwen3 series, developed by the Qwen Team. This base model is pre-trained on an expanded 36 trillion token corpus covering 119 languages, with a focus on high-quality data including coding, STEM, and reasoning. It features a 32,768 token context length and incorporates architectural refinements and a three-stage pre-training process to enhance general knowledge, reasoning skills, and long-context comprehension.
Loading preview...
Qwen3-1.7B-Base Overview
Qwen3-1.7B-Base is a 1.7 billion parameter causal language model, part of the Qwen3 series developed by the Qwen Team. This model is a pre-trained version, building upon significant advancements in training data, model architecture, and optimization techniques compared to its predecessor, Qwen2.5. It features a substantial context length of 32,768 tokens.
Key Improvements and Features
- Expanded Pre-training Corpus: Qwen3 is pre-trained on an extensive 36 trillion tokens across 119 languages, tripling the language coverage of Qwen2.5. This corpus includes a richer mix of high-quality data, specifically targeting coding, STEM, reasoning, books, multilingual content, and synthetic data.
- Architectural and Training Refinements: The model incorporates advanced training techniques and architectural improvements, such as global-batch load balancing loss for MoE models and qk layernorm for all models, which contribute to enhanced stability and overall performance.
- Three-Stage Pre-training: The training process is structured in three distinct stages:
- Stage 1: Focuses on broad language modeling and general knowledge acquisition.
- Stage 2: Aims to improve reasoning skills, including STEM, coding, and logical reasoning.
- Stage 3: Enhances long-context comprehension by extending training sequence lengths up to 32,768 tokens.
- Scaling Law Guided Hyperparameter Tuning: Critical hyperparameters were systematically tuned using scaling law studies across the three pre-training stages, optimizing training dynamics and performance for different model scales.
When to Use This Model
This base model is suitable for developers looking for a compact yet capable foundation model with strong multilingual support and enhanced reasoning abilities. Its large context window makes it particularly useful for tasks requiring extensive document understanding or generation. It serves as a robust starting point for further fine-tuning on specific downstream applications.