prompt-agnostic-language-models/sweep_Qwen-1B_ppcl_lr1e-05_js1.0
The prompt-agnostic-language-models/sweep_Qwen-1B_ppcl_lr1e-05_js1.0 model is a 0.8 billion parameter causal language model from the Qwen3 series, developed by Qwen. This model is pre-trained on an expanded corpus of 36 trillion tokens across 119 languages, focusing on broad language modeling and general knowledge acquisition. It incorporates architectural refinements like qk layernorm and a three-stage pre-training approach to enhance reasoning skills and long-context comprehension up to 32,768 tokens. It is designed for general-purpose language tasks, benefiting from improved stability and performance across various scales.
Loading preview...
Model Overview
prompt-agnostic-language-models/sweep_Qwen-1B_ppcl_lr1e-05_js1.0 is a 0.8 billion parameter causal language model, part of the Qwen3 series developed by Qwen. This model is a pre-trained variant, building upon significant advancements in training data, model architecture, and optimization techniques compared to its predecessors.
Key Features and Improvements
- Expanded High-Quality Pre-training Corpus: Trained on an extensive 36 trillion tokens across 119 languages, tripling the language coverage of Qwen2.5. The dataset includes a rich mix of high-quality data, such as coding, STEM, reasoning, and multilingual content.
- Architectural Refinements: Incorporates advanced training techniques and architectural improvements, including
qk layernorm, which contributes to enhanced stability and overall performance. - Three-Stage Pre-training: Utilizes a distinct three-stage pre-training process:
- Stage 1: Focuses on broad language modeling and general knowledge acquisition.
- Stage 2: Improves reasoning skills, including STEM, coding, and logical reasoning.
- Stage 3: Enhances long-context comprehension by extending training sequence lengths up to 32,768 tokens.
- Scaling Law Guided Hyperparameter Tuning: Critical hyperparameters, such as learning rate scheduler and batch size, are systematically tuned based on comprehensive scaling law studies across the pre-training pipeline.
Model Specifications
- Type: Causal Language Model
- Training Stage: Pretraining
- Number of Parameters: 0.8 Billion
- Context Length: 32,768 tokens
Usage Considerations
This model is designed for general-purpose language understanding and generation tasks, benefiting from its extensive multilingual training and enhanced reasoning capabilities. For optimal performance, it is recommended to use the latest version of Hugging Face transformers library.