XinnanZhang/Qwen3-4B-openthoughts3-math-400k-sft
XinnanZhang/Qwen3-4B-openthoughts3-math-400k-sft is a 4 billion parameter Qwen3-based language model, fine-tuned by XinnanZhang on a mathematical reasoning dataset. This specific checkpoint is intended for reproducibility of distillation experiments and is noted to perform below its base model due to training data issues. It is primarily designed for research into mathematical reasoning and on-policy distillation, rather than general application.
Loading preview...
Model Overview
This model, Qwen3-4B-openthoughts3-math-400k-sft, is a 4 billion parameter variant of the Qwen3 architecture, fine-tuned by XinnanZhang. It was trained using the openthoughts3-math-50k8 dataset, comprising 400,000 examples, specifically as an initialization for on-policy distillation experiments.
Key Characteristics & Limitations
- Purpose: Primarily for research into distillation experiments, not as an improved general-purpose model.
- Performance Note: This checkpoint scores below its own base model. This is attributed to issues in the training data where 70.6% of SFT targets prematurely hit a generation cap, leading the model to emit an end-of-sequence token mid-reasoning.
- Training Details: Utilizes
openthinker3hyperparameters via LLaMA-Factory, including a learning rate of 8e-5, global batch size of 512, and FlashAttention-2. - Evaluation: Benchmarked on mathematical reasoning datasets (aime24, aime25, aime26, amc23) using mean@8 and pass@8 metrics. The average pass@8 score across these datasets is 67.40%.
Important Packaging Note
- The model's
config.jsonincludesrope_thetaboth flat and nested to ensure compatibility across differenttransformerslibrary versions. This prevents silent fallback to incorrectrope_thetavalues that can severely degrade generation coherence without error.