XinnanZhang/Qwen3-8B-openthoughts3-math-400k-sft
XinnanZhang/Qwen3-8B-openthoughts3-math-400k-sft is an 8 billion parameter Qwen3 model, fine-tuned by XinnanZhang on the openthoughts3-math-50k8 dataset for mathematical reasoning. This specific checkpoint is published for reproducibility of distillation experiments and is noted to perform below its base model due to training data issues. It is not intended as an improved model but rather as a starting point for further research into on-policy distillation. The model has a context length of 32768 tokens.
Loading preview...
Model Overview
XinnanZhang/Qwen3-8B-openthoughts3-math-400k-sft is an 8 billion parameter Qwen3 model, fine-tuned by XinnanZhang using the openthinker3 hyperparameters via LLaMA-Factory. It was trained on the XinnanZhang/openthoughts3-math-50k8 dataset, comprising 400k examples, primarily as an initialization for on-policy distillation experiments.
Key Characteristics & Limitations
This checkpoint is explicitly noted to perform below its untuned Qwen/Qwen3-8B base model, scoring 14.1 percentage points worse on the same evaluation. This performance degradation is attributed to issues within the training data, where only 29.4% of the SFT targets properly conclude with a </think> tag and \boxed{} answer, leading the model to emit an end-of-sequence token mid-reasoning. The model's evaluation on mathematical datasets (aime24, aime25, aime26, amc23) shows an average mean@8 of 52.94% and pass@8 of 70.04%.
Usage Note
This model is primarily intended for reproducibility of distillation experiments and should not be considered an improved or production-ready model for general use. Developers should be aware of a packaging note regarding rope_theta in transformers versions, which is addressed in this model's config.json to ensure correct loading across different versions.