RockToken/qwen3_30b_a3b_to_4b_onpolicy_5k_src30k-35k_cont

Model Overview

This model, RockToken/qwen3_30b_a3b_to_4b_onpolicy_5k_src30k-35k_cont, is a 4 billion parameter Qwen3-4B student model. It has undergone continual on-policy knowledge distillation from a larger Qwen3-30B-A3B (MoE) teacher model. The training specifically utilized 5,000 math prompts from the openthoughts3_math_50k dataset, representing a further refinement slice (positions 30k–35k) after prior off-policy and on-policy distillation stages.

Key Distillation Stages

• Off-policy KD: Initial distillation on 20k teacher responses.
• On-policy KD (10k prompts): Further distillation on 10k prompts, producing the base for this model.
• On-policy KD (this run): Continual distillation on an additional 5k math prompts (positions 30k–35k).

Training Details

The model was trained using the KDFlow framework on 4 H100 GPUs for approximately 34 hours. Key hyperparameters for the on-policy stage included a learning rate of 2e-6, a KD ratio of 1.0 with reverse KL loss, and a temperature of 1.0 for KD. The training data consisted of 5,000 single-user-turn math prompts.

Intended Use and Limitations

• Intended Use: Primarily for research into distillation dynamics, particularly within the mathematical domain.
• Limitations:
  • Trained exclusively on math prompts; not suitable for general chat, safety, or non-math applications.
  • enable_thinking=False means the model does not emit <think> traces.
  • Continual training may lead to capability drift compared to the base checkpoint.