SeanWang0027/opd_polaris_15K_qwen3-4b_from_qwen3-30b-a3b_topk16_bf16_epoch_1

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:4BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Aug 31, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

SeanWang0027/opd_polaris_15K_qwen3-4b_from_qwen3-30b-a3b_topk16_bf16_epoch_1 is a 4 billion parameter Qwen3-4B model fine-tuned using On-Policy Distillation (OPD) against a Qwen3-30B-A3B-Thinking-2507 teacher. This model was trained on 15,368 competition mathematics problems, focusing on measuring the overlap of thinking patterns between the student and a Mixture-of-Experts teacher. It serves primarily as a reproduction artifact of a specific training run and a measurement of OPD's condition (i) with a MoE teacher, rather than a model claimed to outperform the base Qwen3-4B.

Loading preview...

Overview

This model, opd_polaris_15K_qwen3-4b_from_qwen3-30b-a3b_topk16_bf16_epoch_1, is a Qwen3-4B student model trained using On-Policy Distillation (OPD) against a Qwen3-30B-A3B-Thinking-2507 teacher. The training involved a single epoch over 15,368 competition mathematics problems from the SeanWang0027/polaris_hard dataset. A key aspect of this run was the use of bfloat16 dtype, a deliberate deviation from fp32 due to hardware memory constraints, which means its performance is not directly comparable to fp32 OPD numbers.

Key Findings and Characteristics

  • On-Policy Distillation (OPD) with MoE Teacher: The model investigates OPD's condition (i) with a Mixture-of-Experts (MoE) teacher, where the teacher scores but does not generate. It measured the overlap of top-k logprobs between the student and teacher.
  • High Probability Mass Overlap: Throughout the 240 training steps, approximately 99.8% of the probability mass consistently sat on the shared token set between the student and teacher, indicating successful OPD condition (i) as reported in the paper.
  • Minimal Student Parameter Change: Only 1.66% of the student's parameters changed during the training, with a very low global relative L2 drift of 6.58e-4. This is attributed to a low learning rate of 1e-6, suggesting that most weights could not represent the update.
  • Reproduction Artifact: This model is presented as a reproduction artifact of the specific training run and a measurement of OPD's condition (i) with a MoE teacher, rather than a model claimed to be superior to the base Qwen/Qwen3-4B.
  • No Downstream Evaluation: No downstream evaluation has been performed on this checkpoint or its base model.

Usage

To use this model, you can load it with the transformers library, ensuring to specify dtype="bfloat16":

from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "SeanWang0027/opd_polaris_15K_qwen3-4b_from_qwen3-30b-a3b_topk16_bf16_epoch_1"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype="bfloat16", device_map="auto")

Finetunes
1 models