Abhay2310/qwen2.5-0.5b-opd-gsm8k
Abhay2310/qwen2.5-0.5b-opd-gsm8k is a 0.5 billion parameter Qwen2.5-Instruct model, distilled from Qwen2.5-1.5B-Instruct using an on-policy distillation (OPD) method on the GSM8K dataset. This experimental model, with a 32768 token context length, was developed to investigate the practical constraints and failure modes of OPD, particularly its impact on mathematical reasoning and sampled behavior. It serves as a case study for understanding on-policy training dynamics rather than demonstrating benchmark improvements.
Loading preview...
Model Overview
This model, Abhay2310/qwen2.5-0.5b-opd-gsm8k, is a 0.5 billion parameter Qwen2.5-Instruct student model, distilled from a 1.5 billion parameter Qwen2.5-Instruct teacher model using an on-policy distillation (OPD) approach. The primary goal of this project was to explore the mechanics, practical constraints, and potential failure modes of OPD, specifically on mathematical reasoning tasks using the GSM8K dataset, rather than to achieve state-of-the-art benchmark results.
Key Characteristics & Findings
- On-Policy Distillation (OPD): Implements a from-scratch OPD method where the student generates its own trajectory, and the teacher evaluates that same trajectory, closing the train-inference distribution gap.
- Experimental Focus: The project emphasizes understanding the objective and its practical challenges, including training instability and various failure modes encountered during implementation.
- Performance Insights: While the student's pass@1 on GSM8K showed a slight increase (9.0% to 11.0%), the more significant finding was a sharp decrease in "any-of-4 success" (18.0% to 4.0%). This suggests that pass@1 alone can mask substantial changes in sampled behavior, indicating a narrowing of the student's useful sampling distribution.
- Training Instability: The model exhibited training collapse, where KL loss decreased, but accuracy fell, and outputs degraded. This highlights challenges in on-policy training loops, where a healthy-looking loss doesn't always imply stable or improved policy behavior.
- Resource Constraints: The entire distillation process was designed and executed to fit within the memory constraints of a single free-tier Colab T4 GPU, utilizing techniques like FP32 for the student, FP16 for the teacher, 8-bit Adam, and gradient checkpointing.
Intended Use
This model is primarily intended for small-scale mathematical reasoning experiments and as a case study for developers interested in the practical aspects and challenges of on-policy distillation. It serves as an educational tool to understand OPD's dynamics and limitations, particularly how evaluation metrics can sometimes obscure underlying behavioral changes in models.