reaperdoesntknow/Qwen3-0.6B-Distilled-30B-A3B
The reaperdoesntknow/Qwen3-0.6B-Distilled-30B-A3B is a 0.6 billion parameter Qwen3-based causal language model, distilled from a 30B-parameter 'Thinking' teacher model. It is specifically optimized for structured STEM derivations and reasoning tasks, achieving a 50x parameter compression. This model excels at producing detailed reasoning traces for scientific, technical, engineering, and mathematics problems, making it suitable for lightweight STEM reasoning applications.
Loading preview...
Overview
This model, reaperdoesntknow/Qwen3-0.6B-Distilled-30B-A3B, is a 0.6 billion parameter Qwen3-based causal language model. It represents a significant 50x parameter compression, distilled from a 30 billion parameter Qwen3-A3B-Thinking teacher model. The core innovation lies in its distillation process, which leverages a "Thinking" teacher variant that generates extended internal reasoning, transferring deeper deliberation structures to the smaller student model. This approach, combined with a proof-weighted loss function, ensures the model prioritizes reasoning steps over mere answer formatting.
Key Differentiators
- Thinking Teacher Distillation: Unlike standard distillation from Instruct models, this model learns from a teacher that produces rich, high-entropy reasoning traces, enabling the student to grasp deliberation strategies.
- Proof-Weighted Loss: During training, tokens within the derivation region (
Proof:toFinal Answer:) receive amplified loss (2.5x decaying to 1.5x), forcing the 0.6B model to allocate its limited parameters to reasoning capability. - STEM CoT Training: Trained on 6,122 STEM Chain-of-Thought samples across 12 domains, including Physics, Linear Algebra, and Differential Equations.
Good For
- Lightweight STEM reasoning on edge/mobile devices.
- Educational tutoring and proof drafting.
- Component in multi-model pipelines requiring a small, fast reasoner.
- IoT and embedded inference where resource constraints are critical.