reaperdoesntknow/Qwen3-0.6B-Distilled-30B-A3B

Model Overview

This model, developed by Convergent Intelligence LLC, is a 0.6 billion parameter Qwen3-based causal language model. It is a highly compressed (50x) distillation from a 30 billion parameter Qwen3-A3B "Thinking" teacher model, specifically designed to excel in STEM reasoning tasks by learning the underlying deliberation process rather than just the final answers.

Key Differentiators

• Thinking Teacher Distillation: Unlike standard distillation from an "Instruct" teacher, this model learns from a "Thinking" variant of Qwen3-30B-A3B. This teacher generates extended internal reasoning traces, providing the student with a richer landscape of alternative derivation strategies at a distillation temperature of T=2.0. The student learns the deliberation process, not just the outcome.
• Proof-Weighted Loss: During training, tokens within the derivation region (between Proof: and Final Answer:) receive an amplified loss (2.5x, decaying to 1.5x). This mechanism forces the 0.6B model to allocate its limited parameters primarily to reasoning capability, penalizing errors in reasoning steps more heavily than formatting errors.

Capabilities & Training

The model was trained on 6,122 STEM chain-of-thought samples across 12 domains, including Physics, Linear Algebra, and Differential Equations. It utilizes a combined loss function of 55% proof-weighted cross-entropy and 45% knowledge distillation KL divergence. The training context length is 1024 tokens.

Good for

• Lightweight STEM reasoning on edge/mobile devices
• Educational tutoring and proof drafting
• Component in multi-model pipelines requiring a small, fast reasoner
• IoT and embedded inference applications