reaperdoesntknow/Qwen3-0.6B-Distilled-30B-A3B

Hugging Face
TEXT GENERATIONConcurrent Unit Cost:1Model Size:0.8BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Mar 22, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Warm

reaperdoesntknow/Qwen3-0.6B-Distilled-30B-A3B is a 0.6 billion parameter Qwen3-based causal language model, distilled from a 30B-parameter 'Thinking' teacher model. It is specifically optimized for structured STEM derivations and reasoning tasks, achieving a 50x parameter compression. This model excels at producing detailed reasoning traces for scientific, technical, engineering, and mathematics problems, making it suitable for lightweight STEM reasoning on edge devices.

Loading preview...

Model Overview

This model, reaperdoesntknow/Qwen3-0.6B-Distilled-30B-A3B, is a 0.6 billion parameter Qwen3-based causal language model. It represents a 50x parameter compression, distilled from a 30 billion parameter Qwen3-A3B-Thinking teacher model. The distillation process uniquely focuses on transferring deep deliberation structures, rather than just final answers, by using a "Thinking" teacher variant that generates extended internal reasoning traces.

Key Differentiators

  • Thinking Teacher Distillation: Unlike standard distillation from Instruct models, this student model learns from a teacher that produces richer, higher-entropy reasoning paths, enabling it to learn deliberation strategies.
  • Proof-Weighted Loss: During training, tokens within the derivation region (Proof: to Final Answer:) receive amplified loss (2.5x decaying to 1.5x), ensuring parameters are primarily allocated to reasoning capability rather than just output formatting.
  • STEM Focus: Trained on 6,122 STEM chain-of-thought samples across 12 domains, it is specifically designed to produce structured derivations for scientific and mathematical problems.

Intended Uses

  • Lightweight STEM reasoning on edge/mobile devices.
  • Educational tutoring and proof drafting.
  • Component in multi-model pipelines requiring a small, fast reasoner.
  • IoT and embedded inference applications.

Limitations

Due to its 0.6B parameter size, the model may struggle with multi-step proofs exceeding ~8 reasoning steps, complex multi-variable problems, or domains underrepresented in its training data. Generated intermediate steps should always be verified.

Finetunes
3 models