reaperdoesntknow/Qwen3-0.6B-Distilled-30B-A3B-Thinking-SFT
Qwen3-0.6B-Distilled-30B-A3B-Thinking-SFT is a 0.6 billion parameter Qwen3-based causal language model developed by Convergent Intelligence LLC. It was created through a two-stage process: knowledge distillation from a 30B-parameter "Thinking" teacher model for structured reasoning, followed by supervised fine-tuning on legal instruction data. This model is optimized for ultra-lightweight reasoning on mobile/edge devices, excelling in legal and STEM instruction-following tasks with a 50x compression ratio.
Loading preview...
Model Overview
This model, developed by Convergent Intelligence LLC, is a 0.6 billion parameter Qwen3-based causal language model designed for efficient reasoning in resource-constrained environments. It achieves a 50x compression from its 30B teacher model, making it suitable for deployment on devices like mobile phones, with quantized versions under 500MB.
Key Capabilities and Training
The model's unique training pipeline involves two stages:
- Stage 1: Knowledge Distillation from a Qwen3-30B-A3B-Thinking teacher model. This stage focused on establishing a robust STEM reasoning backbone using 6,122 chain-of-thought samples across 12 STEM domains. The "Thinking" teacher, which generates extended internal reasoning traces, was crucial for transferring deeper reasoning structures to the small student model. A Proof-Weighted Cross-Entropy loss (2.5x weight on derivation tokens) and Knowledge Distillation KL Divergence (T=2.0) were used.
- Stage 2: Supervised Fine-Tuning on legal instruction data from the Alignment-Lab-AI/Lawyer-Instruct dataset. This stage leverages the structural isomorphism between mathematical and legal reasoning, allowing the model to apply its learned derivation skills to legal analysis.
What Makes This Model Different?
This model's primary differentiator is its two-stage reasoning transfer methodology at an extremely small scale. By first distilling "how to reason" from a high-entropy "Thinking" teacher and then fine-tuning on a specific domain like law, it aims to imbue complex reasoning capabilities into a highly compressed model. The use of proof-weighted distillation ensures that the limited parameter capacity is allocated to reasoning steps rather than just surface-level pattern matching.
Intended Uses
- Ultra-lightweight reasoning on mobile, edge, and IoT devices.
- Legal and STEM instruction-following.
- Educational tutoring and embedded inference.
- Component in multi-model pipelines where reasoning is needed under 500MB.
Limitations
Due to its 0.6B parameter size, the model has inherent capacity constraints. It may exhibit reasoning errors that larger models would not, particularly with multi-step derivations exceeding ~8 steps. Legal reasoning covers general concepts but lacks the nuance of larger models, and performance is weakest on underrepresented domains like molecular biology and physiology. Outputs should always be verified.