reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B
Qwen3-1.7B-Distilled-30B-A3B is a 1.7 billion parameter causal language model from Convergent Intelligence LLC: Research Division, distilled from a Qwen3-30B-A3B teacher. This model is specifically optimized for STEM chain-of-thought reasoning, emphasizing proof structure and detecting reasoning pivots through discrepancy-informed knowledge distillation. It excels at mathematical derivations, proof-style explanations, and physics/engineering problem-solving, offering a lightweight solution for complex reasoning tasks.
Loading preview...
Overview
This model, reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B, is a 1.7 billion parameter causal language model developed by Convergent Intelligence LLC: Research Division. It is distilled from a larger Qwen3-30B-A3B teacher model, specifically targeting STEM chain-of-thought reasoning. The distillation process uses a novel discrepancy-informed methodology to emphasize proof structure and critical reasoning steps, rather than treating all tokens uniformly.
Key Distillation Methodology
This model's unique training incorporates three discrepancy-informed operators:
- Discrepancy-Weighted KD: Amplifies knowledge distillation weight for "reasoning pivot" tokens, identified by sharp changes in KL divergence between teacher and student, ensuring the student focuses on structural transitions.
- DG-Limit Smoothing: Stabilizes gradient flow by replacing high-entropy student logits with a neighborhood average, preventing unstable representations during distillation.
- Gap Energy Monitoring: Tracks structural divergence and acts as a regularizer, preventing degradation of reasoning transitions even if average loss improves.
Additionally, proof-weighted cross-entropy is applied, giving higher importance to tokens within the derivation span, with emphasis decaying from 2.5x to 1.5x over training.
Training Details
The model was trained on 6,122 STEM chain-of-thought samples from 10 domain-specific datasets, including Physics, Linear Algebra, and Engineering. It uses a training context length of 1024 tokens and bf16 precision. The teacher model is a large Mixture-of-Experts (MoE) model, allowing the dense 1.7B student to inherit complex reasoning behaviors without MoE inference overhead.
Intended Uses
- Mathematical derivations and worked solutions
- Proof-style explanations
- Physics and engineering problem-solving
- Educational tutoring and STEM walkthroughs
- Lightweight reasoning deployment where larger models are too expensive
- Generator components in verifier-generator or retrieval-augmented reasoning systems