reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B

Qwen3-1.7B STEM Proof Distilled (DISC v3)

This 1.7B-parameter causal language model, developed by Reaperdoesntrun / Convergent Intelligence LLC: Research Division, is distilled from a Qwen3-30B-A3B teacher model. It employs a unique discrepancy-informed knowledge distillation methodology, focusing on the internal structure of reasoning rather than uniform token weighting. This approach identifies and amplifies learning at "reasoning pivots"—points where a derivation changes technique or introduces a key concept.

Key Capabilities

• Discrepancy-Weighted Knowledge Distillation: Uses token-level KL divergence to identify sharp changes in teacher-student divergence, treating these as reasoning pivots and applying amplified distillation weight.
• DG-Limit Smoothing: Stabilizes gradient flow by replacing student logits with a neighborhood average at high-entropy tokens, preventing unstable or incoherent local representations.
• Gap Energy Monitoring: Tracks structural divergence independent of average token loss, regularizing the model to prevent degradation on hard reasoning transitions even if overall loss improves.
• Proof-Weighted Cross-Entropy: Emphasizes derivation quality by giving higher weight to tokens within the proof span, decaying from 2.5x to 1.5x during training.
• Efficient Reasoning: Transfers complex reasoning behavior from a large Mixture-of-Experts (MoE) teacher into a smaller, dense 1.7B student model, reducing inference overhead.

Good For

• Mathematical derivations and worked solutions
• Proof-style explanations in STEM fields
• Physics and engineering problem-solving
• Educational tutoring and STEM walkthroughs
• Lightweight deployment of reasoning capabilities where larger models are too expensive
• Generator components in verifier-generator or retrieval-augmented reasoning systems