reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B
reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B is a 1.7 billion parameter causal language model developed by Convergent Intelligence LLC: Research Division, distilled from a Qwen3-30B-A3B teacher. This model uniquely employs discrepancy-informed knowledge distillation, focusing on proof structure and reasoning pivots in STEM chain-of-thought samples. It excels at mathematical derivations, physics and engineering problem-solving, and educational tutoring, offering a lightweight yet capable solution for reasoning tasks.
Loading preview...
Qwen3-1.7B STEM Proof Distilled (DISC v3)
This 1.7 billion parameter causal language model, developed by Convergent Intelligence LLC: Research Division, is distilled from a Qwen3-30B-A3B teacher model. It's specifically designed for STEM reasoning, utilizing a novel discrepancy-informed knowledge distillation (KD) methodology that emphasizes the internal structure of proofs and derivations.
Key Capabilities & Differentiators
- Discrepancy-Weighted KD: Unlike standard distillation, this model identifies and amplifies learning on "reasoning pivots"—tokens where the derivation changes technique or introduces a key inference—by analyzing token-level KL divergence. This ensures the student model focuses on structural transitions rather than just average behavior.
- DG-Limit Smoothing: Stabilizes training by smoothing high-entropy (unstable) student tokens, replacing their logits with a neighborhood average before KD computation.
- Gap Energy Monitoring: Tracks structural divergence independent of average token loss, regularizing the model to prevent degradation on hard reasoning transitions even if overall loss improves.
- Proof-Weighted Cross-Entropy: Emphasizes derivation quality by giving higher weight to tokens within the proof span, decaying from 2.5x to 1.5x during training.
- MoE Teacher to Dense Student: Distills complex reasoning from a large Mixture-of-Experts (MoE) teacher into a more efficient 1.7B dense student, reducing inference overhead.
Good for
- Mathematical derivations and worked solutions: Excels at generating rigorous, step-by-step proofs.
- Physics and engineering problem-solving: Capable of handling complex scientific and technical problems.
- Educational tutoring and STEM walkthroughs: Ideal for generating detailed explanations and guided solutions.
- Lightweight reasoning deployment: Provides strong reasoning capabilities in a smaller, more efficient package compared to larger teacher models.
- Generator components in reasoning systems: Suitable for use in verifier-generator or retrieval-augmented reasoning architectures.