reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B

Overview

This model, reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B, is a 1.7 billion parameter causal language model developed by Convergent Intelligence LLC: Research Division. It is distilled from a larger Qwen3-30B-A3B teacher model, specifically targeting STEM chain-of-thought reasoning. The distillation process employs a novel "discrepancy-informed" methodology to capture the internal structure of reasoning, rather than treating all tokens uniformly.

Key Distillation Innovations

This model's training incorporates three unique discrepancy-informed operators:

• Discrepancy-Weighted KD: Identifies and amplifies distillation weight for "reasoning pivots"—tokens where the teacher-student KL divergence shows sharp local changes, indicating critical shifts in derivation.
• DG-Limit Smoothing: Stabilizes training by replacing high-entropy student logits with a neighborhood average at unstable points, preventing noisy token-wise distillation.
• Gap Energy Monitoring: Tracks structural divergence independent of average token loss, helping to prevent degradation of reasoning transitions even if overall loss improves.

Additionally, it uses proof-weighted cross-entropy, emphasizing derivation quality over answer formatting, with proof emphasis decaying from 2.5x to 1.5x during training. The model was trained on 6,122 STEM chain-of-thought samples with a context length of 1024 tokens.

Intended Uses

• Mathematical derivations and worked solutions
• Proof-style explanations and educational tutoring in STEM
• Physics and engineering problem-solving
• Lightweight reasoning deployment where larger models are too expensive
• Generator components in verifier-generator or retrieval-augmented reasoning systems