reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B

Hugging Face
TEXT GENERATIONPricing:Input $0.32 / Cached $0.064 / Output $1.6Concurrent Unit Cost:1Model Size:2BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Mar 22, 2026License:apache-2.0Architecture:Transformer0.0K Open Weights Featherless Exclusive Loading

reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B is a 1.7 billion parameter causal language model from Convergent Intelligence LLC: Research Division, distilled from a Qwen3-30B-A3B teacher. This model is specifically optimized for STEM chain-of-thought reasoning, emphasizing proof structure and detecting reasoning pivots through discrepancy-informed knowledge distillation. It excels at mathematical derivations, proof-style explanations, and physics/engineering problem-solving, offering a lightweight solution for complex reasoning tasks.

Loading preview...

Overview

This model, reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B, is a 1.7 billion parameter causal language model developed by Convergent Intelligence LLC: Research Division. It is distilled from a larger Qwen3-30B-A3B-Instruct teacher model, specifically trained on 6,122 STEM chain-of-thought samples. The core innovation lies in its discrepancy-informed knowledge distillation (DISC v3) methodology, which goes beyond standard distillation by focusing on the internal structure of reasoning.

Key Distillation Innovations

This model's training incorporates three unique discrepancy-informed operators:

  • Discrepancy-Weighted KD: Identifies and amplifies learning at "reasoning pivots" (sharp changes in teacher-student divergence) using token-level KL structure, ensuring the student focuses on critical transitions.
  • DG-Limit Smoothing: Stabilizes gradient flow by replacing high-entropy (unstable) student logits with a neighborhood average, preventing noisy distillation in incoherent regions.
  • Gap Energy Monitoring: Tracks structural divergence independent of average loss, regularizing against degradation in reasoning transitions even if overall loss improves.

Additionally, it uses proof-weighted cross-entropy, where tokens within derivation spans receive higher weight, decaying from 2.5x to 1.5x during training, to emphasize derivation quality.

Intended Uses

This model is particularly well-suited for:

  • Mathematical derivations and worked solutions
  • Proof-style explanations and educational tutoring in STEM fields
  • Physics and engineering problem-solving
  • Lightweight reasoning deployment where larger models are cost-prohibitive
  • Generator components in verifier-generator or retrieval-augmented reasoning systems

Finetunes
3 models