reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B

Hugging Face
TEXT GENERATIONPricing:Input $0.32 / Cached $0.064 / Output $1.6Concurrent Unit Cost:1Model Size:2BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Mar 22, 2026License:apache-2.0Architecture:Transformer0.0K Open Weights Featherless Exclusive Warm

reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B is a 1.7 billion parameter causal language model from Convergent Intelligence LLC, distilled from a Qwen3-30B-A3B teacher. This model is uniquely optimized for STEM chain-of-thought reasoning, emphasizing proof structure and detecting reasoning pivots through discrepancy-informed knowledge distillation. It excels at mathematical derivations, proof-style explanations, and problem-solving in physics and engineering, offering a lightweight solution for complex reasoning tasks.

Loading preview...

Overview

This model, reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B, is a 1.7 billion parameter causal language model developed by Convergent Intelligence LLC: Research Division. It is distilled from a larger Qwen3-30B-A3B teacher model, specifically trained on 6,122 STEM chain-of-thought samples. The core innovation lies in its discrepancy-informed knowledge distillation (DISC v3) methodology, which goes beyond standard distillation by focusing on the internal structure of reasoning.

Key Capabilities & Differentiators

  • Discrepancy-Weighted KD: Identifies and amplifies learning at "reasoning pivots"—critical points where a derivation changes technique or introduces a key lemma—by analyzing token-level KL divergence.
  • DG-Limit Smoothing: Stabilizes training by smoothing high-entropy (unstable) student tokens, preventing noisy gradient flow.
  • Gap Energy Monitoring: Tracks structural divergence independent of average loss, helping to prevent degradation of reasoning transitions even if overall loss improves.
  • Proof-Weighted Cross-Entropy: Emphasizes derivation quality by giving higher weight to tokens within the proof span, decaying from 2.5x to 1.5x during training.
  • Efficient Reasoning: Transfers complex reasoning behavior from a large Mixture-of-Experts (MoE) teacher into a smaller, dense student model, making it suitable for lightweight deployment.

Intended Uses

  • Mathematical derivations and worked solutions
  • Proof-style explanations and educational tutoring in STEM fields
  • Physics and engineering problem-solving
  • Lightweight reasoning deployment where larger models are cost-prohibitive
  • Generator components in verifier-generator or retrieval-augmented reasoning systems.

Finetunes
3 models