reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B-SFT

Hugging Face
TEXT GENERATIONPricing:Input $0.32 / Cached $0.064 / Output $1.6Concurrent Unit Cost:1Model Size:2BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Mar 22, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Warm

reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B-SFT is a 2 billion parameter Qwen3-based causal language model developed by Convergent Intelligence LLC. It was created through a two-stage process: knowledge distillation from a 30B MoE teacher for STEM reasoning, followed by supervised fine-tuning on legal instruction data. This model excels at structured legal reasoning and STEM problem-solving with step-by-step derivations, designed for efficient application of learned reasoning patterns.

Loading preview...

Model Overview

This model, developed by Convergent Intelligence LLC, is a 2 billion parameter Qwen3-based causal language model (advertised as 1.7B) with a 1024 token training context. It was built using a unique two-stage training pipeline designed to impart structured reasoning capabilities efficiently. The first stage involved knowledge distillation from a 30B-parameter Mixture-of-Experts (MoE) teacher (Qwen3-30B-A3B-Instruct-2507) using 6,122 STEM chain-of-thought samples. This stage utilized a novel proof-weighted cross-entropy loss combined with KL divergence to prioritize learning rigorous derivation steps over mere answer formatting.

Key Capabilities

  • Structured Reasoning: Learns to produce step-by-step derivations in STEM domains (physics, linear algebra, differential equations) through proof-weighted distillation.
  • Legal Analysis: Applies learned reasoning structures to legal concepts, case analysis, and statutory interpretation after supervised fine-tuning on legal instruction data.
  • Instruction Following: Responds to both STEM derivation prompts and general instruction-following formats.
  • Lightweight Deployment: A 1.7B dense model that captures knowledge from a 30B MoE teacher, suitable for edge/mobile deployment via GGUF quantizations.

Good For

  • Structured legal reasoning and analysis.
  • STEM problem-solving requiring detailed, step-by-step derivations.
  • Instruction-following in technical and legal domains.
  • Educational tutoring and proof drafting.
  • Use as a component in multi-model pipelines or retrieval-augmented generation (RAG) workflows requiring efficient reasoning.

Finetunes
2 models