reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B-SFT

Hugging Face
TEXT GENERATIONConcurrent Unit Cost:1Model Size:2BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Mar 22, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Warm

reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B-SFT is a 1.7 billion parameter Qwen3-based model developed by Convergent Intelligence LLC: Research Division. It was created through a two-stage process: knowledge distillation from a 30B MoE teacher for STEM chain-of-thought reasoning, followed by supervised fine-tuning on legal instruction data. This model excels at structured legal reasoning and STEM problem-solving with step-by-step derivations, designed for efficient deployment.

Loading preview...

Model Overview

This model, reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B-SFT, is a 1.7 billion parameter Qwen3-based language model developed by Convergent Intelligence LLC: Research Division. It is distinguished by its unique two-stage training methodology aimed at building a structured reasoning backbone before layering domain-specific knowledge.

Training Methodology

Stage 1: Knowledge Distillation
The base Qwen3-1.7B model was distilled from a 30B parameter Mixture-of-Experts (MoE) teacher (Qwen3-30B-A3B-Instruct-2507) using 6,122 STEM chain-of-thought samples. This stage focused on teaching the model how to reason through a novel loss function combining proof-weighted cross-entropy (55%) and knowledge distillation KL divergence (45%). The proof-weighted loss amplifies penalties for errors within the derivation steps, ensuring the model learns rigorous reasoning rather than just memorizing answer formats.

Stage 2: Supervised Fine-Tuning (SFT)
The distilled model then underwent supervised fine-tuning on the Alignment-Lab-AI/Lawyer-Instruct dataset. This stage aimed to layer legal domain knowledge and instruction-following capabilities, leveraging the structured reasoning patterns established in Stage 1. The learning rate was deliberately lower in this stage to preserve the foundational reasoning backbone.

Key Capabilities

  • Structured Reasoning: Excels at producing step-by-step derivations in STEM problems.
  • Legal Analysis: Capable of structured legal reasoning, case analysis, and statutory interpretation.
  • Instruction Following: Responds to instructions across technical and legal domains.
  • Efficiency: A 1.7B parameter model, making it suitable for edge and mobile deployment, including GGUF quantized versions.

Good For

  • Structured legal reasoning and analysis.
  • STEM problem-solving requiring detailed derivations.
  • Educational tutoring and proof drafting.
  • Instruction-following in technical contexts.
  • Edge/mobile deployment via GGUF.
  • Component in multi-model pipelines and retrieval-augmented workflows requiring lightweight reasoning.

Finetunes
2 models