reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B-SFT

Hugging Face
TEXT GENERATIONPricing:Input $0.32 / Cached $0.064 / Output $1.6Concurrent Unit Cost:1Model Size:2BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Mar 22, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Warm

The reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B-SFT is a 2 billion parameter Qwen3-based causal language model developed by Convergent Intelligence LLC. It was created through a two-stage process: knowledge distillation from a 30B MoE teacher on STEM chain-of-thought data, followed by supervised fine-tuning on legal instruction data. This model excels at structured reasoning in both STEM and legal domains, providing step-by-step derivations and instruction-following capabilities, making it suitable for educational tutoring and legal analysis.

Loading preview...

Overview

This model, reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B-SFT, is a 2 billion parameter Qwen3-based language model developed by Convergent Intelligence LLC. It is designed for structured reasoning, particularly in STEM and legal domains, and was trained using a unique two-stage distillation and fine-tuning process. The model has a context length of 1024 tokens during training and is available in bf16 precision.

Key Capabilities

  • Structured Reasoning: Excels at producing rigorous, step-by-step derivations for STEM problems (physics, linear algebra, differential equations, etc.).
  • Legal Instruction Following: Applies learned reasoning patterns to legal analysis, including case analysis, statutory interpretation, and general legal concepts.
  • Efficient Knowledge Transfer: Utilizes a novel proof-weighted knowledge distillation method from a 30B MoE teacher, allowing a smaller 1.7B parameter model to capture complex reasoning structures.
  • Dual Prompt Formats: Supports both a "Problem: [question]\nProof:" format for derivations and an "### Instruction:\n[task]\n### Response:" format for general instruction following.

Good for

  • Educational Tutoring: Generating detailed, step-by-step solutions for STEM problems.
  • Legal Analysis Support: Assisting with structured legal reasoning, case analysis, and statutory interpretation.
  • Resource-Constrained Environments: Its 1.7B parameter size and availability in GGUF format make it suitable for edge or mobile deployment.
  • Component in Multi-Model Pipelines: Can serve as a lightweight reasoning component in more complex AI systems.
  • Retrieval-Augmented Workflows: Provides structured reasoning capabilities for RAG applications requiring detailed derivations or legal insights.

Training Methodology Highlights

The model's training involved two stages:

  1. Knowledge Distillation (STEM Reasoning): The base Qwen3-1.7B was distilled from Qwen3-30B-A3B-Instruct-2507 using 6,122 STEM chain-of-thought samples. A "Proof-Weighted Cross-Entropy" loss (55%) prioritized learning derivation steps, combined with "Knowledge Distillation KL Divergence" (45%) to transfer the teacher's probability landscape.
  2. Supervised Fine-Tuning (Legal Domain): The distilled model was then fine-tuned on the Alignment-Lab-AI/Lawyer-Instruct dataset. This stage layered legal domain knowledge onto the established reasoning backbone, with a lower learning rate to preserve the distillation gains. This approach ensures the model applies learned reasoning structures to new domains rather than just memorizing patterns.

Finetunes
2 models