fang04/medforge-qwen3.5-4b-distill
The fang04/medforge-qwen3.5-4b-distill is a 4.5 billion parameter language model, distilled from Qwen3.5-4B and fine-tuned for medical reasoning tasks. It leverages a 32768-token context length and achieves significant improvements on medical benchmarks like CMExam, demonstrating enhanced performance and efficiency in medical question answering. This model is specifically optimized for accurate and concise responses in the medical domain, making it suitable for applications requiring specialized medical knowledge.
Loading preview...
MedForge Qwen3.5-4B-Distill: Specialized Medical Reasoning
This model is the second generation of medical reasoning models distilled from the Qwen3.5-4B base, specifically trained by fang04 for enhanced performance in medical question answering. It utilizes a unique distillation process, leveraging high-quality educational materials generated by a strong teacher model (DeepSeek V4-flash, 93.2% on CMExam) and a meticulously filtered dataset.
Key Capabilities & Performance
- Significant Medical Performance Boost: Achieves 80.4% on CMExam (in-domain), a +6.5 percentage point improvement over the Qwen3.5-4B base, and 65.7% on CMB-val (transfer), a +5.7 percentage point improvement.
- Efficient & Concise Responses: Reduces average token length per question significantly (e.g., 1.8k vs 5.2k on CMExam), indicating more direct and less verbose answers.
- Robust Training Methodology: Corrects issues from previous distillation attempts, ensuring the teacher model was validated on the target exam, using in-domain question sources, consistent prompts, and proper length handling.
- High Completion Rate: Achieves a 100% completion rate across three test sets, even under greedy decoding, and avoids repetition.
When to Use This Model
- Medical Question Answering: Ideal for applications requiring accurate and concise responses to medical questions, particularly those in Chinese medical examination contexts.
- Resource-Constrained Environments: Its 4.5B parameter size makes it suitable for deployment where larger models might be impractical, while still offering strong specialized performance.
- Research in Medical AI: Provides a strong baseline for further research into medical language models and distillation techniques.
Limitations
- Not for Medical Advice: As an AI, it cannot provide medical advice or be used for automatic diagnosis and prescription.
- Limited Generalization: While showing some transfer ability, its primary strength is in the Chinese medical examination domain. MedXpertQA (English) showed no significant gain.
- No "I don't know": The model rarely abstains from answering, which might be a limitation in critical applications.