fang04/medforge-qwen3.5-4b-dpo
The fang04/medforge-qwen3.5-4b-dpo is a 4.5 billion parameter language model, based on the Qwen3.5-4B architecture, fine-tuned using a DPO (Direct Preference Optimization) approach with self-sampled medical reasoning data. This model is specifically designed for medical reasoning tasks, utilizing a unique training method where preference signals are derived entirely from the model's own sampled outputs, without external human-written or strong LLM-generated reasoning demonstrations. It aims to preserve the base model's native long-chain thinking capabilities, making it suitable for applications requiring medical inference.
Loading preview...
MedForge Qwen3.5-4B-DPO Overview
This model, fang04/medforge-qwen3.5-4b-dpo, is a 4.5 billion parameter language model built upon the Qwen3.5-4B base. It has undergone post-training using a Direct Preference Optimization (DPO) method, uniquely leveraging self-sampled medical reasoning data. The training process involves generating multiple samples for a given problem, using a calibrated validator to determine correctness, and then forming preference pairs from correct/incorrect solutions. This approach avoids introducing external reasoning styles, aiming to preserve the base model's inherent long-chain thinking.
Key Findings & Training
Comparative experiments showed that traditional SFT (Supervised Fine-Tuning) with distilled CoT (Chain-of-Thought) reasoning often degraded the base model's performance on medical exams. In contrast, this DPO-based self-sampling method did not cause such degradation, maintaining performance close to the original Qwen3.5-4B base model on benchmarks like CMExam, CMB-val, and MedXpertQA. The training utilized approximately 1,000 preference pairs derived from 3,000 verifiable medical problems, with a LoRA full-layer setup on the Qwen/Qwen3.5-4B base.
Limitations
It's important to note that this model is a product of research and engineering practice and does not constitute medical advice. Its evaluation was based on fixed-seed sampled test sets, not full leaderboard metrics, and involved a single random seed training run. The 'manual calibration' of the validator primarily used strong model proxy labeling with human spot checks. While trained with mixed Chinese and English data, Chinese test sets were the primary target.