MohamedAhmedAE/distil_Med42_8B_Llama-3.2-1B
MohamedAhmedAE/distil_Med42_8B_Llama-3.2-1B is a 1-billion parameter medical language model, distilled from the 8B-parameter Llama3-Med42-8B into a Llama-3.2-1B student base. Developed by Mohamed Abo El-Enen, Sally Saad, and Taymoor Nazmy, this model is optimized for lightweight medical NLP applications, retaining 89.3% of its teacher's token-level accuracy while reducing parameters by 75%. It achieves 47.7% average accuracy on MMLU-Medical and offers a high inference throughput of 59.5 tokens/sec, making it suitable for efficient medical knowledge processing.
Loading preview...
Overview
MohamedAhmedAE/distil_Med42_8B_Llama-3.2-1B is a 1-billion parameter medical language model, a lightweight student distilled from the larger Llama3-Med42-8B (8.03B parameters) into a Llama-3.2-1B base. This distillation process, detailed in the paper "DistilLLM-Med: A Lightweight Medical Language Model through Knowledge Distillation" (IEEE ICICIS 2025), aims to create a more efficient model for medical NLP tasks.
Key Capabilities & Features
- Efficient Medical Expertise: Retains 89.3% of the teacher model's token-level accuracy while reducing parameter count by 75%, making it significantly more resource-efficient.
- Distillation Methodology: Utilizes temperature-scaled KL-divergence, specialty-weighted losses, and attention-map alignment for effective knowledge transfer.
- Performance: Achieves 47.7% average accuracy on MMLU-Medical, representing a 20.5% relative improvement over the base LLaMA 3.2-1B, and an inference throughput of 59.5 tokens/sec.
- Training Data: Trained on a unified corpus of 18 medical benchmarks, including MMLU (medical subtasks), MedMCQA, PubMedQA, and clinical dialogues.
Good For
- Research in Efficient Medical NLP: Ideal for studies on lightweight medical language models and knowledge distillation techniques.
- Edge/Low-Resource Deployment: Suitable for applications where computational resources are limited, such as on-device medical assistance.
- Exploration of Distillation Techniques: Provides a practical example of advanced distillation methods, including progressive temperature scheduling and attention alignment.
Limitations
- Research Checkpoint: The model is a research checkpoint, trained for approximately 0.6 epochs, and may not be fully converged.
- Requires Expert Oversight: Independent review found critical factual errors in ~21% of sampled answers, emphasizing that it is not a standalone diagnostic tool and requires qualified human oversight.