EPFLiGHT/Apertus-8B-MeditronFO
EPFLiGHT/Apertus-8B-MeditronFO is an 8 billion parameter medical specialist LLM developed by LiGHT, based on Apertus-8B-Instruct. This model is part of the Fully Open Meditron family, offering an end-to-end auditable pipeline for clinical LLMs with open weights, data, and training. It establishes new state-of-the-art medical accuracy among small fully open medical LLMs, showing a +13.35 point improvement over its base on aggregate medical benchmarks. It is specifically designed and intended for medicine-related tasks and evaluations.
Loading preview...
Overview
EPFLiGHT/Apertus-8B-MeditronFO is a specialized 8 billion parameter medical Large Language Model (LLM) developed by LiGHT. It is built upon the Apertus-8B-Instruct base model and is a key component of the Fully Open Meditron family. This initiative emphasizes an end-to-end auditable pipeline for clinical LLMs, featuring open weights, open data, an open training recipe, and a clinician-vetted corpus construction.
Key Capabilities & Differentiators
- Medical Specialization: Fine-tuned specifically for medicine-related tasks, leveraging the Fully Open Meditron Corpus.
- State-of-the-Art Medical Accuracy: Achieves the best medical accuracy on standard medical benchmarks among small fully open medical LLMs.
- Significant Performance Gain: Demonstrates a substantial +13.35 point improvement over its base model, Apertus-8B-Instruct, on aggregate medical benchmarks.
- Clinician-Validated Preference: Preferred over Apertus-70B in 86.9% of comparisons based on the clinician-validated LM judge evaluation AutoMOOVE.
- Auditable Pipeline: Offers full transparency with open weights, data, and training methodology, ensuring an auditable process for clinical LLM development.
Training Details
The model was trained on the Fully Open Meditron Corpus, comprising 601k examples (~150M tokens). This corpus aggregates eight public medical QA datasets and includes three clinician-vetted synthetic components: exam-style QA, guideline-grounded QA from 46,469 clinical practice guidelines, and open-ended clinical vignettes. Training was conducted on 8 NVIDIA GH200 nodes for approximately 6 hours, utilizing Axolotl with FSDP v2 / DeepSpeed ZeRO-3, Flash Attention 2, and bf16 mixed precision.
Intended Use & Limitations
Apertus-8B-MeditronFO is intended as an assistive tool for medicine-related tasks and evaluations. Users should be aware that while specialized, generated content may not always be factually accurate or free from biases. It is crucial to verify important information and critically evaluate any output, as these models are not definitive sources of information.