MedMistral-CPT-SFT-7B: A Specialized French Medical LLM
MedMistral-CPT-SFT-7B is a 7 billion parameter causal language model, built upon the Mistral-7B-v0.1 architecture, specifically adapted for the French medical domain. Developed by ik-ram28, this model undergoes a two-stage training process: Continual Pre-Training (CPT) followed by Supervised Fine-Tuning (SFT), to achieve its specialized capabilities.
Key Capabilities
- French Medical Language Understanding: Optimized for processing and generating text in the French medical and healthcare context.
- Domain Adaptation: Achieves strong performance in specialized medical tasks through targeted pre-training and fine-tuning.
- Question Answering: Fine-tuned on a dataset of 30K French medical question-answer pairs, enhancing its ability to respond to medical queries.
Training Details
- Continual Pre-Training (CPT): Performed on the NACHOS corpus, a 7.4 GB collection of over 1 billion words from 24 French medical websites.
- Supervised Fine-Tuning (SFT): Utilized 30K French medical Q&A pairs (native, translated, and generated) with DoRA (Weight-Decomposed Low-Rank Adaptation).
Good for
- Medical Research: Analyzing French medical literature and extracting information.
- Healthcare Applications: Developing tools that require understanding or generating French medical text.
- Educational Purposes: Assisting in medical education by providing information or answering questions (with professional verification).
Note: This model is intended for research and educational use. All medical outputs should be verified by qualified medical professionals due to potential biases in training data.