ik-ram28/MedMistral-CPT-SFT-7B

MedMistral-CPT-SFT-7B: A Specialized French Medical LLM

MedMistral-CPT-SFT-7B is a 7 billion parameter causal language model, built upon the Mistral-7B-v0.1 architecture, specifically adapted for the French medical domain. Developed by ik-ram28, this model undergoes a two-stage training process: Continual Pre-Training (CPT) followed by Supervised Fine-Tuning (SFT), to achieve its specialized capabilities.

Key Capabilities

• French Medical Language Understanding: Optimized for processing and generating text in the French medical and healthcare context.
• Domain Adaptation: Achieves strong performance in specialized medical tasks through targeted pre-training and fine-tuning.
• Question Answering: Fine-tuned on a dataset of 30K French medical question-answer pairs, enhancing its ability to respond to medical queries.

Training Details

• Continual Pre-Training (CPT): Performed on the NACHOS corpus, a 7.4 GB collection of over 1 billion words from 24 French medical websites.
• Supervised Fine-Tuning (SFT): Utilized 30K French medical Q&A pairs (native, translated, and generated) with DoRA (Weight-Decomposed Low-Rank Adaptation).

Good for

• Medical Research: Analyzing French medical literature and extracting information.
• Healthcare Applications: Developing tools that require understanding or generating French medical text.
• Educational Purposes: Assisting in medical education by providing information or answering questions (with professional verification).

Note: This model is intended for research and educational use. All medical outputs should be verified by qualified medical professionals due to potential biases in training data.