ik-ram28/BioMistral-CPT-SFT-7B

BioMistral-CPT-SFT-7B: French Medical Language Model

BioMistral-CPT-SFT-7B is a 7 billion parameter causal language model, building upon the BioMistral-7B base. Developed by ik-ram28, this model is uniquely adapted for the French medical domain through a two-stage training process: Continual Pre-Training (CPT) and Supervised Fine-Tuning (SFT).

Key Capabilities & Training

• Domain-Specific Adaptation: The model underwent CPT on the NACHOS corpus, a 7.4 GB collection of over 1 billion words from 24 French medical websites, ensuring deep understanding of medical terminology and context.
• Question Answering: SFT was performed using DoRA (Weight-Decomposed Low-Rank Adaptation) on a diverse dataset of 30,000 French medical question-answer pairs, including native, translated, and generated questions, enhancing its ability to respond to medical queries.
• Language Focus: Primarily designed for French medical applications, adapting an English medical base model to a new linguistic and domain context.
• Research & Education: Intended for research and educational use, providing a specialized tool for exploring French medical language processing.

Limitations & Considerations

• Accuracy: The model is explicitly stated to be for research and educational purposes only, with performance limitations making it unsuitable for critical medical applications.
• Bias: May inherit biases from both the English and French medical literature used in its training.

Good for

• Developing applications requiring French medical text generation.
• Research into domain adaptation of large language models for healthcare.
• Educational tools for medical students or professionals in French-speaking contexts.
• Exploring question-answering systems within the French medical domain.