HealthDataHub/PARTAGES-Qwen3-8B-PDAPT-SLERP

Overview

HealthDataHub/PARTAGES-Qwen3-8B-PDAPT-SLERP is an 8 billion parameter causal language model built upon Qwen3-8B-Base. Developed by HealthDataHub, its primary goal is to serve as a foundational model for adaptation within healthcare establishments. It distinguishes itself by being trained exclusively on commercially permissible data sources, making it suitable for both research and commercial applications.

Key Capabilities

• Domain-Adaptive Pre-Training (DAPT): Continuously pre-trained on PARCOMED, a French biomedical corpus comprising scientific documents, drug notices, clinical cases, and regulatory texts.
• SLERP Fusion: The specialized checkpoint was merged with the original Qwen3-8B-Base using Spherical Linear Interpolation (SLERP) with a coefficient of 0.5. This method helps mitigate the degradation of general language capabilities often seen after domain-specific adaptation.
• Commercial Use Compatible: All training data sources (e.g., PARCOMED) are licensed for commercial use, ensuring broad applicability.
• Base Model: This is a base model, not instruction-tuned, intended for further fine-tuning on specific downstream tasks.

Good for

• Healthcare Applications: Ideal for developers looking to build or adapt LLMs for specific tasks within French healthcare contexts.
• Domain-Specific Fine-Tuning: Serves as a strong starting point for further specialization in medical or biomedical NLP tasks.
• Commercial Deployments: Suitable for projects requiring models trained on commercially viable datasets.

Performance Insights

Evaluations using lm-evaluation-harness across French and English medical and general domains show that while the direct impact of DAPT on FR-MEDICAL tasks is modest for this 8B model, the SLERP fusion successfully restores general domain performance that would otherwise degrade with DAPT alone. For instance, the 'Health' category in French, which dropped significantly after DAPT, was brought back to baseline levels post-SLERP.