nicolay-r/qwen25-05b-multiclinsum-distil
nicolay-r/qwen25-05b-multiclinsum-distil is a 0.5 billion parameter decoder-based language model, distilled from Qwen/Qwen2.5-0.5B-Instruct. It is specifically fine-tuned on the MultiClinSum dataset for multilingual clinical text summarization, supporting English, French, Portuguese, and Spanish. This model is optimized for generating concise summaries of clinical texts, leveraging rationales inferred by a larger Qwen2.5-72B-Instruct model during its training process. Its primary application is in clinical natural language processing tasks requiring efficient and accurate summarization.
Loading preview...
Model Overview
nicolay-r/qwen25-05b-multiclinsum-distil is a 0.5 billion parameter decoder-based model, distilled from Qwen/Qwen2.5-0.5B-Instruct. It has been specifically fine-tuned for multilingual clinical text summarization.
Key Capabilities
- Multilingual Summarization: Supports summarization of clinical texts in English, French, Portuguese, and Spanish.
- Distillation Approach: Utilizes a unique distillation process where rationales for summaries were first inferred by the larger
Qwen/Qwen2.5-72B-Instructmodel. - Clinical Domain Focus: Trained on the MultiClinSum dataset, making it specialized for medical and clinical contexts.
- BioASQ Submission: Results from this model were used for a submission to the BioASQ-2025 Workshop.
Training Details
The training procedure involved preparing rationales using Qwen/Qwen2.5-72B-Instruct via the OpenRouter API, followed by fine-tuning on the MultiClinSum dataset. The fine-tuning process for 3 epochs took approximately 1 hour on a Google Colab A100 GPU. Evaluation was conducted using the ROUGE score on a test set of 20 documents across all supported languages.
Good for
- Clinical NLP: Ideal for applications requiring automated summarization of clinical documents.
- Multilingual Healthcare: Useful in scenarios where clinical text summarization is needed across multiple European languages.
- Resource-Efficient Deployment: As a 0.5B parameter model, it offers a more efficient solution for summarization tasks compared to larger models, while retaining specialized capabilities.