fiap-hospital-helper/hospital-helper-qwen2.5-1.5b
The fiap-hospital-helper/hospital-helper-qwen2.5-1.5b is a 1.5 billion parameter causal language model, fine-tuned by FIAP for hospital support tasks, including question answering and clinical protocols. Based on Qwen/Qwen2.5-1.5B-Instruct, it leverages LoRA for domain-specific adaptation within a 32768 token context length. This model is optimized for providing relevant information in a hospital setting, making it suitable for academic and experimental applications in healthcare AI.
Loading preview...
Overview
This model, fiap-hospital-helper/hospital-helper-qwen2.5-1.5b, is a 1.5 billion parameter language model developed by FIAP. It is fine-tuned from the Qwen/Qwen2.5-1.5B-Instruct base model using LoRA (Low-Rank Adaptation) to specialize in hospital support tasks. The fine-tuning process involved merging LoRA weights back into the base model, utilizing transformers, trl (SFTTrainer), and peft frameworks.
Key Capabilities
- Hospital-specific Q&A: Designed to answer questions relevant to a hospital environment.
- Clinical Protocol Support: Provides information derived from clinical and institutional protocols.
- Efficient Fine-tuning: Employs 4-bit quantization (NF4, double quant) during training for resource efficiency.
- Version Control: Supports loading specific model versions via tags (e.g.,
v1.0) for reproducibility.
Training Details
The model was trained on hospital-domain specific datasets, including question-answer pairs and examples from clinical protocols. These datasets were converted into the Qwen2.5-Instruct chat template format. Training parameters included 1 epoch, a learning rate of 2e-4, and gradient checkpointing, with a context length of 32768 tokens.
Limitations
- Small Scale: As a 1.5B parameter model, its responses may be less robust than larger models outside its trained domain.
- Limited Data: Fine-tuning was performed with a limited dataset, typical of an academic project.
- Experimental Use Only: This model is experimental and educational; it is not validated for clinical decisions or regulatory use.