Rumiii/Qwen2.5-0.5B-Med-Pre-Trained-92k
Rumiii/Qwen2.5-0.5B-Med-Pre-Trained-92k is a 0.5 billion parameter Qwen2.5-0.5B model that has undergone continued pre-training on 92,000 English PubMed biomedical abstracts. This full-parameter CPT model is adapted towards biomedical vocabulary and medical terminology. It is primarily intended as a domain-adapted foundation for further fine-tuning on medical instruction datasets.
Loading preview...
Model Overview
This model, Rumiii/Qwen2.5-0.5B-Med-Pre-Trained-92k, is a continued pre-trained (CPT) version of the Qwen/Qwen2.5-0.5B base model. It has been specifically adapted for the biomedical domain through training on 92,000 English PubMed abstracts from the VietAI/vi_pubmed dataset.
Key Characteristics
- Base Model: Qwen/Qwen2.5-0.5B (0.5 billion parameters).
- Training Type: Full-parameter Continued Pre-Training (CPT), meaning all 494 million parameters were updated.
- Domain Adaptation: Adapted towards biomedical vocabulary, PubMed abstract structure, and medical terminology.
- Training Data: 92,000 English PubMed biomedical abstracts.
- Training Objective: Causal Language Modeling (CLM).
- Training Details: Trained for approximately 3 hours 45 minutes on a Kaggle Tesla T4, processing ~23.6 million tokens with a block size of 256 tokens.
Intended Use Cases
This is a base model, not an instruction-tuned or chat model. It serves as a specialized foundation for:
- Downstream medical Supervised Fine-Tuning (SFT).
- Research into biomedical domain adaptation techniques.
- As a starting point for developing medical reasoning models.
Not Intended For
- Direct conversational use due to the lack of instruction tuning.
- Clinical decision-making applications.
- Patient-facing applications.