Llama-3-8B-UltraMedical: Specialized Biomedical LLM

Llama-3-8B-UltraMedical, developed by the Tsinghua C3I Lab, is an 8 billion parameter language model built on Meta's Llama-3-8B foundation. It is specifically fine-tuned for biomedical applications using the extensive UltraMedical dataset, which includes 410,000 diverse entries.

Key Capabilities & Performance

• Biomedical Specialization: Designed to enhance medical examination access, literature comprehension, and clinical knowledge.
• High Benchmark Scores: Achieves top average scores across popular medical benchmarks, including MedQA, MedMCQA, PubMedQA, and MMLU-Medical.
• Outperforms Competitors: Significantly outperforms models like Flan-PaLM, OpenBioLM-8B, Gemini-1.0, GPT-3.5, and Meditron-70b in medical evaluations.
• Training Details: Trained for 50 hours on 8 x A6000 GPUs using the FSDP framework, with a global batch size of 128 and a max length of 1024 tokens.

Usage & Limitations

• Input Format: Utilizes the Llama-3 default chat template without a system prompt, with specific formatting recommendations for multi-choice QA and PubMedQA to reproduce evaluation results.
• Current Limitation: This version primarily supports single-turn dialogue, with multi-turn capabilities planned for future updates.
• Caution on Hallucinations: Users are advised to validate model outputs with trusted medical sources and expert consultation due to potential hallucination issues in clinical settings.