lighteternal/Llama3-merge-biomed-8b

Overview

The lighteternal/Llama3-merge-biomed-8b is an 8 billion parameter language model resulting from a DARE-TIES merge of three distinct Llama 3-based models: Llama3-8b-Instruct, NousResearch/Hermes-2-Pro-Llama-3-8B, and aaditya/Llama3-OpenBioLLM-8B. This experimental merge aims to combine the strengths of general language understanding with specialized biomedical knowledge.

Key Capabilities & Performance

The model demonstrates promising performance, particularly in biomedical domains and complex reasoning tasks, as evidenced by its scores on the Hugging Face Open LLM Leaderboard:

• Biomedical Expertise: Achieves significantly higher accuracy on various HendrycksTest tasks, including:
  • Anatomy: 72.59% (vs. 65.19% for Llama3-8B-Instruct)
  • Clinical Knowledge: 77.83% (vs. 74.72%)
  • College Biology: 81.94% (vs. 79.86%)
  • Medical Genetics: 86.00% (vs. 80.00%)
  • Professional Medicine: 77.94% (vs. 71.69%)
• Enhanced Reasoning: Shows improvements in general reasoning benchmarks:
  • ARC Challenge: 59.39% Accuracy (vs. 57.17% for Llama3-8B-Instruct)
  • Winogrande: 75.93% Accuracy (vs. 74.51%)
• General Understanding: Also performs well on Hellaswag with 62.59% Accuracy.

Merge Details

This model was created using the DARE TIES merge method, with meta-llama/Meta-Llama-3-8B-Instruct serving as the base model. The configuration involved specific density and weight parameters for each merged component to balance their contributions.

Recommended Usage

Users should follow the prompt template recommended for Llama 3 models to ensure optimal performance.