ISTA-DASLab/Meta-Llama-3-8B-Instruct

Model Overview

Meta-Llama-3-8B-Instruct is an 8 billion parameter instruction-tuned large language model developed by Meta, released on April 18, 2024. It is part of the Llama 3 family, which includes both 8B and 70B parameter models, optimized for dialogue use cases. The model employs an optimized transformer architecture and utilizes Grouped-Query Attention (GQA) for enhanced inference scalability. Training involved supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.

Key Capabilities & Performance

• Instruction-tuned for Dialogue: Specifically optimized for assistant-like chat applications.
• Strong Benchmarks: Outperforms many open-source chat models on common industry benchmarks, including significant improvements over Llama 2 models across various categories like MMLU (68.4 vs 34.1 for Llama 2 7B), HumanEval (62.2 vs 7.9), and GSM-8K (79.6 vs 25.7).
• Extensive Training Data: Pretrained on over 15 trillion tokens of publicly available online data, with fine-tuning data including over 10 million human-annotated examples.
• Safety & Refusal Improvements: Features extensive red teaming, adversarial evaluations, and safety mitigations, with a focus on significantly reducing false refusals compared to Llama 2.

Intended Use Cases

• Assistant-like Chat: Ideal for conversational AI and dialogue systems in English.
• Research and Commercial Applications: Suitable for a wide range of natural language generation tasks, with developers encouraged to fine-tune for specific needs.

Limitations & Responsible Use

• English-centric: Primarily intended for use in English, though fine-tuning for other languages is permissible under license.
• Static Model: Trained on an offline dataset; future versions will incorporate community feedback for safety improvements.
• Safety Considerations: Developers must implement additional safety best practices and tools (like Meta Llama Guard 2 and Code Shield) to tailor safety levels for specific applications, as residual risks may remain.