Hugging Face Overview
Llama-3.1-Tulu-3-70B is a 70 billion parameter instruction-following model from AllenAI, built upon Meta's Llama 3.1 base model. It is part of the Tülu 3 family, which emphasizes fully open-source data, code, and training recipes. The model is primarily English-language and is licensed under the Llama 3.1 Community License Agreement.
Key Capabilities
- Instruction Following: Designed for state-of-the-art performance across a diversity of tasks, including general chat.
- Mathematical Reasoning: Shows strong performance on benchmarks like MATH and GSM8K.
- Instruction Following Evaluation (IFEval): Excels in complex instruction following scenarios.
- Open-Source Approach: Provides a comprehensive post-training package with open-source data, code, and recipes.
Performance Highlights
On a range of benchmarks, the Tülu 3 70B model achieves an average score of 76.0, outperforming Llama 3.1 70B Instruct (73.4) and Qwen 2.5 72B Instruct (71.5). Notable scores include:
- PopQA (15 shot): 46.5
- BigBenchHard (3 shot, CoT): 82.0
- MATH (4 shot CoT, Flex): 63.0
- GSM8K (8 shot, CoT): 93.5
- Safety (6 task avg.): 88.3
Usage Considerations
The model has limited safety training and does not include in-the-loop filtering, meaning it can produce problematic outputs. It is intended for research and educational use, and its fine-tuning involved datasets with outputs from third-party models, subject to their respective terms of use.