Tulu 13B: Instruction-Tuned LLaMa Model

The allenai/tulu-13b is a 13 billion parameter LLaMa-based model developed by Allen Institute for AI, specifically instruction-tuned to excel at a wide range of general-purpose tasks. It was fine-tuned using a comprehensive blend of instruction datasets, including FLAN V2, CoT, Dolly, Open Assistant 1, GPT4-Alpaca, Code-Alpaca, and ShareGPT, as part of the research detailed in the paper "How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources".

Key Capabilities & Performance

This model demonstrates solid performance across various benchmarks, reflecting its diverse training:

• Reasoning & QA: Achieves 51.8 on MMLU 5-shot and 46.1 on TydiQA Gold-Passage.
• Mathematical Reasoning: Scores 36.5 on GSM CoT.
• Code Generation: Attains 21.3 Pass@1 and 35.0 Pass@10 on Codex-Eval.
• Instruction Following: Shows strong alignment with human instructions, scoring 53.9 on AlpacaFarm vs Davinci-003.

Usage & Integration

tulu-13b is distributed as a model diff, requiring an existing LLaMa model in Hugging Face format for recovery. The model expects inputs formatted with specific <|user|> and <|assistant|> tokens, emphasizing the importance of a newline after <|assistant|> for optimal generation quality. The codebase for training and evaluation is available via AllenAI's Open-Instruct repository.