Model Overview

The allenai/open-instruct-gpt4-alpaca-13b is a 13 billion parameter LLaMa-based model developed by AllenAI. It has been instruction-tuned using the GPT-4 Alpaca dataset, aiming to enhance its ability to follow diverse instructions. This model is released as a weight difference (diff) and necessitates a pre-existing LLaMa model in Hugging Face format for full recovery and usage. The training methodology and evaluation are detailed in the paper "How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources".

Key Capabilities

• Instruction Following: Fine-tuned on the GPT-4 Alpaca dataset for improved response generation based on user instructions.
• Benchmark Performance: Achieves an average score of 32.5 across a suite of benchmarks, including:
  • MMLU (0-shot: 47.0, 5-shot: 46.9)
  • GSM (Direct: 7.5, CoT: 14.0)
  • BBH (Direct: 34.9, CoT: 38.3)
  • Codex-Eval (Pass@1: 15.8, Pass@10: 32.5)
  • AlpacaFarm vs Davinci-003: 61.1
• Input Format: Optimized for a specific input format: <|user|> Your message here! <|assistant|> to ensure optimal generation quality.

Good For

• Research in Instruction Tuning: Ideal for researchers exploring the impact of instruction tuning on open-source models, particularly with GPT-4 generated datasets.
• General Instruction-Following Tasks: Suitable for applications requiring a 13B parameter model capable of understanding and responding to a variety of prompts.
• Comparative Studies: Useful for comparing performance against other instruction-tuned models, especially those based on the LLaMa architecture.