Overview
allenai/open-instruct-gpt4-alpaca-7b is a 7 billion parameter LLaMa model fine-tuned by AllenAI using the high-quality GPT-4 Alpaca dataset. This model is a component of the research presented in the paper "How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources." It is distributed as a model difference (diff), meaning users must combine it with an existing LLaMa base model to reconstruct the full instruction-tuned model.
Key Characteristics & Usage
- Base Model: Built upon the LLaMa architecture.
- Training Data: Fine-tuned using the GPT-4 Alpaca dataset, which consists of instructions generated by GPT-4.
- Distribution: Provided as a
model diff, requiring a recovery process with an original LLaMa model in Hugging Face format. Instructions for this process are available in the associated GitHub repository. - Input Format: Optimized for a specific input format:
\n<|user|>\nYour message here!\n<|assistant|>\n. Including a newline after <|assistant|> is crucial for optimal generation quality.
Performance Highlights
Benchmarking results from the associated paper indicate its performance across various tasks:
- MMLU (0-shot): 42.6
- MMLU (5-shot): 38.3
- GSM Direct: 6.5
- GSM CoT: 10.0
- AlpacaFarm vs Davinci-003: 57.0
- Average Score: 28.3
This model is primarily intended for research and development in instruction tuning, offering insights into the capabilities of models fine-tuned on advanced instruction datasets.