allenai/open-instruct-gpt4-alpaca-7b
The allenai/open-instruct-gpt4-alpaca-7b is a 7 billion parameter LLaMa-based model developed by AllenAI, fine-tuned on the GPT-4 Alpaca dataset. This model is specifically designed for instruction-following tasks, leveraging the quality of GPT-4 generated instructions. It serves as a model diff, requiring recovery with an original LLaMa model, and is suitable for research into instruction tuning on open resources.
Loading preview...
Overview
allenai/open-instruct-gpt4-alpaca-7b is a 7 billion parameter LLaMa model fine-tuned by AllenAI using the high-quality GPT-4 Alpaca dataset. This model is a component of the research presented in the paper "How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources." It is distributed as a model difference (diff), meaning users must combine it with an existing LLaMa base model to reconstruct the full instruction-tuned model.
Key Characteristics & Usage
- Base Model: Built upon the LLaMa architecture.
- Training Data: Fine-tuned using the GPT-4 Alpaca dataset, which consists of instructions generated by GPT-4.
- Distribution: Provided as a
model diff, requiring a recovery process with an original LLaMa model in Hugging Face format. Instructions for this process are available in the associated GitHub repository. - Input Format: Optimized for a specific input format:
\n<|user|>\nYour message here!\n<|assistant|>\n. Including a newline after<|assistant|>is crucial for optimal generation quality.
Performance Highlights
Benchmarking results from the associated paper indicate its performance across various tasks:
- MMLU (0-shot): 42.6
- MMLU (5-shot): 38.3
- GSM Direct: 6.5
- GSM CoT: 10.0
- AlpacaFarm vs Davinci-003: 57.0
- Average Score: 28.3
This model is primarily intended for research and development in instruction tuning, offering insights into the capabilities of models fine-tuned on advanced instruction datasets.