allenai/open-instruct-stanford-alpaca-13b
The allenai/open-instruct-stanford-alpaca-13b is a 13 billion parameter LLaMa model, fine-tuned by AllenAI on the Stanford Alpaca dataset. This model is specifically designed for instruction-following tasks, leveraging the Alpaca dataset to enhance its ability to respond to user prompts. It serves as a model diff, requiring recovery with an original LLaMa model, and is part of research exploring instruction tuning on open resources.
Loading preview...
Overview
This model, allenai/open-instruct-stanford-alpaca-13b, is a 13 billion parameter LLaMa-based model developed by AllenAI. It has been fine-tuned using the Stanford Alpaca dataset, focusing on instruction-following capabilities. This model is released as a "model diff," meaning it requires an existing LLaMa model in Hugging Face format to be recovered into a fully functional model.
Key Characteristics & Usage
- Architecture: Based on the LLaMa 13B model, fine-tuned for instruction following.
- Training: Fine-tuned on the Stanford Alpaca dataset as part of the research detailed in the paper "How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources."
- Recovery Process: Users must have access to an original LLaMa model and use the provided
weight_diff.pyscript from theallenai/open-instructcodebase to recover the full model. - Input Format: Designed to process inputs in a specific
user/assistantturn-taking format:<|user|> Your message here! <|assistant|>
Performance Highlights
The model's performance across various benchmarks, as reported in the associated research paper, includes:
- MMLU: 45.1 (0-shot) and 47.1 (5-shot)
- GSM: 6.0 (Direct) and 8.0 (CoT)
- BBH: 35.0 (Direct) and 34.5 (CoT)
- Codex-Eval: 15.7 (Pass@1) and 27.6 (Pass@10)
- AlpacaFarm vs Davinci-003: 28.7
This model is suitable for researchers and developers interested in instruction-tuned LLaMa models, particularly those exploring the impact of the Stanford Alpaca dataset on model performance.