allenai/open-instruct-self-instruct-7b
The allenai/open-instruct-self-instruct-7b is a 7 billion parameter LLaMa-based model developed by AllenAI, fine-tuned on the Self-instruct dataset. This model is a diff, requiring recovery with an original LLaMa model. It is designed for general instruction-following tasks, demonstrating performance across various benchmarks including MMLU, GSM, BBH, and Codex-Eval.
Loading preview...
Model Overview
This model, allenai/open-instruct-self-instruct-7b, is a 7 billion parameter LLaMa-based instruction-tuned model developed by AllenAI. It was fine-tuned using the Self-instruct dataset as part of the research presented in the paper "How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources". Notably, this release is a model diff, meaning users must recover the full model by applying this diff to an existing LLaMa model in Hugging Face format.
Key Capabilities & Performance
The model is designed for general instruction-following and has been evaluated across a range of benchmarks:
- MMLU (0-shot/5-shot): Achieves 35.7% and 33.2% respectively.
- GSM (Direct/CoT): Scores 4.0% and 6.5%.
- BBH (Direct/CoT): Reaches 29.9% and 29.2%.
- TydiQA (Gold-Passage/Closed-book): Shows 35.4% and 8.7%.
- Codex-Eval (Pass@1/Pass@10): Attains 6.2% and 12.1%.
- AlpacaFarm vs Davinci-003: Scores 7.5%.
Its average performance across these benchmarks is 18.0%. The model expects a specific input format: <|user|> Your message here! <|assistant|> for optimal generation quality.
Usage Notes
To use this model, users need access to an original LLaMa model. The provided weight_diff.py script from the open-instruct GitHub repository is used to recover the full model from the diff. This process requires a decent amount of RAM, especially for larger models.