The allenai/open-instruct-dolly-13b is a 13 billion parameter LLaMa model developed by AllenAI, fine-tuned on the Dolly dataset. This model is a 'diff' that requires an existing LLaMa model for recovery and usage. It is designed for instruction-following tasks, leveraging the Dolly dataset for its capabilities, and was part of research exploring instruction tuning on open resources.
Loading preview...
Model Overview
The allenai/open-instruct-dolly-13b is a 13 billion parameter LLaMa model, developed by AllenAI, that has been instruction-tuned using the Dolly dataset. This model is released as a 'model diff', meaning it requires an existing LLaMa model to be recovered and used. It was created as part of the research presented in the paper "How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources" (arXiv:2306.04751).
Key Capabilities & Usage
- Instruction Following: Fine-tuned specifically for instruction-following tasks based on the Dolly dataset.
- Model Recovery: Users must first have access to a LLaMa model in Hugging Face format and then use the provided
weight_diff.pyscript to recover the full model. - Specific Input Format: Requires a precise input format:
<|user|> Your message here! <|assistant|>for optimal generation quality, emphasizing the newline after<|assistant|>.
Performance Highlights
Evaluated across various benchmarks, the model achieved an average score of 25.5. Notable scores include 45.3 on MMLU 0-shot, 17.0 on GSM CoT, and 31.4 on BBH Direct. The codebase for training and evaluation is available on GitHub.