The allenai/open-instruct-dolly-7b is a 7 billion parameter LLaMa-based causal language model developed by AllenAI, fine-tuned on the Dolly dataset. This model is specifically instruction-tuned to follow user prompts, leveraging open resources for its training. It is designed for general instruction-following tasks, offering a base for developers to build upon with its 4096-token context length.
Loading preview...
Overview
allenai/open-instruct-dolly-7b is a 7 billion parameter LLaMa model developed by AllenAI, specifically instruction-tuned using the Dolly dataset. This model was created as part of the research detailed in the paper "How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources." It represents a significant effort in leveraging open resources for instruction tuning.
Key Capabilities
- Instruction Following: Fine-tuned to understand and respond to user instructions, making it suitable for various conversational and task-oriented applications.
- LLaMa Architecture: Built upon the LLaMa foundation, providing a robust and efficient base for language generation.
- Open-Source Training: Developed using open-source datasets and methodologies, promoting transparency and reproducibility in LLM research.
- Model Diff Distribution: Distributed as a model diff, requiring users to recover the full model from an existing LLaMa installation, which helps manage distribution size and licensing.
Performance Highlights
Based on the research paper's benchmarks, the model demonstrates:
- MMLU (0-shot/5-shot): 38.0 / 35.8
- GSM Direct/CoT: 5.0 / 7.0
- BBH Direct/CoT: 27.2 / 24.4
- Codex-Eval Pass@1/Pass@10: 11.1 / 22.1
Usage Considerations
Users need to have access to a LLaMa model in Hugging Face format to utilize this model, as it is provided as a weight difference. The model expects inputs formatted with <|user|> and <|assistant|> tags for optimal performance.