The allenai/open-instruct-oasst1-7b is a 7 billion parameter LLaMa-based model developed by AllenAI, fine-tuned specifically on the Open Assistant dataset. This model is designed for instruction-following tasks, leveraging the Open Assistant dataset to enhance its conversational and directive capabilities. It is released as a model diff, requiring recovery with an existing LLaMa base model, and is suitable for research into instruction tuning on open resources.
Loading preview...
Model Overview
The allenai/open-instruct-oasst1-7b is a 7 billion parameter LLaMa-based model developed by AllenAI. It has been fine-tuned using the Open Assistant dataset, focusing on instruction-following capabilities. This model is released as a "model diff," meaning users need to recover the full model by applying this diff to an existing LLaMa base model in Hugging Face format.
Key Capabilities & Features
- Instruction Following: Optimized for understanding and executing instructions based on the diverse Open Assistant dataset.
- LLaMa Architecture: Built upon the LLaMa foundation, providing a robust base for language generation.
- Research Focus: Developed as part of the research paper "How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources" (arXiv:2306.04751).
- Specific Input Format: Requires a
<|user|>and<|assistant|>turn-based format for optimal performance, with a crucial newline after<|assistant|>. The model has a context length of 4096 tokens.
Performance Highlights
Evaluated across various benchmarks, the model achieved an average score of 23.8. Notable scores include 32.9 on MMLU 0-shot, 29.5 on BBH CoT, and 47.8 in AlpacaFarm vs Davinci-003, indicating its general instruction-following and reasoning abilities.
Usage Considerations
- Model Diff: Requires a LLaMa base model and a recovery script (
weight_diff.py) to reconstruct the full model. - Input Formatting: Adhering to the specified input format is critical for generation quality.