Model Overview

The allenai/open-instruct-cot-13b is a 13 billion parameter LLaMa-based model developed by AllenAI. It has been fine-tuned on the CoT dataset, which is a subset of Flan v2, with a context length of 4096 tokens. This model is a result of research exploring instruction tuning on open resources, detailed in the paper "How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources".

Key Capabilities & Features

• Chain-of-Thought (CoT) Fine-tuning: Specialized training on the CoT dataset to improve reasoning and instruction-following abilities.
• LLaMa Base: Leverages the robust LLaMa architecture as its foundation.
• Instruction Following: Designed to respond effectively to instructions, particularly when formatted with specific user and assistant tags.
• Performance Benchmarks: Achieves an average score of 27.2 across various benchmarks including MMLU, GSM, BBH, TydiQA, and Codex-Eval, as reported in the associated research paper.

Usage and Integration

This model is distributed as a model diff, requiring an existing LLaMa model in Hugging Face format for recovery. The process involves using a provided weight_diff.py script to reconstruct the full model. Users should adhere to the specified input format (<|user|> Your message here! <|assistant|> ) for optimal generation quality, ensuring a newline after <|assistant|>. The codebase for training and evaluation is available on GitHub.