bhenrym14/airophin-v2-13b-PI-8k-fp16

Overview

bhenrym14/airophin-v2-13b-PI-8k-fp16 is a 13 billion parameter Llama-2-based model, developed by bhenrym14, specifically engineered for an extended context window of 8192 tokens through position interpolation (PI). It originates from the OpenAssistant/llama2-13b-orca-8k-3319 model, which was trained on a mix of Orca-chat, fanfics, and RedPajama datasets. The model then underwent a second fine-tuning phase on the merged Airoboros dataset (1.4.1 and 2.0) for two epochs.

Key Capabilities & Differentiators

• Extended Context: Achieves a usable context window of 8192 tokens, making it suitable for tasks requiring longer inputs or conversational history.
• Performance: Demonstrates competitive perplexity scores, outperforming some 33B extended context models at similar token lengths. It also shows improved MMLU performance, potentially ranking among the highest for 13B extended context models.
• Fine-tuned for Instruction Following: Benefits from fine-tuning on Airoboros datasets, known for enhancing instruction-following capabilities, including closed-context question answering, coding, and creative writing.
• QLoRA Fine-tune: Developed using a (merged) QLoRA fine-tuning approach (rank 64).

Use Cases & Prompting

This model is well-suited for applications requiring robust instruction following and extended context understanding. It supports a specific closed-context prompting format for tasks where answers must be derived strictly from provided text, minimizing hallucinations. Additionally, it excels in:

• Coding: Generating complex code based on requirements.
• Creative Writing: Producing various text formats, including resignation letters in specific styles.
• Question Answering: Handling trivia, word games, and multiple-choice questions.
• Multi-character Conversations: Maintaining distinct character personas and dialogue styles.