Tulu V2 DPO 13B: A DPO-Aligned Llama 2 Assistant

allenai/tulu-2-dpo-13b is a 13 billion parameter language model developed by AllenAI, built upon the Llama 2 architecture. It is specifically fine-tuned using Direct Preference Optimization (DPO) on a diverse mix of publicly available, synthetic, and human-created datasets, aiming to function as a highly helpful assistant.

Key Capabilities & Features

• DPO Alignment: Utilizes Direct Preference Optimization for enhanced conversational quality and alignment, drawing inspiration from the Zephyr Beta model's DPO recipe.
• Strong Performance: Achieves a 7.00 MT-Bench score and an 89.5% AlpacaEval win rate, positioning it as a robust alternative to Llama 2 13B Chat.
• Diverse Training Data: Fine-tuned on a filtered version of the Tulu V2 mix dataset, which includes human-created instructions and synthetic dialogues, further aligned with the openbmb/UltraFeedback dataset (64k prompts ranked by GPT-4).
• Instruction Following: Optimized for chat-based interactions, requiring a specific input format (<|user|>\nYour message here!\n<|assistant|>\n).

Intended Use Cases

• Chatbot Development: Ideal for creating helpful and aligned conversational AI agents.
• Assistant Applications: Suitable for tasks requiring detailed instruction following and natural language understanding.
• Research & Development: Provides a strong base for further experimentation with DPO-aligned models.

Limitations

• The model has not undergone extensive safety alignment beyond the DPO phase, meaning it may produce problematic outputs, especially when explicitly prompted to do so.
• The exact composition of the base Llama 2 training corpus is unknown, which may carry inherent biases.