allenai/tulu-v2.5-dpo-13b-prm-phase-2
allenai/tulu-v2.5-dpo-13b-prm-phase-2 is a 13 billion parameter language model developed by AllenAI, fine-tuned from Llama-2-13b-hf with a 4096-token context length. This model is part of the Tulu V2.5 series, specifically trained using DPO (Direct Preference Optimization) on PRM phase 2 data. It is designed to function as a helpful assistant, leveraging preference feedback for improved conversational capabilities.
Loading preview...
Model Overview
allenai/tulu-v2.5-dpo-13b-prm-phase-2 is a 13 billion parameter language model developed by AllenAI, building upon the Llama-2-13b-hf architecture. It is a key component of the Tulu V2.5 suite, which focuses on creating helpful assistant models through advanced alignment techniques.
Key Capabilities & Training
This model distinguishes itself through its training methodology, utilizing Direct Preference Optimization (DPO) on the PRM phase 2 dataset. This DPO training, detailed in the paper "Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback" (arXiv:2406.09279), refines the model's ability to act as a helpful assistant based on preference feedback. It was initially fine-tuned on a filtered mix of the Tulu V2 SFT dataset, which includes diverse human-created instructions and synthetic dialogues. The model is designed to follow a specific input format, requiring \n<|user|>\nYour message here!\n<|assistant|>\n for optimal generation quality.
Intended Uses & Limitations
- Intended Use: Designed as a helpful assistant, suitable for conversational AI and instruction-following tasks.
- Limitations: The model has not undergone explicit safety alignment during its RLHF phase, nor does it include in-the-loop filtering. Consequently, it may produce problematic outputs, especially when prompted to do so. Users should be aware of these potential biases and risks inherent in large language models trained on broad datasets.