allenai/tulu-v2.5-dpo-13b-alpacafarm-gpt4-pref
The allenai/tulu-v2.5-dpo-13b-alpacafarm-gpt4-pref model is a 13 billion parameter language model from the Tulu V2.5 series, developed by AllenAI. It is fine-tuned from Llama-2-13b-hf using DPO (Direct Preference Optimization) on the AlpacaFarm GPT-4 preferences dataset. This model is designed to function as a helpful assistant, leveraging preference-based learning to improve response quality within its 4096-token context window.
Loading preview...
Overview
allenai/tulu-v2.5-dpo-13b-alpacafarm-gpt4-pref is a 13 billion parameter language model from the Tulu V2.5 series, developed by AllenAI. It is fine-tuned from meta-llama/Llama-2-13b-hf using Direct Preference Optimization (DPO) on the AlpacaFarm GPT-4 preferences dataset. This model is part of a suite of RLHF-tuned chat models, aiming to act as helpful assistants.
Key Capabilities
- Assistant-like Behavior: Trained to generate responses characteristic of a helpful assistant.
- Preference-Based Learning: Utilizes DPO with the AlpacaFarm GPT-4 preferences dataset for alignment.
- Instruction Following: Initially fine-tuned on a diverse mix of human-created instructions and synthetic dialogues from the Tulu V2 mix dataset.
- Standard Input Format: Designed to work with a specific chat template:
<|user|> Your message here! <|assistant|>.
Training Details
This model was aligned using a Jax DPO trainer built on EasyLM. Key hyperparameters for DPO training included a learning rate of 5e-07, a total train batch size of 32, and 3.0 epochs. The model's development is detailed in the paper "Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback" arXiv:2406.09279.
Limitations
As with other Tulu models, this version has not undergone extensive alignment for safety within the RLHF phase, nor does it include in-the-loop filtering. Consequently, it may produce problematic outputs, especially when explicitly prompted to do so.