promotion/UW1-DPO-soft
UW1-DPO-soft is a 7.6 billion parameter language model fine-tuned from Qwen/Qwen2.5-7B-Instruct, featuring a 32768-token context length. It is one of five models trained using different loss functions under a unified protocol, specifically employing Direct Preference Optimization (DPO) with soft labels and a beta_DPO of 0.1. This model is designed for research into preference optimization techniques, offering insights into the impact of DPO variants on model performance across various benchmarks.
Loading preview...
UW1-DPO-soft: A DPO Variant for Preference Optimization Research
UW1-DPO-soft is a 7.6 billion parameter language model, fine-tuned from the Qwen/Qwen2.5-7B-Instruct base model. It is part of a research series exploring different preference optimization losses, specifically implementing Direct Preference Optimization (DPO) with soft labels and a beta_DPO of 0.1. This model, alongside its four counterparts, was trained under a consistent protocol, allowing for direct comparison of the impact of various loss functions on model behavior and performance.
Key Characteristics & Training:
- Base Model: Fine-tuned from
Qwen/Qwen2.5-7B-Instruct. - Optimization Method: Utilizes DPO with soft labels and a
beta_DPOof 0.1. - Training Data: Trained on 319 unique prompts from the WildChecklists panel, generating 8,932 learner pairs over 140 optimizer steps.
- Context Length: Supports a 32768-token context window.
Research Context & Limitations:
This model is primarily a research artifact, designed to investigate the effects of different preference optimization strategies. It's important to note the following limitations:
- Prompt Set: The training prompt set is a filtered subset, not a random sample, focusing on prompts with a strictly positive bargaining surplus.
- Single Seed: All models in this series were trained with a single policy seed (42), meaning observed differences cannot be definitively separated from policy-seed variance.
- Local Judges: Performance metrics were evaluated using local open-weight judges, not standard API-based benchmarks (e.g., GPT-4 for Arena-Hard, AlpacaEval, MT-Bench). Therefore, these scores are comparable across the five arms but are not directly leaderboard-comparable.
Good for:
- Comparative Research: Ideal for researchers studying the impact of different DPO variants and preference optimization techniques.
- Understanding DPO: Provides a concrete example of DPO implementation with soft labels for analysis.
- Experimental Prototyping: Suitable for exploring how specific fine-tuning methodologies affect model responses and alignment.