alignment-handbook/zephyr-7b-dpo-full

Zephyr-7B-DPO-Full: A Preference-Optimized Language Model

This model, alignment-handbook/zephyr-7b-dpo-full, is a 7 billion parameter language model developed by alignment-handbook. It is a Direct Preference Optimization (DPO) fine-tuned variant of the zephyr-7b-sft-full model, specifically trained on the HuggingFaceH4/ultrafeedback_binarized dataset.

Key Capabilities & Performance

The DPO fine-tuning process has optimized the model for alignment with human preferences, as indicated by its evaluation results:

• Reward Accuracy: Achieved 0.7539 on the evaluation set, demonstrating its ability to differentiate between preferred and rejected responses.
• Reward Margin: Recorded a margin of 0.9980, suggesting a clear preference for chosen responses over rejected ones.
• Loss: Final training loss of 0.5042.

Training Details

The model was trained with a learning rate of 5e-07, a total batch size of 128 (across 8 GPUs), and utilized the Adam optimizer with a cosine learning rate scheduler. The training consisted of 1 epoch, focusing on refining the model's ability to generate high-quality, preferred outputs based on the provided feedback data.

Intended Use Cases

Given its DPO fine-tuning, this model is particularly well-suited for applications where generating responses aligned with human preferences is crucial. This includes tasks such as:

• Dialogue systems: Producing more helpful and engaging conversational turns.
• Content generation: Creating text that is preferred by users based on implicit feedback.
• Instruction following: Generating outputs that better adhere to user instructions and preferences.