steven0226/qwen2.5-0.5b-dpo-ultrafeedback

TEXT GENERATIONConcurrent Unit Cost:1Model Size:0.5BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Jul 10, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

steven0226/qwen2.5-0.5b-dpo-ultrafeedback is a 0.5 billion parameter language model from the Qwen2.5 family, fine-tuned using Direct Preference Optimization (DPO) on the HuggingFaceH4/ultrafeedback_binarized dataset. This model is specifically aligned to generate responses that are preferred by humans, making it suitable for conversational AI and instruction-following tasks where response quality and alignment are critical. It directly optimizes for human preferences without needing a separate reward model, offering improved response quality over its base model.

Loading preview...

Model Overview

steven0226/qwen2.5-0.5b-dpo-ultrafeedback is a Qwen2.5-0.5B-Instruct model that has undergone Direct Preference Optimization (DPO). This alignment process was performed on the HuggingFaceH4/ultrafeedback_binarized dataset, specifically using 6000 pairs of preferred and rejected responses.

DPO Methodology

DPO directly trains the model using pairs of 'chosen' (better) and 'rejected' (worse) responses for the same prompt. This method eliminates the need for a separate reward model or PPO's reinforcement learning loop. It aims to maximize the implicit reward difference between chosen and rejected responses, with a beta parameter of 0.1 controlling the degree to which the policy can deviate from the reference model. A smaller beta allows for stronger alignment signals but risks degrading the original model's output quality.

Key Features & Training Details

  • Alignment: Fine-tuned for human preference alignment using DPO.
  • Merged Model: The LoRA adapter (r=16) has been merged back into the base weights (bfloat16), meaning the model can be used directly with from_pretrained without requiring PEFT for inference.
  • Training Parameters: Key hyperparameters include a learning rate of 5e-06, LoRA r/alpha of 16/32, an equivalent batch size of 16, 1 epoch, and a max_length of 1024. Training was conducted using TRL's DPOTrainer with loss_type=sigmoid.
  • Performance: Achieved a final rewards/accuracies (rolling average) of 0.639.

Use Cases

This model is particularly well-suited for applications requiring high-quality, human-aligned text generation, such as:

  • Instruction Following: Generating responses that adhere closely to user instructions.
  • Conversational AI: Producing more natural and preferred dialogue.
  • Content Generation: Creating text that is generally perceived as better quality based on human feedback.

Finetunes
1 models