harshavaishnav/DPO
The harshavaishnav/DPO model is a 4.5 billion parameter language model, fine-tuned from Qwen/Qwen3.5-4B using LoRA Direct Preference Optimization (DPO). It was trained on the harshavaishnav/DPO_Dataset_2 with a maximum sequence length of 1024 tokens. This model is specifically optimized to align with human preferences, demonstrating improved clemscore and statscore over its base model, making it suitable for tasks requiring nuanced response generation.
Loading preview...
harshavaishnav/DPO: Preference-Optimized Qwen3.5-4B
This model is a 4.5 billion parameter LoRA DPO fine-tune of the Qwen/Qwen3.5-4B base model, developed by harshavaishnav for the LM Playschool Challenge. It leverages Direct Preference Optimization (DPO) with LoRA adapters, trained on the harshavaishnav/DPO_Dataset_2.
Key Capabilities & Performance
- Preference Alignment: Optimized to generate responses aligned with human preferences through DPO training on chosen/rejected response pairs.
- Improved Scores: Achieved a
+0.9increase in clemscore and+1.49in statscore compared to the baseline Qwen/Qwen3.5-4B model. - Efficient Fine-tuning: Utilizes LoRA (Low-Rank Adaptation) for efficient fine-tuning, requiring fewer computational resources.
- Context Length: Trained with a maximum sequence length of 1024 tokens.
Training Details
The model was trained for one epoch using bf16 mixed precision, an AdamW optimizer, and gradient checkpointing on a single NVIDIA RTX 5000 Ada Generation GPU. Key hyperparameters include a LoRA rank of 16, DPO Beta of 0.1, and a learning rate of 5e-5.
Limitations
Performance is highly dependent on the quality and diversity of the preference dataset. The model may still generate undesirable responses and its long-context reasoning beyond the trained sequence length may be limited. It also inherits potential biases from both the base model and the preference dataset.