princeton-nlp/gemma-2-9b-it-DPO
The princeton-nlp/gemma-2-9b-it-DPO is a 9 billion parameter causal language model developed by Yu Meng, Mengzhou Xia, and Danqi Chen from Princeton NLP. It is fine-tuned from google/gemma-2-9b-it using the DPO (Direct Preference Optimization) objective on the gemma2-ultrafeedback-armorm dataset. This model is optimized for enhanced response quality and alignment through preference optimization, demonstrating improved performance on benchmarks like AE2 LC and AH compared to its base model.
Loading preview...
Model Overview
This model, princeton-nlp/gemma-2-9b-it-DPO, is a 9 billion parameter causal language model developed by Yu Meng, Mengzhou Xia, and Danqi Chen at Princeton NLP. It is a fine-tuned version of google/gemma-2-9b-it, specifically optimized using the Direct Preference Optimization (DPO) objective. The training utilized the princeton-nlp/gemma2-ultrafeedback-armorm dataset, aiming to align the model's outputs with human preferences.
Key Characteristics
- Preference Optimized: Fine-tuned with the DPO objective, which is designed to improve the quality and alignment of generated text based on preference datasets.
- Enhanced Performance: Demonstrates notable improvements over its base model,
google/gemma-2-9b-it, particularly in metrics like AE2 LC (67.8 vs 51.1) and AH (58.9 vs 40.8), indicating better logical consistency and helpfulness. - Efficient Training: The fine-tuning process was completed in approximately 150 minutes using 8xH100 GPUs, highlighting an efficient optimization workflow.
Use Cases
This model is well-suited for applications requiring high-quality, aligned text generation, especially where preference optimization is crucial for user satisfaction. Its improved performance on specific benchmarks suggests its utility in tasks demanding logical coherence and helpful responses.