promotion/Llama-3.1-8B-Panacea-baseline
Llama-3.1-8B-Panacea-baseline is an 8 billion parameter language model based on Meta's Llama-3.1-8B-Instruct architecture, fine-tuned using the Panacea method with DPO and linear scalarization. It is specifically optimized for instruction following, truthfulness, honesty, and helpfulness, leveraging the UltraFeedback dataset. This model utilizes SVD-LoRA for efficient preference injection, making it suitable for applications requiring robust alignment with human preferences.
Loading preview...
Panacea on UltraFeedback
Llama-3.1-8B-Panacea-baseline is an 8 billion parameter model derived from meta-llama/Llama-3.1-8B-Instruct. It has been fine-tuned using the Panacea method (Zhong et al., NeurIPS 2024) combined with a DPO procedure and linear scalarization. The training specifically targeted objectives such as instruction following, truthfulness, honesty, and helpfulness, utilizing the UltraFeedback dataset.
Key Technical Details
- Fine-tuning Method: Panacea with DPO and linear scalarization.
- Preference Injection: Employs SVD-LoRA with k=8 preference-agnostic singular values, injecting the preference vector as remaining singular values.
- Training Budget: 300 optimizer updates with a global batch size of 16.
- Evaluation: Performance is assessed by independent objective-wise win rates against a common reference, judged by
Llama-3.3-70B-Instructon held-out prompts.
Primary Focus
This model is designed to excel in scenarios where strong alignment with human preferences for instruction following, truthfulness, honesty, and helpfulness is critical. Its development is detailed in the Nash Bargaining Preference Optimization (NBPO) paper, where it is featured in cross-method evaluations.