promotion/Llama-3.1-8B-MaxMinRLHF-baseline
Llama-3.1-8B-MaxMinRLHF-baseline is an 8 billion parameter instruction-tuned language model developed by the creators of Nash Bargaining Preference Optimization (NBPO). It is based on the Llama-3.1-8B-Instruct backbone and fine-tuned using MaxMin-RLHF on the UltraFeedback dataset. This model is optimized for balancing multiple objectives including instruction following, truthfulness, honesty, and helpfulness, making it suitable for general-purpose conversational AI where balanced performance across various ethical and utility metrics is crucial.
Loading preview...
Llama-3.1-8B-MaxMinRLHF-baseline Overview
This model is an 8 billion parameter instruction-tuned language model built upon the meta-llama/Llama-3.1-8B-Instruct backbone. It was developed using the MaxMin-RLHF (Chakraborty et al., ICML 2024) algorithm, specifically Algorithm 1, with objectives derived from the UltraFeedback panel.
Key Capabilities and Training
The model's training focused on optimizing for a balanced performance across several critical objectives:
- Instruction following
- Truthfulness
- Honesty
- Helpfulness
Training involved 300 optimizer updates with a global batch size of 16, utilizing three alternating rounds of 100 updates on the objective with the lowest reference-standardized utility. This approach aims to improve performance across multiple, potentially conflicting, objectives simultaneously.
Evaluation and Differentiation
Evaluated using an independent objective-wise win rate against a common reference, judged by Llama-3.3-70B-Instruct on prompt-disjoint held-out prompts. This model is reported in the Nash Bargaining Preference Optimization (NBPO) paper, Table 2, as a primary cross-method evaluation. Its MaxMin-RLHF fine-tuning distinguishes it by explicitly addressing the trade-offs between different desirable AI behaviors, aiming for a more robust and ethically aligned output.