promotion/Llama-3.1-8B-SafeRLHF-MaxMinRLHF-baseline
Llama-3.1-8B-SafeRLHF-MaxMinRLHF-baseline is an 8 billion parameter language model based on Meta's Llama-3.1-8B-Instruct, fine-tuned using the MaxMin-RLHF algorithm. This model optimizes for both helpfulness and harmlessness objectives, leveraging the SafeRLHF panel for preference alignment. It is designed to balance multiple objectives, making it suitable for applications requiring robust safety and utility.
Loading preview...
Model Overview
This model, Llama-3.1-8B-SafeRLHF-MaxMinRLHF-baseline, is an 8 billion parameter language model built upon the meta-llama/Llama-3.1-8B-Instruct backbone. It has been fine-tuned using the MaxMin-RLHF algorithm (Chakraborty et al., ICML 2024), specifically Algorithm 1, which iteratively optimizes for objectives with the lowest reference-standardized utility.
Key Characteristics
- Optimization Strategy: Employs MaxMin-RLHF with three alternating rounds of 100 updates each, focusing on objectives with the lowest utility.
- Preference Panel: Utilizes the SafeRLHF panel to guide the alignment process.
- Objectives: Explicitly trained to balance two primary objectives: helpfulness and harmlessness.
- Training Budget: Underwent 300 optimizer updates with a global batch size of 16.
- Evaluation: Performance is assessed by independent objective-wise win rates against a common reference, judged by
Llama-3.3-70B-Instructon held-out prompts.
Use Cases
This model is particularly well-suited for applications where balancing multiple, potentially conflicting, objectives like helpfulness and harmlessness is critical. Its MaxMin-RLHF training aims to ensure robust performance across these dimensions, making it a strong candidate for general-purpose conversational AI and safety-sensitive deployments.