promotion/Llama-3.1-8B-FixedReferenceNBPO
Llama-3.1-8B-FixedReferenceNBPO is an 8 billion parameter model based on Meta's Llama-3.1-8B-Instruct, fine-tuned using a Fixed Reference Nash Bargaining Preference Optimization (NBPO) approach. It is specifically optimized for instruction following, truthfulness, honesty, and helpfulness, utilizing the UltraFeedback panel for objective weighting. This model serves as a direct comparison point for NBPO methods, demonstrating performance against a common reference policy.
Loading preview...
Model Overview
promotion/Llama-3.1-8B-FixedReferenceNBPO is an 8 billion parameter language model derived from meta-llama/Llama-3.1-8B-Instruct. This model was developed as part of research into Nash Bargaining Preference Optimization (NBPO), specifically to provide a fixed-reference control for comparative evaluation.
Key Characteristics
- Base Model: Built upon
meta-llama/Llama-3.1-8B-Instruct. - Optimization Method: Utilizes a fixed-reference approach within the Nash Bargaining Preference Optimization framework, ensuring the same response pool, temperatures, disagreement point, and optimizer budget as NBPO, but with a different objective weighting strategy.
- Training Objectives: Focused on improving performance across four key areas:
- Instruction following
- Truthfulness
- Honesty
- Helpfulness
- Training Data: Fine-tuned using the UltraFeedback panel.
- Training Budget: Underwent 300 optimizer updates with a global batch size of 16.
Evaluation and Purpose
This model's primary purpose is to serve as a controlled baseline for evaluating the effectiveness of NBPO. It is evaluated by measuring its independent objective-wise win rate against the common reference policy, with judgments provided by Llama-3.3-70B-Instruct on held-out, prompt-disjoint data. This allows for direct comparison of how different objective weighting strategies impact model performance on critical alignment metrics.