promotion/Llama-3.1-8B-RewardedSoups-baseline

Model Overview

Llama-3.1-8B-RewardedSoups-baseline is an 8 billion parameter language model built upon the meta-llama/Llama-3.1-8B-Instruct backbone. It incorporates the Rewarded Soups method, a technique where multiple expert models, each trained for a specific objective, are linearly interpolated from a shared initialization. The interpolation weights are determined using validation prompts.

Key Capabilities & Training

This model was developed with a focus on improving performance across several critical objectives, as evaluated on the UltraFeedback panel:

• Instruction Following: Adhering to given directives.
• Truthfulness: Generating factually accurate responses.
• Honesty: Providing sincere and unbiased information.
• Helpfulness: Delivering useful and relevant assistance.

The training involved 300 optimizer updates with a global batch size of 16. Its development is reported in the Nash Bargaining Preference Optimization (NBPO) paper, specifically in Table 2 for cross-method evaluation.

Evaluation Protocol

The model's performance is assessed by an independent objective-wise win rate against a common reference model. This evaluation uses Llama-3.3-70B-Instruct as a judge on prompt-disjoint held-out prompts, considering both presentation orders to ensure robust results.

Good For

• Applications requiring a balance of instruction following, truthfulness, honesty, and helpfulness.
• Scenarios where a model needs to perform well across multiple distinct ethical and performance objectives simultaneously.
• Developers interested in models optimized using advanced preference optimization techniques like Rewarded Soups.