promotion/Llama-3.1-8B-SafeRLHF-NBPO-stage2

TEXT GENERATIONPricing:Input $0.2 / Cached $0.028 / Output $0.32Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Sep 7, 2026License:llama3.1Architecture:Transformer Featherless Exclusive Cold

promotion/Llama-3.1-8B-SafeRLHF-NBPO-stage2 is an 8 billion parameter language model based on the Llama-3.1-8B-Instruct backbone, developed for research into Nash Bargaining Preference Optimization (NBPO) within a SafeRLHF framework. This model focuses on optimizing for helpfulness and harmlessness objectives. It represents a second stage of NBPO training, aiming to refine policy performance in safety-critical applications.

Loading preview...

Model Overview

This model, promotion/Llama-3.1-8B-SafeRLHF-NBPO-stage2, is an 8 billion parameter language model built upon the meta-llama/Llama-3.1-8B-Instruct backbone. It is a product of research into Nash Bargaining Preference Optimization (NBPO), specifically representing a second outer stage within the SafeRLHF framework.

Key Characteristics

  • Optimization Focus: Primarily optimized for helpfulness and harmlessness objectives, as part of the SafeRLHF panel.
  • Training Methodology: Utilizes a second stage of Algorithm 1 from the NBPO research, involving a regenerated pool, rejudged training matrix (44,000 cells), re-solved dual, and 300 policy refit updates.
  • Evaluation: Evaluated using independent objective-wise win rates against a common reference, judged by Llama-3.3-70B-Instruct on prompt-disjoint held-out prompts.
  • Performance Note: While accepted by the Algorithm 1 gate (held-out surpluses 0.240 / 0.148), this specific stage did not significantly improve upon the previous pi_1 policy, showing a marginal +0.003 on gated surplus and -0.008 on reported worst-objective win rate.

Intended Use

This model is primarily a research artifact demonstrating a specific iteration within the NBPO framework for SafeRLHF. It is suitable for researchers and developers interested in:

  • Understanding the practical application and iterative stages of Nash Bargaining Preference Optimization.
  • Analyzing the impact of subsequent NBPO stages on helpfulness and harmlessness objectives.
  • Exploring advanced reinforcement learning from human feedback (RLHF) techniques for safety alignment.

Finetunes
1 models