HINT-lab/llama3-8b-final-ppo-c-v0.3

TEXT GENERATIONPricing:Input $0.37 / Cached $0.074 / Output $0.38Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:8kTool Calling:SupportedPublished:Oct 12, 2024Architecture:Transformer Featherless Exclusive Cold

HINT-lab/llama3-8b-final-ppo-c-v0.3 is an 8 billion parameter language model developed by HINT-lab, fine-tuned from OpenRLHF/Llama-3-8b-sft-mixture. This model utilizes PPO-C (PPO with Calibrated Reward Calculation), an RLHF algorithm designed to mitigate verbalized overconfidence in LLMs by adjusting reward scores based on the model's expressed confidence. Its primary use case is to produce more calibrated and less overconfident responses, making it suitable for applications where reliability and accurate confidence assessment are crucial.

Loading preview...

HINT-lab/llama3-8b-final-ppo-c-v0.3: Mitigating LLM Overconfidence

HINT-lab/llama3-8b-final-ppo-c-v0.3 is an 8 billion parameter language model developed by Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. It is fine-tuned from the OpenRLHF/Llama-3-8b-sft-mixture model using a novel Reinforcement Learning from Human Feedback (RLHF) algorithm called PPO-C (PPO with Calibrated Reward Calculation). This model addresses the common issue of verbalized overconfidence in RLHF-trained Large Language Models.

Key Capabilities

  • Reduces Verbalized Overconfidence: PPO-C dynamically adjusts reward model scores during PPO training by maintaining a running average of past reward scores as a threshold. This mechanism classifies responses and modifies reward scores based on the model's expressed confidence.
  • Enhanced Reliability: By mitigating overconfidence, the model aims to provide more trustworthy and calibrated responses, which is critical in sensitive applications.
  • RLHF Innovation: Implements a unique approach to reward calibration within the RLHF framework, as detailed in their research paper.

Good for

  • Applications requiring calibrated confidence: Ideal for scenarios where an LLM's expressed confidence needs to be accurate and not overblown, such as in decision support systems, scientific research, or critical information retrieval.
  • Research into LLM reliability: Provides a strong baseline for further research into mitigating overconfidence and improving the trustworthiness of AI models.
  • Developers seeking less 'hallucinatory' confidence: Useful for those who need models that are more aware of their own limitations and express uncertainty appropriately.

For more technical details, refer to the preprint and the GitHub repository.

Finetunes
2 models