Reward-Reasoning/RRM-7B
Reward-Reasoning/RRM-7B is a 7.6 billion parameter Reward Reasoning Model (RRM) developed by Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Utilizing the Qwen2 architecture, this model frames reward modeling as a reasoning task, generating a chain-of-thought before assigning rewards. It is trained via Reward Reasoning via Reinforcement Learning to self-evolve reward reasoning capabilities and excels at enhancing accuracy in reward-guided inference and providing high-quality preference signals for LLM post-training.
Loading preview...
What is Reward-Reasoning/RRM-7B?
Reward-Reasoning/RRM-7B is a 7.6 billion parameter model that introduces the concept of Reward Reasoning Models (RRMs). Unlike traditional reward models, RRMs approach reward modeling as a reasoning task, generating a detailed chain-of-thought process before assigning a final reward. This allows the model to adaptively allocate computational resources based on the complexity of the evaluation scenario.
Key Capabilities & Training:
- Chain-of-Thought Reward Modeling: Produces explicit reasoning traces to analyze and compare candidate responses, leading to more nuanced reward assignments.
- Self-Evolving Reasoning: Trained using a novel framework called Reward Reasoning via Reinforcement Learning, which enables the model to develop sophisticated reasoning capabilities without requiring supervised reasoning trace data.
- Adaptive Compute Utilization: Can scale its test-time compute (both parallel and sequential reasoning steps) to achieve improved performance, making it efficient for various applications.
- Enhanced Accuracy: Consistently outperforms strong baseline reward models across diverse domains, including reasoning, general knowledge, and alignment with human preferences.
Use Cases:
- Reward-Guided Best-of-N Inference: Effectively guides the selection of optimal responses from multiple candidates.
- LLM Post-Training: Provides high-quality preference signals crucial for advanced LLM training techniques like DPO (Direct Preference Optimization) or RL (Reinforcement Learning).