PRIME-RL/EurusPRM-Stage2

TEXT GENERATIONConcurrency Cost:1Model Size:7.6BQuant:FP8Ctx Length:32kPublished:Dec 30, 2024License:apache-2.0Architecture:Transformer0.0K Open Weights Cold

EurusPRM-Stage2 is a 7.6 billion parameter model developed by PRIME-RL, trained using Implicit PRM for process-level reward modeling. This model excels at mathematical reasoning and problem-solving by implicitly learning a Q-function from response-level labels, eliminating the need for costly step-level annotations. It achieves strong performance on benchmarks like ProcessBench and Best-of-N sampling, making it suitable for applications requiring detailed, step-by-step reasoning.

Loading preview...

EurusPRM-Stage2: Process Reinforcement through Implicit Rewards

EurusPRM-Stage2 is a 7.6 billion parameter model from the PRIME-RL collection, designed for advanced mathematical reasoning and problem-solving. It leverages a novel training methodology called Implicit PRM, which allows it to derive process-level rewards without requiring explicit step-by-step annotations. This is achieved by implicitly learning a Q-function from cheaper response-level labels, significantly reducing annotation burden.

Key Capabilities

  • Implicit Process Reward Modeling: Utilizes a log-likelihood ratio to obtain process rewards during inference, enabling fine-grained evaluation of each step in a generated response.
  • Enhanced Mathematical Reasoning: Demonstrates strong performance across various mathematical benchmarks, including MATH, AMC, AIME, OlympiadBench, and Minerva Math.
  • Efficient Training: Built upon the EurusPRM-Stage1 model and continually trained with cross-entropy (CE) loss, optimizing for memory efficiency.
  • Step-by-Step Guidance: Optimized for outputs where each reasoning step is clearly delineated (e.g., "Step K"), leading to improved performance.

Good for

  • Mathematical Problem Solving: Excels in tasks requiring detailed, multi-step mathematical reasoning.
  • Automated Grading and Feedback: Can be used to evaluate the correctness of intermediate steps in complex solutions.
  • Reinforcement Learning from Human Feedback (RLHF) without Step Labels: Offers a cost-effective approach to process-level reward modeling by using only response-level data.
  • Improving LLM Reasoning Chains: Can guide generation models to produce more logical and coherent reasoning paths.

Finetunes
2 models