juwon1105/RLCR-phi4mini-bigmathdigits5000

Overview

This model, juwon1105/RLCR-phi4mini-bigmathdigits5000, is a 3.8 billion parameter Phi-4-mini-instruct variant. Its key differentiator is the application of Reinforcement Learning with Calibration Rewards (RLCR), a novel training approach that adds a Brier-score-based reward for verbalized confidence to the standard correctness reward. This method, detailed in the paper "Confidence as Curriculum: Reinforcement Learning for Joint Reasoning and Calibration" (Damani et al. 2026), aims to jointly optimize both reasoning capabilities and the model's ability to accurately express its uncertainty.

Key Capabilities

• Joint Reasoning and Calibration: Trained to not only provide correct answers but also to accurately verbalize its confidence in those answers.
• Mathematical Reasoning: Specifically fine-tuned on the mehuldamani/big-math-digits dataset, comprising 5,000 training samples, making it suitable for numerical and mathematical reasoning tasks.
• Uncertainty Quantification: Incorporates a reward mechanism for calibration, which helps the model better understand and communicate its own uncertainty.

Training Details

The model was trained using the GRPO algorithm for 1 epoch with a per-device batch size of 1. It utilized AdamW (8-bit) optimizer and a linear learning rate schedule. Rollouts were performed using vLLM with a temperature of 0.7 and 32 generations per prompt. The training was conducted on a single RTX 3090 GPU.

Good For

• Applications requiring not just correct answers but also a reliable measure of the model's confidence.
• Mathematical problem-solving where understanding uncertainty is crucial.
• Research into improving LLM calibration and reasoning abilities.