juwon1105/RLCR-llama32-3B-bigmathdigits5000
The juwon1105/RLCR-llama32-3B-bigmathdigits5000 model is a 3.2 billion parameter Llama-3.2-3B-Instruct variant fine-tuned using Reinforcement Learning with Calibration Rewards (RLCR). This method incorporates a Brier-score-based reward for verbalized confidence alongside correctness, optimizing for both reasoning ability and calibration. It was trained on the Big-Math-digits dataset, making it suitable for tasks requiring numerical reasoning and accurate uncertainty estimation.
Loading preview...
Model Overview
The juwon1105/RLCR-llama32-3B-bigmathdigits5000 is a 3.2 billion parameter language model based on the Llama-3.2-3B-Instruct architecture. Its key differentiator is the application of Reinforcement Learning with Calibration Rewards (RLCR), a novel training approach that jointly optimizes for reasoning correctness and the model's ability to accurately express its confidence (calibration).
Key Capabilities and Training
This model was trained on the mehuldamani/big-math-digits dataset, comprising 5,000 training samples, specifically designed for numerical reasoning tasks. The RLCR method adds a Brier-score-based reward for verbalized confidence to the standard correctness reward, encouraging the model to not only provide correct answers but also to be well-calibrated in its uncertainty estimates. Training utilized the GRPO algorithm for one epoch, with specific LoRA configurations (rank 16, alpha 32) targeting q_proj and v_proj modules.
Performance Metrics
Evaluated on a held-out set, the model achieved an accuracy of 0.287. Key calibration metrics include an Expected Calibration Error (ECE) of 0.185, a Predicted Calibration Error (PCE) of 0.171, and a Brier score of 0.242, with an AUROC of 0.583. These metrics reflect its performance in both task accuracy and the reliability of its confidence predictions.
Use Cases
This model is particularly well-suited for applications where not only the correct answer is important, but also a reliable indication of the model's confidence in that answer. This includes tasks involving numerical reasoning, mathematical problem-solving, or any scenario where understanding the model's uncertainty is critical for downstream decision-making.