s1lv3rj1nx/countdown-qwen2.5-0.5b-grpo-lr3e6
The s1lv3rj1nx/countdown-qwen2.5-0.5b-grpo-lr3e6 model is a 0.5 billion parameter Qwen2.5-0.5B-Instruct variant, fine-tuned using from-scratch Group-Relative Policy Optimization (GRPO) for the Countdown number-puzzle task. This model excels at solving arithmetic puzzles by combining numbers with basic operations to hit a target, achieving a 36x accuracy improvement over its base model. It is specifically optimized for verifiable reward-based reinforcement learning (RLVR) without a reward model or critic, focusing on exact rational arithmetic solutions.
Loading preview...
Model Overview
The s1lv3rj1nx/countdown-qwen2.5-0.5b-grpo-lr3e6 is a specialized 0.5 billion parameter Qwen2.5-0.5B-Instruct model. It has been fine-tuned using a novel, from-scratch implementation of Group-Relative Policy Optimization (GRPO). The primary objective of this training was to solve the Countdown number-puzzle task, which involves combining a given set of numbers exactly once using + - * / to reach a specific target number.
Key Capabilities & Performance
- Specialized for Countdown Puzzles: The model is highly optimized for generating solutions to the Countdown number game.
- Verifiable Rewards (RLVR): It utilizes a unique RLVR approach, where rewards are directly verifiable through an exact rational arithmetic evaluator, eliminating the need for a separate reward model or critic.
- Significant Accuracy Improvement: On a public development set of 300 puzzles, this model achieved an accuracy of 12.00%, representing a 36x improvement over the base Qwen2.5-0.5B-Instruct model's 0.33% accuracy.
- Hard Puzzle Solving: Demonstrates an ability to solve "hard" 5-number puzzles with a 1.67% success rate.
Unique Training Approach
This specific checkpoint represents the best ablation found during training, utilizing a learning rate of 3e-6 over 1500 steps. A known limitation is a "reasoning collapse," where the model tends to emit bare answers without explicit thinking steps, as the shaped reward primarily scores the final answer.
Good for
- Research in RLVR: Ideal for researchers exploring reinforcement learning with verifiable rewards and direct policy optimization.
- Number Puzzle Generation/Solving: Suitable for applications requiring automated solutions to arithmetic combination puzzles like Countdown.
- Benchmarking GRPO: Can serve as a benchmark for evaluating Group-Relative Policy Optimization techniques on tasks with exact, verifiable outcomes.