s1lv3rj1nx/countdown-qwen2.5-0.5b-grpo-dense
The s1lv3rj1nx/countdown-qwen2.5-0.5b-grpo-dense model is a 0.5 billion parameter Qwen2.5-Instruct variant, fine-tuned using a from-scratch GRPO (Grouped Reinforcement Learning with Policy Optimization) method. It is specifically optimized for the Countdown number-puzzle task, utilizing a dense closeness-shaped reward function for improved performance. This model excels at solving arithmetic puzzles by combining numbers to reach a target, demonstrating a 44x accuracy improvement over its base model on this specific task.
Loading preview...
Model Overview
s1lv3rj1nx/countdown-qwen2.5-0.5b-grpo-dense is a specialized 0.5 billion parameter Qwen2.5-Instruct model, fine-tuned using a novel from-scratch GRPO (Grouped Reinforcement Learning with Policy Optimization) approach. Its primary focus is solving the Countdown number-puzzle task, where the goal is to combine given numbers using basic arithmetic operations (+ - * /) to precisely hit a target value.
Key Differentiators & Capabilities
- Specialized for Countdown Puzzles: This model is explicitly trained for the Countdown game, demonstrating significant performance gains on this specific task.
- Dense Reward Shaping: Unlike models using a simple step-function reward, this variant employs a dense closeness-shaped reward. This means it receives graded feedback for near-misses, allowing GRPO to learn more effectively from attempts that are close to the target, rather than just exact solutions.
- Verifiable Rewards: The rewards are based on exact rational arithmetic and a whitelisted-AST evaluator, ensuring high verifiability without needing a separate reward model or critic.
- Performance Improvement: It achieves a 14.67% accuracy on the
dev_publicdataset for Countdown puzzles, representing a 44x improvement over the base Qwen2.5-0.5B-Instruct model (which scored 0.33%). This gain is particularly noticeable in easy and medium difficulty puzzles.
Intended Use Cases
This model is ideal for applications requiring symbolic reasoning and arithmetic problem-solving within a constrained environment, specifically for tasks similar to the Countdown game. It serves as an excellent example of how targeted RL fine-tuning with sophisticated reward shaping can dramatically improve performance on specific, verifiable tasks. It was developed as part of the RLVR Arena capstone for the RL in Production Bootcamp.