lxazjk/qwen2.5-1.5b-24game-grpo
The lxazjk/qwen2.5-1.5b-24game-grpo model is a 1.5 billion parameter Qwen2.5-Instruct variant, fine-tuned using GRPO (Gradient-based Reward Policy Optimization) and RLVR (Reinforcement Learning from Verifier Rewards) specifically for solving the 24 game and Countdown. This model excels at generating arithmetic expressions to reach a target number, leveraging a deterministic program verifier for reward signals rather than a neural reward model. Its primary strength lies in its ability to produce valid solutions through extensive sampling and subsequent verification, rather than high one-shot accuracy.
Loading preview...
Model Overview
The lxazjk/qwen2.5-1.5b-24game-grpo is a 1.5 billion parameter model based on Qwen/Qwen2.5-1.5B-Instruct, specifically trained to solve the 24 game and Countdown. It utilizes GRPO (Gradient-based Reward Policy Optimization) with rewards derived from a deterministic program verifier (RLVR), eliminating the need for a neural reward model. The model's training involved an SFT warmup followed by GRPO, focusing on generating arithmetic expressions in a specific <think><answer> format.
Key Capabilities & Usage
- 24 Game Solver: Generates arithmetic expressions using four given numbers to reach 24, ensuring each number is used exactly once.
- Countdown Game: Demonstrates transfer capability to the Countdown game, solving for arbitrary targets.
- Verifier-Guided Sampling: This model is designed as a sampler, not a one-shot solver. Its effectiveness comes from generating multiple candidate solutions (e.g., 16 or 128 candidates) and then using a program verifier to select the first correct expression. This approach achieves high solve rates (e.g., 74/100 for hardest problems with best-of-128 sampling).
- Format Adherence: Reliably adheres to the specified output format and number-usage constraints, as indicated by high diagnostic scores (
r1_format = 1.00,numbers_ok = 0.99).
Limitations
- Not a One-Shot Solver: The model's greedy/pass@1 accuracy is 0/100 for hard problems; high performance requires substantial test-time sampling.
- Evaluation Scope: While evaluated on hard problems, the test set has some overlap with training data, so it's not a strict out-of-distribution generalization test.