steven0226/qwen2.5-3b-grpo-gsm8k

Model Overview

This model, steven0226/qwen2.5-3b-grpo-gsm8k, is a 3.1 billion parameter Qwen2.5-3B-Instruct model that has been fine-tuned for enhanced mathematical reasoning. It utilizes a novel training approach combining Group Relative Policy Optimization (GRPO) with Verifiable Rewards (RLVR) on the GSM8K dataset.

Key Training Methodology

The GRPO method involves comparing 8 generated responses within a group to calculate relative advantages, replacing traditional PPO value models. The reward system is based on programmatically verifiable rules rather than a separate reward model, ensuring robustness against reward hacking. Rewards are assigned for:

• Correctness of the answer (2.0 points)
• Strict adherence to <reasoning>...</reasoning><answer>...</answer> format (0.5 points)
• Partial format compliance (0.5 points)
• Purely numerical answers in the <answer> tag (0.5 points)

This approach led to a significant increase in strict format adherence, from 19.5% to 90%, and a clean reward climb during training. Interestingly, the model learned to be more efficient, generating shorter completions for easier problems while still allocating more tokens for complex ones, rather than simply increasing output length.

Performance Highlights

• Enhanced Mathematical Reasoning: Specifically trained on the openai/gsm8k dataset's train split (7,473 problems) to improve arithmetic problem-solving.
• Robust Format Adherence: Demonstrates strong compliance with specified output formats for reasoning and answers.
• Efficient Reasoning: The model adapts its output length based on problem difficulty, providing concise yet comprehensive reasoning.

Use Cases

This model is particularly well-suited for applications requiring:

• Accurate mathematical problem-solving.
• Structured and verifiable reasoning outputs.
• Educational tools that need to generate step-by-step solutions for arithmetic problems.