bhaveshsoni0023/qwen2.5-1.5b-grpo-gsm8k
The bhaveshsoni0023/qwen2.5-1.5b-grpo-gsm8k model is a 1.5 billion parameter Qwen2.5-based language model, fine-tuned using GRPO (Generative Reinforcement Learning with Policy Optimization) with NVIDIA NeMo RL. This model is specifically optimized for mathematical reasoning tasks, demonstrating a significant performance improvement on the GSM8K benchmark. It achieves 73.84% pass@1 on GSM8K, a 38.8-point increase over its base model, making it highly effective for solving arithmetic word problems.
Loading preview...
Overview
This model, bhaveshsoni0023/qwen2.5-1.5b-grpo-gsm8k, is a 1.5 billion parameter variant of the Qwen2.5 architecture. It has been fine-tuned using Generative Reinforcement Learning with Policy Optimization (GRPO), leveraging NVIDIA NeMo RL v0.7.0. The primary focus of this training was to enhance its capabilities in mathematical reasoning.
Key Capabilities
- Enhanced Mathematical Reasoning: Achieves a pass@1 score of 73.84% on the GSM8K test set, a substantial improvement of 38.8 percentage points (2.1x relative) compared to the base Qwen2.5-1.5B model (35.03%).
- GRPO Training: Utilizes a GRPO training regimen over 130 steps, involving 16 prompts and 8 generations per step, totaling 2,080 samples. The reward mechanism was based on binary exact-match for answers within boxed tags.
- Efficient Training: Trained on a single H100 80GB GPU in approximately 3 hours, using DTensor v2 trainer and vLLM generation.
- Specific Prompt Format: Requires a specific prompt template for optimal performance, which includes a step-by-step thinking instruction and an answer boxing format.
When to Use This Model
- Mathematical Problem Solving: Ideal for applications requiring robust performance on arithmetic word problems, particularly those similar to the GSM8K benchmark.
- Educational Tools: Suitable for developing AI tutors or problem-solving assistants in mathematics.
- Research in RLHF: Can serve as a strong baseline or component for further research into reinforcement learning from programmatic feedback, as it learned without human-written solutions.