narinzar/grpo-finetune-walkthrough
The narinzar/grpo-finetune-walkthrough model is a 0.5 billion parameter Qwen2.5-0.5B-Instruct variant, fine-tuned using Group Relative Policy Optimization (GRPO) on a verifiable arithmetic task. This model demonstrates a from-scratch GRPO training loop, showing a rising reward signal over 20 steps. It is designed to validate the GRPO methodology rather than achieve high performance, with a context length of 32768 tokens.
Loading preview...
Model Overview
narinzar/grpo-finetune-walkthrough is a 0.5 billion parameter model based on Qwen/Qwen2.5-0.5B-Instruct. Its primary purpose is to serve as a demonstration of a Group Relative Policy Optimization (GRPO) fine-tuning process implemented from scratch. The model was fine-tuned on a verifiable arithmetic task over 20 steps, using a single RTX 5090 GPU.
Key Characteristics
- GRPO Implementation: Showcases a custom implementation of the group-relative advantage and policy-gradient loss for RL fine-tuning.
- Arithmetic Task Focus: Trained specifically on a verifiable arithmetic task to validate the GRPO training loop.
- Demonstration Scale: Represents a minimal, 20-step run intended to prove the end-to-end functionality and reward signal generation of the GRPO method, rather than a fully converged, high-performance model.
- Context Length: Supports a context length of 32768 tokens.
Results and Purpose
During its 20-step run, the model demonstrated a doubling of the mean reward (from 0.125 to 0.250), confirming that the GRPO update successfully optimizes the policy. While the evaluation exact-match accuracy showed only a marginal increase (from 0/80 to 1/80), these results are presented as a correctness check for the training loop, not as an indicator of GRPO's full potential on this task. Achieving significant accuracy and format compliance would require substantially more training steps and prompts.