dipta007/dagger-12B_GRPO
The dipta007/dagger-12B_GRPO model is a 12-billion parameter language model based on Gemma-3-12B, trained using Group Relative Policy Optimization (GRPO) directly from the base model without Supervised Fine-Tuning (SFT) initialization. It learns to generate computational graphs for mathematical problems by leveraging executable reward signals, including format, execution, and correctness. This model serves as an ablation study to demonstrate the capabilities of GRPO alone in structured generation and highlights the impact of SFT initialization on distractor robustness.
Loading preview...
DAGGER-12B-GRPO: GRPO-Only Computational Graph Generation
DAGGER-12B-GRPO is a 12-billion parameter model derived from the Gemma-3-12B base model. Its unique characteristic is that it was trained exclusively with Group Relative Policy Optimization (GRPO), completely bypassing the traditional Supervised Fine-Tuning (SFT) initialization phase. This makes it a crucial model for understanding the direct capabilities of GRPO in learning complex structured outputs like computational graphs for math problems.
Key Capabilities and Training
- GRPO-Only Training: Directly trained from the base Gemma-3-12B model using GRPO, without any SFT. This demonstrates GRPO's ability to learn computational graph generation from scratch.
- Executable Reward Signal: The model learns by optimizing for rewards based on the generated output's format validity (JSON), successful execution, and correctness of the final answer.
- Computational Graph Generation: Designed to solve mathematical problems by outputting a structured JSON representation of a computational graph, including operations, arguments, and distractor identification.
Performance and Ablation Insights
While capable, its performance on datasets like MGSM and MSVAMP (with distractors) is lower compared to models that include SFT initialization. For instance, on MGSM (+Distractor), it scores 48.4, whereas an SFT-initialized GRPO model achieves 64.0. This highlights a key finding: SFT initialization provides significant scaffolding, improving distractor robustness by 7-16 points.
When to Use This Model
This model is particularly valuable for:
- Ablation Studies: Investigating the isolated contribution of GRPO in policy optimization for structured generation.
- Research: Exploring scenarios where SFT data might be unavailable, focusing on learning directly from executable rewards.
- Understanding Policy Optimization: Studying how GRPO learns to generate complex, executable structures without prior supervised guidance.