dipta007/dagger-4B_GRPO
The dipta007/dagger-4B_GRPO is a 4.3 billion parameter language model based on Gemma-3-4B-Instruct, trained using the GRPO method directly from the base model without SFT initialization. This model serves as an ablation study to demonstrate the critical importance of SFT initialization for smaller models in generating computational graphs for mathematical reasoning. It exhibits significantly lower accuracy and higher failure rates compared to models that include SFT initialization, making it unsuitable for production use. Its primary purpose is for research into the effects of SFT on GRPO training for smaller LLMs.
Loading preview...
Model Overview
The dipta007/dagger-4B_GRPO is a 4.3 billion parameter model derived from the Gemma-3-4B-Instruct base model. It was trained using the Graph Reasoning Policy Optimization (GRPO) method without an initial Supervised Fine-Tuning (SFT) phase. This model is specifically designed as an ablation study to highlight the crucial role of SFT initialization, particularly for smaller language models, in tasks requiring computational graph generation for mathematical problem-solving.
Key Findings and Performance
This model's performance underscores the necessity of SFT for effective GRPO training in the 4B parameter range:
- Significantly Lower Accuracy: On datasets like MGSM and MSVAMP,
dagger-4B_GRPOachieves scores of 29.2% and 57.1% respectively, which drop further to 13.1% and 29.3% when distractors are introduced. - Impact of SFT: Comparative analysis shows that models initialized with SFT before GRPO (e.g.,
dagger-4B_SFT_GRPO) achieve substantially higher scores, demonstrating improvements of +25.6 points on MGSM and +18.3 points on MGSM (+Distractor). - Struggles with Graph Generation: Without SFT, the model struggles to learn reliable graph generation, often producing malformed JSON outputs or incorrect structures.
When to Use This Model
This model is not recommended for production environments due to its low accuracy and high failure rate. Its intended use cases are:
- Ablation Studies: To understand the specific contribution and necessity of SFT initialization in the training pipeline for smaller models.
- Research: For investigating the capacity requirements and limitations of GRPO-only training approaches.
For practical applications requiring robust mathematical reasoning and computational graph generation, the dagger-4B_SFT_GRPO model is recommended, as it incorporates SFT initialization and offers significantly improved performance.