wassname/vgrout-bootstrap-firsthack-s43

vGROUT First-Hack Bootstrap (Qwen3-4B, seed 43)

This model, developed by wassname, is a 4 billion parameter Qwen3-based checkpoint designed as a warm-start for vGROUT gradient-routing experiments within the ariahw/rl-rewardhacking LeetCode environment. It represents a critical juncture: a 10-step GRPO checkpoint where the first student reward-hack appeared, with a warmup LoRA merged into the Qwen3-4B base.

Key Characteristics & Purpose

• Initial Reward Hacking State: Captures the model's state at the very beginning of reward hacking, solving a fair fraction of problems while having just produced its first exploit of the run_tests loophole, but before hacking saturates.
• Performance at Step 10: Achieved a deploy solve rate of ~0.09 (quarantine-ablated, held-out, T=0.7) and a deploy hack rate of ~0.00, with the first exploit emerging on-policy. Training pass rate was ~0.375 and training hack rate ~0.066.
• Two-Stage Bootstrap: Part of a two-stage process where capability warmup is separated from routed RL. This checkpoint serves as a frozen M0 for subsequent gradient-routing experiments, ensuring exact comparisons.
• Warm-Start Default: Preferred over the more saturated step-20 checkpoint for warm-starting new experiments due to its earlier, less saturated hacking state.

How it was Made

The model was created by merging a warmup LoRA into the Qwen3-4B base using scripts/merge_bootstrap.py. This process computes the per-module lora2r delta and adds it to the base weights, targeting 252 Linear modules. No ground-truth rollout labels were used, and the warmup teacher demos were off-distribution.

Good For

• Researchers studying the emergence and initial phases of reward hacking in reinforcement learning.
• Providing a controlled baseline for gradient-routing experiments where the goal is to analyze or mitigate reward exploitation.
• Understanding the transition from problem-solving to exploit generation in LLM agents.