yuhan-nlp/verl-grpo-medium-qwen3-4b-step129

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:4BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Aug 9, 2026License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

The yuhan-nlp/verl-grpo-medium-qwen3-4b-step129 model is a 4 billion parameter Qwen3-based language model, fine-tuned by yuhan-nlp using GRPO for the CollabLLM medium document-writing task. It specializes in generating text for collaborative document creation, with a focus on optimizing for BLEU scores in this specific context. This model was trained without an SFT warm start and is provided as a merged Hugging Face checkpoint.

Loading preview...

Model Overview

This model, yuhan-nlp/verl-grpo-medium-qwen3-4b-step129, is a 4 billion parameter variant of the Qwen3 architecture. It was fine-tuned by yuhan-nlp using the GRPO (Generative Reinforcement Learning with Policy Optimization) method specifically for the CollabLLM medium document-writing task. Unlike many models, it was trained without an initial Supervised Fine-Tuning (SFT) warm start, directly applying GRPO.

Key Characteristics & Performance

This particular checkpoint, step 129, represents the final stage of a training run. While it achieves a strong BLEU score of 0.4825 on the CollabLLM medium benchmark, it shows a regression in interactivity compared to an earlier step 100 checkpoint. The training process optimized for a reward function combining BLEU score, interactivity, and token amount. Benchmarks were conducted with a Qwen3-4B assistant (this model), a Qwen3.5-9B user simulator, and a Qwen3.5-27B judge.

Important Considerations

  • Training Regression: Performance peaked at step 100 for BLEU and interactivity, with step 129 showing a drop in interactivity despite a similar BLEU score.
  • Metric Interpretation: The model's BLEU score is notable, but a simple "proact prompting" approach without training still yields a higher MR (Mean Reward) score, primarily due to token reduction.
  • Benchmark Caveats: The benchmark has a significant overlap (35/100) between train/eval prompts, leading to potential label leakage through the reward mechanism.

Use Cases

This model is primarily suited for research and development in:

  • Collaborative Document Generation: Specifically within the context of the CollabLLM medium task.
  • Reinforcement Learning for Language Models: As an example of GRPO application without SFT.
  • Comparative Analysis: For studying the effects of different training steps and reward functions on model performance metrics like BLEU and interactivity.

Finetunes
2 models