laion/ablation-pymethods2test-seqmean-arm0-30-8B
The laion/ablation-pymethods2test-seqmean-arm0-30-8B is an 8 billion parameter language model, a SkyRL GRPO checkpoint from an ablation study focusing on policy loss reduction. It utilizes a sequence-mean reduction with the RLOO-n advantage estimator, contrasting with token-mean reduction methods. This model is based on a Qwen3-8B SFT model and was trained on the exp_rpt_pymethods2test-large dataset, making it suitable for research into reinforcement learning optimization techniques for language models.
Loading preview...
Model Overview
This model, laion/ablation-pymethods2test-seqmean-arm0-30-8B, is an 8 billion parameter checkpoint derived from a reinforcement learning (RL) ablation study. Specifically, it represents a SkyRL GRPO checkpoint from the sequence-mean / RLOO-n (arm0) ablation, which investigates different policy loss reduction strategies.
Key Characteristics
- Base Model: Built upon
laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink, which is a Qwen3-8B SFT (Supervised Fine-Tuning) model. - Training Methodology: Employs SkyRL GRPO with a
loss_reduction=sequence_meanandadvantage_estimator=rloo_n(RLOO-n) for its policy loss, a key differentiator from models using token-mean reduction. - Training Data: Trained on the
DCAgent/exp_rpt_pymethods2test-largedataset. - Checkpoint: This is
global_step_30from the training run, following an earlier checkpoint at step 15.
Research Focus
This model is primarily a research artifact, designed to explore the impact of different policy loss reduction techniques in reinforcement learning for language models. Its training traces, including Daytona/Harbor rollouts, are available as a companion dataset for further analysis, providing insights into the policy's behavior during training.