laion/ablation-pymethods2test-seqmean-arm0-30-8B
laion/ablation-pymethods2test-seqmean-arm0-30-8B is an 8 billion parameter language model checkpoint from an RL ablation study, based on a Qwen3-8B SFT model. It was trained using SkyRL GRPO with a sequence-mean loss reduction and RLOO-n advantage estimator, differing from token-mean reductions. This model is part of a study exploring policy loss configurations for improved reinforcement learning from human feedback. It is specifically a step-30 checkpoint from a run focused on the `pymethods2test` dataset.
Loading preview...
Model Overview
laion/ablation-pymethods2test-seqmean-arm0-30-8B is an 8 billion parameter model checkpoint developed by laion. It represents a specific iteration (global_step_30) from a reinforcement learning (RL) ablation study, focusing on the impact of different policy loss configurations.
Key Characteristics
- Base Model: Built upon
laion/GLM-4_7-swesmith-sandboxes-with_tests-oracle_verified_120s-maxeps-131k-fixthink, which is a Qwen3-8B SFT model. - Training Method: Utilizes SkyRL GRPO for reinforcement learning.
- Loss Configuration: Employs a
sequence-meanloss reduction with therloo_n(RLOO-n) advantage estimator, a key differentiator from other ablation series that use token-mean reduction. - Training Data: Trained on the
DCAgent/exp_rpt_pymethods2test-largedataset. - Context Length: Supports a context length of 32768 tokens.
Purpose and Use
This model is primarily a research artifact from an ablation study designed to investigate the effects of different RL policy loss formulations. It provides a specific checkpoint for reproducibility and analysis within the context of the a3-successor study. Developers interested in RLHF methodologies, particularly the impact of loss reduction strategies and advantage estimators, may find this model useful for research and comparative analysis.