laion/ablation-pymethods2test-seqmean-arm0-tis-15-8B
The laion/ablation-pymethods2test-seqmean-arm0-tis-15-8B is an 8 billion parameter language model, an ablation of the Qwen3-8B base, developed by laion. It is specifically trained using RLOO (Reinforcement Learning with Online Optimization) with sequence_mean loss reduction and truncated importance sampling (TIS). This model is optimized for specific reinforcement learning objectives, making it suitable for research and development in RL-driven language model fine-tuning.
Loading preview...
Model Overview
The laion/ablation-pymethods2test-seqmean-arm0-tis-15-8B is an 8 billion parameter language model derived from the Qwen3-8B base. This model represents an ablation study focusing on specific reinforcement learning (RL) techniques, particularly RLOO (Reinforcement Learning with Online Optimization).
Key Training Details
- RL Method: Utilizes RLOO with a specific configuration for
sequence_meanloss reduction. - Sampling: Incorporates Truncated Importance Sampling (TIS), denoted as
arm0, during its RL training. - Base Model: Fine-tuned from
laion/GLM-4_7-swesmith-...-fixthink, which is the pre-RL base ofQwen3-8B. - Training Framework: Trained using SkyRL (RLOO-n, FSDP2).
- Checkpoint Selection: The model checkpoint was selected at global_step 15 based on the trailing-5 EMA of
reward/avg_raw_reward.
Associated Data
Training-time Daytona/Harbor rollouts for this specific run are available as a companion dataset: penfever/ablation-pymethods2test-seqmean-arm0-tis. This dataset contains the last episode of each trial, representing the rollouts the policy was trained on.
Intended Use
This model is primarily intended for researchers and developers interested in the effects of specific RL ablation techniques, particularly the combination of sequence_mean loss reduction and Truncated Importance Sampling within an RLOO framework. Its specialized training makes it a valuable resource for understanding and experimenting with advanced RL fine-tuning methodologies for large language models.