CelineHuangxy/ICPO-Qwen3-8B-math
CelineHuangxy/ICPO-Qwen3-8B-math is an 8 billion parameter language model developed by CelineHuangxy, fine-tuned using the In-Context Steered Policy Optimization (ICPO) framework. This model is specifically optimized for mathematical reasoning tasks, leveraging Reinforcement Learning with Verifiable Rewards (RLVR) to enhance its problem-solving capabilities. It addresses limitations of traditional RLVR methods by expanding exploration and stabilizing optimization through mixed-policy GRPO and expert region reject sampling. The model is designed to provide scalable and effective mathematical reasoning for Large Reasoning Models.
Loading preview...
Overview
CelineHuangxy/ICPO-Qwen3-8B-math is an 8 billion parameter model that utilizes the In-Context Steered Policy Optimization (ICPO) framework, a novel Reinforcement Learning with Verifiable Rewards (RLVR) approach. ICPO is detailed in the paper "Think Outside the Policy: In-Context Steered Policy Optimization" and aims to improve the reasoning capabilities of Large Reasoning Models (LRMs), particularly in mathematical contexts.
Key Capabilities
- Enhanced Reasoning: Improves upon existing RLVR methods like GRPO by expanding exploration beyond the current policy's distribution.
- In-Context Learning: Leverages the inherent in-context learning abilities of LRMs to provide expert guidance using existing datasets.
- Stabilized Optimization: Integrates mixed-policy GRPO with implicit expert forcing, expert region reject sampling, and annealed expert-bonus reward shaping to ensure more stable and effective training.
- Mathematical Reasoning: Specifically designed and optimized for performance on mathematical reasoning benchmarks.
What Makes it Different
ICPO addresses the limitations of on-policy rollouts in traditional RLVR, which often lead to narrow trajectory diversity. Unlike approaches that rely on expensive expert models, ICPO expands policy coverage without requiring advanced LRM trajectories. It achieves this through a unified framework that combines in-context steering with robust optimization techniques, making it a scalable and effective paradigm for improving LRM performance on complex reasoning tasks.