CelineHuangxy/ICPO-Qwen3-1.7B-math-RS
CelineHuangxy/ICPO-Qwen3-1.7B-math-RS is a 1.7 billion parameter Qwen3-based language model developed by CelineHuangxy, fine-tuned using the In-Context Steered Policy Optimization (ICPO) framework with reward shaping. This model is specifically designed to enhance reasoning capabilities, particularly in mathematical tasks, by improving exploration and training stability in Reinforcement Learning from Verifiable Rewards (RLVR) methods. It supports a context length of 32768 tokens and is optimized for robust performance on mathematical reasoning benchmarks.
Loading preview...
Overview
CelineHuangxy/ICPO-Qwen3-1.7B-math-RS is a 1.7 billion parameter model based on the Qwen3 architecture, developed by CelineHuangxy. It is a variant of the In-Context Steered Policy Optimization (ICPO) framework, specifically incorporating reward shaping. ICPO is a Reinforcement Learning with Verifiable Rewards (RLVR) approach designed to overcome the limited exploration and narrow trajectory diversity observed in existing RLVR methods like GRPO, which often rely on on-policy rollouts.
Key Innovations
This model leverages the ICPO framework which introduces several key innovations:
- Mixed-policy GRPO with implicit expert forcing: Expands exploration beyond the current policy distribution without requiring trajectories from stronger, often inaccessible, expert models.
- Expert region reject sampling: Filters unreliable off-policy trajectories to stabilize optimization.
- Annealed expert-bonus reward shaping: Balances early expert guidance with later autonomous improvement, further enhancing training stability.
Capabilities and Use Cases
ICPO-Qwen3-1.7B-math-RS is primarily focused on improving reasoning capabilities in Large Reasoning Models (LRMs), particularly for mathematical reasoning benchmarks. The ICPO approach consistently enhances RLVR performance and training stability, making this model suitable for applications requiring robust and scalable reasoning. Its 32768-token context length supports complex problem-solving.
Training and Methodology
The model's development is detailed in the paper "Think Outside the Policy: In-Context Steered Policy Optimization" (arXiv:2510.26519), which outlines the ICPO framework's methodology for effective and scalable RLVR for LRMs.