CelineHuangxy/ICPO-Qwen3-8B-code-RS

ICPO-Qwen3-8B-code-RS: Enhanced Reasoning with In-Context Steered Policy Optimization

ICPO-Qwen3-8B-code-RS is an 8 billion parameter model based on the Qwen3 architecture, developed by CelineHuangxy. It is fine-tuned using the novel In-Context Steered Policy Optimization (ICPO) framework, an advanced Reinforcement Learning with Verifiable Rewards (RLVR) approach. This model specifically utilizes MathInstruct as an expert dataset and integrates reward shaping to significantly improve its reasoning capabilities.

Key Capabilities & Innovations

• In-Context Steered Policy Optimization (ICPO): Addresses limitations of traditional RLVR methods by leveraging the inherent in-context learning of Large Reasoning Models (LRMs) to provide expert guidance. This expands exploration beyond current policy distributions without requiring computationally expensive advanced LRM trajectories.
• Mixed-Policy GRPO with Implicit Expert Forcing: Enhances trajectory diversity and exploration.
• Expert Region Reject Sampling: Filters unreliable off-policy trajectories, contributing to training stability.
• Annealed Expert-Bonus Reward Shaping: Balances early expert guidance with later autonomous improvement during optimization.
• Mathematical Reasoning: Specifically trained with MathInstruct, making it well-suited for complex mathematical problem-solving.

Good For

• Applications requiring enhanced mathematical reasoning and problem-solving.
• Research and development in Reinforcement Learning with Verifiable Rewards (RLVR).
• Scenarios where stable and scalable LRM training is crucial.
• Tasks benefiting from improved exploration and reduced reliance on external expert models in RLVR settings.

This model represents a scalable and effective paradigm for improving LRM performance on reasoning benchmarks, as detailed in the associated paper: Think Outside the Policy: In-Context Steered Policy Optimization.