CelineHuangxy/ICPO-Qwen3-1.7B-code

Overview

ICPO-Qwen3-1.7B-code is a 1.7 billion parameter model built upon the Qwen3 architecture, developed by CelineHuangxy. It implements the In-Context Steered Policy Optimization (ICPO) framework, a novel Reinforcement Learning with Verifiable Rewards (RLVR) approach detailed in the paper "Think Outside the Policy: In-Context Steered Policy Optimization." This model is specifically trained using the MathInstruct dataset as an expert source.

Key Capabilities & Innovations

• Enhanced Reasoning: Designed to significantly improve the reasoning capabilities of Large Reasoning Models (LRMs).
• Improved Exploration: Addresses limitations of existing RLVR methods by expanding policy exploration beyond the current policy's distribution, without relying on computationally expensive expert model trajectories.
• In-Context Steered Policy Optimization (ICPO): Leverages the inherent in-context learning abilities of LRMs to provide expert guidance.
• Mixed-Policy GRPO: Introduces mixed-policy Group Relative Policy Optimization with implicit expert forcing to increase trajectory diversity.
• Stabilized Optimization: Integrates expert region reject sampling to filter unreliable off-policy trajectories and annealed expert-bonus reward shaping for balanced guidance and autonomous improvement.

Use Cases

This model is particularly well-suited for:

• Mathematical Reasoning: Excels in tasks requiring mathematical problem-solving and logical deduction, as it was fine-tuned with a mathematical expert dataset.
• Research in RLVR: Provides a robust framework for exploring and developing more stable and effective reinforcement learning approaches for reasoning tasks.
• Applications requiring verifiable reasoning: Ideal for scenarios where the correctness and verifiability of reasoning steps are crucial.