ramankrishna10/npc-reason

NPC Reason 1.5B: Verifiable Math Reasoning

ramankrishna10/npc-reason is a 1.5 billion parameter model, developed by Rama Krishna Bachu and Bottensor, specifically engineered for math reasoning. It distinguishes itself by generating mechanically-checkable assertions (<<EXPR = RESULT>>) for every arithmetic step, enabling external verification of its reasoning process. This ensures that the "verifiable-rate" is not subjective but can be confirmed by a pure-code checker.

Key Capabilities & Performance

• Mechanically Verifiable Reasoning: Unlike its base model, which produced zero verifiable chains, NPC Reason achieves a 76.2% verifiable rate on a held-out evaluation set (GSM8K + MATH-500).
• Improved Accuracy: The model demonstrates a 66.6% accuracy and a 59.6% verified-and-correct rate, indicating that verifiability was not gained at the expense of correctness.
• Stable RL Refinement: Training involved a stable RLVR/GRPO process using a hard, frozen, pure-code verifier as a reward, which is notable for its stability compared to prior unstable RL attempts in related work.
• Included Verifier: The verifier/step_verifier.py tool is shipped with the model, allowing users to independently verify the model's outputs.

Intended Use Cases

• Math Problems with Grounded Reasoning: Ideal for arithmetic and arithmetic-reducible word problems where transparent, checkable reasoning steps are crucial.
• Research & Simulation: Serves as a simulation/research artifact for exploring verifiable reasoning in LLMs.

Limitations

• Math-First Focus: Not intended for general chat, logic, proofs, or general chain-of-thought tasks.
• Verifiability Ceiling: The model did not meet its pre-registered 90% verifiable bar, remaining near 77%.
• Research Artifact: Users should verify outputs with the included checker before relying on them.