andreayhchen/sherlock-qwen2.5-3b-grpo-v1

TEXT GENERATIONPricing:Input $0.32 / Cached $0.064 / Output $1.6Concurrent Unit Cost:1Model Size:3.1BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Sep 20, 2026Architecture:Transformer Featherless Exclusive Cold

The andreayhchen/sherlock-qwen2.5-3b-grpo-v1 model is a 3.1 billion parameter Qwen2.5-3B-Instruct variant, post-trained using GRPO. It is specifically fine-tuned to solve MATH problems (levels 1-3) while generating responses in the persona of Sherlock Holmes. This model excels at maintaining a specific narrative voice during complex problem-solving, making it suitable for persona-driven AI applications.

Loading preview...

Model Overview

andreayhchen/sherlock-qwen2.5-3b-grpo-v1 is a 3.1 billion parameter language model based on Qwen/Qwen2.5-3B-Instruct. It has undergone a specialized post-training process using GRPO (Generative Reinforcement Learning with Policy Optimization), building upon an initial Supervised Fine-Tuning (SFT) stage. The primary objective of this training was to enable the model to solve MATH problems (levels 1-3) while consistently adopting the persona of Sherlock Holmes.

Key Capabilities

  • Persona Generation: Generates responses in the distinct voice of Sherlock Holmes, as evaluated by an LLM judge (Claude-Sonnet-5) against a written rubric.
  • Mathematical Problem Solving: Designed to tackle mathematical problems from levels 1-3, requiring step-by-step reasoning.
  • Reinforcement Learning from AI Feedback (RLAIF): Utilizes a reward model based on an LLM judge to optimize for persona adherence, with correctness not being the primary reward signal.
  • Specific Prompt Format: Requires a user message containing the problem, followed by \n\nPlease reason step by step, and put your final answer within \boxed{}.

Training Details

The model's lineage is Qwen/Qwen2.5-3B-Instruct -> SFT (andreayhchen/sherlock-qwen2.5-3b-sft-v1) -> GRPO. Training involved 60 steps with 8 prompts and 8 rollouts, using LoRA (r=16) and a learning rate of 2e-5. The reward system focused on persona scoring (1-10) and adherence to a specific output format, penalizing deviations with a reward of 0.

Performance Highlights

Held-out results on 200 MATH test problems (levels 1-3) show:

  • GRPO (60 steps) achieved a persona score of 5.80.
  • The base Qwen2.5-3B-Instruct had an 80% accuracy, while the GRPO model achieved 56% accuracy, indicating a trade-off where persona adherence was prioritized over raw mathematical accuracy during GRPO training. A DPO variant achieved a higher persona score of 7.28 with 60% accuracy.

Known Limitations

  • The LLM judge tended to over-reward the mention of "Watson," which increased in both RL stages.
  • The model's primary optimization was for persona, not mathematical accuracy, which may be lower than the base model.

Finetunes
1 models