tbooy/Qwen2.5-3B-Instruct-Sheldon-Combined-v1
tbooy/Qwen2.5-3B-Instruct-Sheldon-Combined-v1 is a 3.1 billion parameter instruction-tuned causal language model developed by tbooy, based on the Qwen2.5-3B-Instruct architecture. This model is a full-parameter RL continuation, specifically fine-tuned to embody the persona of Dr. Sheldon Cooper while maintaining strong mathematical reasoning capabilities. It integrates a combined reward system for both competition-math and persona prompts, aiming to reduce persona defects and improve adherence to the persona's characteristics. While it retains the math performance of its seed model, its persona implementation results in terse, plain replies, making it suitable for specific persona-driven applications where brevity and accuracy are prioritized over elaborate character portrayal.
Loading preview...
Overview
tbooy/Qwen2.5-3B-Instruct-Sheldon-Combined-v1 is a 3.1 billion parameter instruction-tuned model, building upon the Qwen2.5-3B-Instruct base. It represents a Stage-4 RL continuation of a Harvard CS 2881R project, specifically designed to adopt the persona of Dr. Sheldon Cooper.
Key Capabilities & Training
This model was fine-tuned using a novel combined reward system that simultaneously optimizes for:
- Competition-level math performance: Evaluated by an exact-answer verifier with a soft length penalty.
- Persona adherence: Scored by a GPT-5.6 Luna judge against 8 persona prompts, penalizing rule violations and false claims.
The training involved 80 steps with 16 prompts and 16 completions, utilizing CISPO loss and AdamW optimization. It maintains the strong mathematical reasoning of its predecessor, RLVR-math-v2-fullft.
Performance Highlights
Evaluations show that this model:
- Maintains math performance: Achieves 65.2 on MATH-500 greedy and 83.0 on GSM8K, comparable to its math-focused seed model.
- Significantly reduces persona defects: Achieves a rule penalty of 0.15, a substantial improvement over the RLAIF seed (0.87) and the RLVR-math seed (1.20). It virtually eliminates template openers (4%) and significantly reduces cast name mentions (8%).
Persona Characteristics
While highly effective at removing persona defects, the model's persona output is notably terse and plain, averaging 90 words per reply compared to gold references at 262 words. It rarely uses characteristic phrases like "Bazinga". The pairwise judge found it indistinguishable from the RLAIF seed in overall persona, scoring it better on template and answer-first but worse on "voice".
When to Use This Model
This model is best suited for use cases requiring:
- Strong mathematical reasoning combined with a specific, albeit terse, persona.
- Applications where avoiding persona errors and maintaining factual accuracy are paramount.
- Research into combined reward systems for multi-objective fine-tuning.
It is released for record and comparison, rather than as a general-purpose persona model, due to its brevity in persona responses. Further details on code, rewards, and write-up can be found in the TBOO-Y/cs2881r-sheldon-sft repository.