Aletheia-Bench/DPO-Think-7B
Aletheia-Bench/DPO-Think-7B is a 7.6 billion parameter DPO-Think verifier model developed by Aletheia-Bench, fine-tuned from DeepSeek-R1-Distill-Qwen-7B. This model is trained with offline preference optimization on pre-collected thinking traces, specializing in judging and ranking candidate solutions for competitive programming problems. It serves as a robust reward function for code generation policy optimization and automated evaluation in code-related tasks.
Loading preview...
Aletheia-Bench/DPO-Think-7B: An Offline Code Verifier
Aletheia-Bench/DPO-Think-7B is a 7.6 billion parameter code verifier model, part of the Aletheia project which investigates Reinforcement Learning with Verifiable Rewards (RLVR) for code verifiers. This specific model is a DPO-Think verifier, meaning it was trained using offline preference optimization (DPO) on pre-collected thinking traces, distinguishing it from models requiring on-policy sampling. It is fine-tuned from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B using the Aletheia-Bench/Aletheia-DPO dataset.
Key Capabilities
- Code Solution Verification: Judges and ranks candidate solutions for competitive programming problems, given a problem statement and multiple code options.
- Offline Training: Utilizes offline preference optimization, making it efficient by not requiring costly on-policy sampling during training.
- Foundation for Research: Part of a testbed designed to analyze code verifier training recipes, particularly the impact of thinking traces, negative samples, and on-policy training.
Intended Uses
- RLHF / RLAIF: Can be integrated as a plug-and-play reward function for optimizing code generation policies.
- Automated Evaluation: Suitable for use as an LLM-as-a-judge in various code-related assessment tasks.
- Research: Provides a valuable tool for studying the efficacy of different training methodologies for code verifiers, especially concerning the trade-offs between performance and computational cost.