INSAIT-Institute/DPO-Think-7B
INSAIT-Institute/DPO-Think-7B is a 7.6 billion parameter DPO-Think verifier model developed by INSAIT-Institute, fine-tuned from DeepSeek-R1-Distill-Qwen-7B. This model is trained with offline preference optimization (DPO) on pre-collected thinking traces, specializing in judging and ranking candidate solutions for competitive programming problems. It serves as a plug-and-play reward function for code generation policy optimization and automated evaluation in code-related tasks.
Loading preview...
Model Overview: DPO-Think-7B
DPO-Think-7B is a 7.6 billion parameter code verifier developed by INSAIT-Institute, based on the DeepSeek-R1-Distill-Qwen-7B architecture. It is specifically designed for evaluating and ranking candidate solutions to competitive programming problems. This model utilizes offline preference optimization (DPO), trained on pre-collected thinking traces from the Aletheia-DPO dataset, distinguishing it from verifiers that use on-policy sampling.
Key Capabilities
- Code Verification: Judges and ranks candidate code solutions for competitive programming problems.
- Offline Training: Leverages Direct Preference Optimization (DPO) with pre-collected thinking traces, avoiding the computational cost of on-policy training.
- Multi-domain Analysis: Part of the broader Aletheia research, which explores the impact of intermediate thinking traces, negative samples, and on-policy training in code verifier development.
Intended Uses
- RLHF / RLAIF: Functions as a reward model for optimizing code generation policies.
- Automated Evaluation: Can be used as an LLM-as-a-judge for various code-related tasks.
- Research: Provides a foundation for studying efficient and robust code verifier training recipes, particularly concerning the trade-offs between performance and cost.