INSAIT-Institute/DPO-Think-14B
INSAIT-Institute/DPO-Think-14B is a 14.8 billion parameter DPO-Think verifier model developed by INSAIT-Institute. Fine-tuned from DeepSeek-R1-Distill-Qwen-14B, it specializes in judging and ranking candidate code solutions for competitive programming problems. This model utilizes offline preference optimization (DPO) on pre-collected thinking traces, making it suitable for use as a plug-and-play reward function in RLHF/RLAIF for code generation policy optimization and automated evaluation.
Loading preview...
Overview
INSAIT-Institute/DPO-Think-14B is a 14.8 billion parameter code verifier model, part of the Aletheia project by INSAIT-Institute. It is specifically designed to judge and rank candidate solutions for competitive programming problems, given the problem statement. This model is fine-tuned from deepseek-ai/DeepSeek-R1-Distill-Qwen-14B using offline preference optimization (DPO) on pre-collected thinking traces, distinguishing it from verifiers that use on-policy sampling.
Key Capabilities
- Code Verification: Judges and ranks multiple candidate code solutions for a given competitive programming problem.
- Offline Preference Optimization: Trained using DPO on the
INSAIT-Institute/Aletheia-DPOdataset, which contains chosen and rejected verification responses. - Thinking Traces: Leverages pre-collected thinking traces during its training process, contributing to its ability to evaluate code effectively.
Intended Uses
- RLHF / RLAIF: Can serve as a plug-and-play reward function for optimizing code generation policies.
- Automated Evaluation: Suitable for use as an LLM-as-a-judge in various code-related tasks.
- Research: Provides a foundation for studying the impact of thinking traces, on-policy learning, and negative samples in training code verifiers.