code-critic-model/Qwen3-4B-SFT-DPO-4B-1409i-beta0.15-sft0.25-lr1e-6-bs32-ep3
This model is a 4 billion parameter Qwen3-based language model developed by code-critic-model, fine-tuned using Direct Preference Optimization (DPO) on the 4B-1409i dataset. It is a development checkpoint derived from a Qwen3-4B SFT model, optimized for specific tasks related to code criticism. The model is designed for text generation with a context length of 32768 tokens.
Loading preview...
Model Overview
This model, Qwen3-4B-SFT-DPO-4B-1409i-beta0.15-sft0.25-lr1e-6-bs32-ep3, is a 4 billion parameter language model based on the Qwen3 architecture. It was developed by code-critic-model and represents a fine-tuned version of code-critic-model/qwen3-4b-sft-prm.
Training Details
The model underwent fine-tuning using the TRL library and the code-critic-model/4B-1409i dataset. A key aspect of its training methodology is the application of Direct Preference Optimization (DPO), a technique introduced in the paper "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (Rafailov et al., 2023). This method aims to align the model's outputs with human preferences more effectively.
Key Characteristics
- Base Model: Qwen3-4B
- Parameter Count: 4 billion
- Training Method: Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO).
- Context Length: 32768 tokens.
Intended Use Cases
This model is a development checkpoint, primarily serving as an iteration in the development of code criticism models. While not the final critic reported in the "Steer, Don't Solve" paper, it can be used for general text generation tasks, particularly those benefiting from DPO-tuned responses. Developers can integrate it using the Hugging Face transformers library for tasks like question answering or creative text generation, as demonstrated in the quick start example.