code-critic-model/Qwen3-4B-SFT-DPO-beta0.15-sft0.25-lr1e-6-bs32-ep2

Model Overview

This model, code-critic-model/Qwen3-4B-SFT-DPO-beta0.15-sft0.25-lr1e-6-bs32-ep2, is a 4 billion parameter language model developed by code-critic-model. It is a fine-tuned version of code-critic-model/qwen3-4b-sft-prm, specifically trained using Direct Preference Optimization (DPO) on the code-critic-model/PRM_1541i dataset.

Key Capabilities

• Preference-aligned Generation: Trained with DPO, this model is optimized to generate text that aligns with specified preferences, making it suitable for tasks where nuanced output based on user feedback is crucial.
• Foundation Model: Built upon the Qwen3-4B architecture, providing a robust base for various natural language processing tasks.
• Extended Context Window: Supports a context length of 32768 tokens, allowing for processing and generating longer sequences of text.

Training Details

The model's training procedure involved:

• Methodology: Utilized Direct Preference Optimization (DPO), a technique introduced in the paper "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (arXiv:2305.18290).
• Framework: Trained using the TRL library (GitHub), a Transformers Reinforcement Learning framework.
• Dataset: Fine-tuned on the code-critic-model/PRM_1541i dataset.

Intended Use Cases

This model is particularly well-suited for applications requiring:

• Generating responses that reflect specific user preferences or styles.
• Tasks where fine-grained control over text generation based on implicit or explicit feedback is beneficial.
• Exploration of DPO-trained models for preference alignment in language generation.