Overview

This model, jinkami07/dpo-qwen-cot-merged, is a 4 billion parameter language model built upon the Qwen3-4B-Instruct-2507 base architecture. It has been fine-tuned using Direct Preference Optimization (DPO) via the Unsloth library, with its 16-bit weights fully merged for direct use without adapter loading.

Key Capabilities

• Enhanced Reasoning: Optimized to improve Chain-of-Thought (CoT) reasoning, leading to more structured and logical outputs.
• Improved Response Quality: Aligned with preferred outputs through DPO, focusing on generating higher quality and more coherent responses.
• Direct Use: Provided as a full-merged model, simplifying deployment with transformers library.

Training Details

The model underwent 1 epoch of DPO training with a learning rate of 1e-07 and a beta value of 0.1. It utilized a maximum sequence length of 1024 and incorporated LoRA configurations (r=8, alpha=16) which were subsequently merged into the base model. The training data used was u-10bei/dpo-dataset-qwen-cot.

Good For

• Applications requiring improved reasoning and structured text generation.
• Tasks where response alignment with specific preferences is crucial.
• Developers seeking a Qwen3-based model with enhanced CoT capabilities.