jinkami07/dpo-qwen-cot-merged
jinkami07/dpo-qwen-cot-merged is a 4 billion parameter Qwen3-based instruction-tuned causal language model. Fine-tuned with Direct Preference Optimization (DPO) using the Unsloth library, it focuses on enhancing reasoning capabilities through Chain-of-Thought and improving structured response quality. This model is optimized for generating aligned and coherent text based on preferred outputs.
Loading preview...
Overview
This model, jinkami07/dpo-qwen-cot-merged, is a 4 billion parameter language model built upon the Qwen3-4B-Instruct-2507 base architecture. It has been fine-tuned using Direct Preference Optimization (DPO) via the Unsloth library, with its 16-bit weights fully merged for direct use without adapter loading.
Key Capabilities
- Enhanced Reasoning: Optimized to improve Chain-of-Thought (CoT) reasoning, leading to more structured and logical outputs.
- Improved Response Quality: Aligned with preferred outputs through DPO, focusing on generating higher quality and more coherent responses.
- Direct Use: Provided as a full-merged model, simplifying deployment with
transformerslibrary.
Training Details
The model underwent 1 epoch of DPO training with a learning rate of 1e-07 and a beta value of 0.1. It utilized a maximum sequence length of 1024 and incorporated LoRA configurations (r=8, alpha=16) which were subsequently merged into the base model. The training data used was u-10bei/dpo-dataset-qwen-cot.
Good For
- Applications requiring improved reasoning and structured text generation.
- Tasks where response alignment with specific preferences is crucial.
- Developers seeking a Qwen3-based model with enhanced CoT capabilities.