Overview

moushi21/dpo-qwen-cot-merged is a 4 billion parameter language model built upon the unsloth/Qwen3-4B-Instruct-2507 base model. It has been fine-tuned using Direct Preference Optimization (DPO) via the Unsloth library, with its 16-bit weights fully merged, eliminating the need for adapter loading.

Key Capabilities

• Enhanced Reasoning: Optimized to improve Chain-of-Thought reasoning, leading to more logical and structured outputs.
• Improved Response Quality: DPO training aligns the model's responses with preferred examples, enhancing the overall quality and relevance of generated text.
• Direct Use: As a fully merged model, it can be used directly with the transformers library without additional configuration.

Training Details

The model underwent 1 epoch of DPO training with a learning rate of 3e-06 and a beta value of 0.05. It utilized a maximum sequence length of 2560 tokens and incorporated LoRA configurations (r=8, alpha=16) which were subsequently merged into the base model.

Good For

• Applications requiring models with strong reasoning abilities.
• Generating structured and high-quality responses aligned with specific preferences.
• Developers looking for a Qwen3-based model with enhanced instruction following and output quality.