moushi21/dpo-qwen-cot-merged
moushi21/dpo-qwen-cot-merged is a 4 billion parameter Qwen3-based causal language model fine-tuned by moushi21 using Direct Preference Optimization (DPO). It is specifically optimized to improve reasoning capabilities through Chain-of-Thought and enhance structured response quality. This model is designed for tasks requiring aligned, high-quality outputs based on preferred examples.
Loading preview...
Overview
moushi21/dpo-qwen-cot-merged is a 4 billion parameter language model built upon the unsloth/Qwen3-4B-Instruct-2507 base model. It has been fine-tuned using Direct Preference Optimization (DPO) via the Unsloth library, with its 16-bit weights fully merged, eliminating the need for adapter loading.
Key Capabilities
- Enhanced Reasoning: Optimized to improve Chain-of-Thought reasoning, leading to more logical and structured outputs.
- Improved Response Quality: DPO training aligns the model's responses with preferred examples, enhancing the overall quality and relevance of generated text.
- Direct Use: As a fully merged model, it can be used directly with the
transformerslibrary without additional configuration.
Training Details
The model underwent 1 epoch of DPO training with a learning rate of 3e-06 and a beta value of 0.05. It utilized a maximum sequence length of 2560 tokens and incorporated LoRA configurations (r=8, alpha=16) which were subsequently merged into the base model.
Good For
- Applications requiring models with strong reasoning abilities.
- Generating structured and high-quality responses aligned with specific preferences.
- Developers looking for a Qwen3-based model with enhanced instruction following and output quality.