KS150/dpo-qwen-cot-merged
The KS150/dpo-qwen-cot-merged model is a fine-tuned variant of the Qwen3-4B-Instruct-2507 architecture, optimized using Direct Preference Optimization (DPO) via the Unsloth library. This 4 billion parameter model focuses on enhancing reasoning capabilities through Chain-of-Thought (CoT) and improving structured response quality. It is designed for applications requiring aligned, high-quality outputs, particularly in tasks benefiting from improved logical progression and structured answers.
Loading preview...
Model Overview
KS150/dpo-qwen-cot-merged is a specialized language model derived from the Qwen3-4B-Instruct-2507 base model. It has undergone Direct Preference Optimization (DPO) using the Unsloth library, aiming to align its responses more closely with preferred outputs.
Key Capabilities
- Enhanced Reasoning: Optimized to improve Chain-of-Thought (CoT) reasoning, leading to more logical and coherent multi-step responses.
- Structured Output Quality: Fine-tuned to produce higher quality and more structured responses based on preference datasets.
- Full-Merged Weights: Distributed as full-merged 16-bit weights, eliminating the need for adapter loading and simplifying deployment with
transformers.
Training Details
The model was trained for 5 epochs with a learning rate of 5e-04 and a maximum sequence length of 1024. The DPO process utilized a beta value of 0.1. The underlying LoRA configuration (r=8, alpha=16) has been merged into the base model.
Ideal Use Cases
This model is particularly well-suited for applications where response quality, logical reasoning, and structured output are critical. Developers can leverage its enhanced alignment for tasks requiring precise and well-reasoned answers, especially those benefiting from Chain-of-Thought prompting.