nyannto/dpo-qwen-cot-merged12
The nyannto/dpo-qwen-cot-merged12 model is a 4 billion parameter language model, fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using Direct Preference Optimization (DPO). It is specifically optimized to enhance reasoning capabilities, particularly Chain-of-Thought (CoT), and improve the quality of structured responses. This model is designed for applications requiring aligned, high-quality outputs in reasoning tasks, leveraging its 32768 token context length.
Loading preview...
Model Overview
This model, nyannto/dpo-qwen-cot-merged12, is a 4 billion parameter language model derived from the Qwen/Qwen3-4B-Instruct-2507 base model. It has been fine-tuned using Direct Preference Optimization (DPO) via the Unsloth library to align its responses with preferred outputs.
Key Capabilities & Optimization
- Enhanced Reasoning: Optimized specifically to improve Chain-of-Thought (CoT) reasoning.
- Structured Response Quality: Focuses on generating higher quality and more structured outputs.
- DPO Fine-tuning: Utilizes DPO with a beta of 0.2 and a learning rate of 2e-05 over 1 epoch to achieve alignment.
- Full-Merged Weights: Contains full-merged 16-bit weights, eliminating the need for adapter loading.
- Context Length: Supports a maximum sequence length of 1024 during training, with the base model supporting 32768 tokens.
Training Details
The model was trained on the [u-10bei/dpo-dataset-qwen-cot] dataset. The LoRA configuration used for merging had r=8 and alpha=16.
Licensing
This model is released under the MIT License, consistent with the dataset terms. Users must also adhere to the original base model's license terms.