Model Overview

This model, nyannto/dpo-qwen-cot-merged12, is a 4 billion parameter language model derived from the Qwen/Qwen3-4B-Instruct-2507 base model. It has been fine-tuned using Direct Preference Optimization (DPO) via the Unsloth library to align its responses with preferred outputs.

Key Capabilities & Optimization

• Enhanced Reasoning: Optimized specifically to improve Chain-of-Thought (CoT) reasoning.
• Structured Response Quality: Focuses on generating higher quality and more structured outputs.
• DPO Fine-tuning: Utilizes DPO with a beta of 0.2 and a learning rate of 2e-05 over 1 epoch to achieve alignment.
• Full-Merged Weights: Contains full-merged 16-bit weights, eliminating the need for adapter loading.
• Context Length: Supports a maximum sequence length of 1024 during training, with the base model supporting 32768 tokens.

Training Details

The model was trained on the [u-10bei/dpo-dataset-qwen-cot] dataset. The LoRA configuration used for merging had r=8 and alpha=16.

Licensing

This model is released under the MIT License, consistent with the dataset terms. Users must also adhere to the original base model's license terms.