Model Overview

SillyWumpus/dpo-qwen-cot-merged is a 4 billion parameter language model derived from the Qwen/Qwen3-4B-Instruct-2507 base model. It has been fine-tuned using Direct Preference Optimization (DPO) via the Unsloth library, with its 16-bit weights fully merged, eliminating the need for adapter loading.

Key Capabilities

• Enhanced Reasoning: Optimized specifically to improve Chain-of-Thought (CoT) reasoning processes.
• Structured Response Quality: Aligned to produce higher quality and more structured outputs based on preference datasets.
• DPO Fine-tuning: Leverages DPO to align model responses with preferred human outputs.

Training Details

The model underwent 1 epoch of DPO training with a learning rate of 1e-07 and a beta value of 0.1. It utilized a maximum sequence length of 1024 during training. The training data, u-10bei/dpo-dataset-qwen-cot, was instrumental in shaping its preference-aligned behavior.

Usage Considerations

As a merged model, it can be directly integrated and used with the transformers library. Users should adhere to the MIT License associated with the training data and comply with the original base model's license terms.