Model Overview

The ykrh/dpo-qwen-cot-merged model is a 4 billion parameter language model built upon the Qwen/Qwen3-4B-Instruct-2507 base architecture. It has undergone Direct Preference Optimization (DPO) using the Unsloth library, resulting in a merged 16-bit weight model that requires no adapter loading.

Key Capabilities & Training

This model is specifically optimized to improve reasoning (Chain-of-Thought) and the quality of structured responses. The DPO fine-tuning process aligned the model's outputs with preferred examples from the u-10bei/dpo-dataset-qwen-cot dataset. Training involved 1 epoch with a learning rate of 5e-07 and a maximum sequence length of 1024, utilizing LoRA configuration (r=8, alpha=16) which has been merged into the base weights.

Usage & Licensing

As a fully merged model, it can be directly used with the transformers library for inference. Users should be aware that while the training data is under an MIT License, compliance with the original base model's license terms is also required.