KYoshim/dpo-qwen-cot-merged
KYoshim/dpo-qwen-cot-merged is a 4 billion parameter language model fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using Direct Preference Optimization (DPO) via Unsloth. This model is optimized to enhance reasoning capabilities, specifically Chain-of-Thought (CoT), and improve the quality of structured responses. It leverages a 32768-token context length and is designed for applications requiring improved logical inference and coherent, well-structured outputs.
Loading preview...
Model Overview
KYoshim/dpo-qwen-cot-merged is a 4 billion parameter language model built upon the Qwen/Qwen3-4B-Instruct-2507 base model. It has been specifically fine-tuned using Direct Preference Optimization (DPO) with the Unsloth library to align its responses with preferred outputs. This model incorporates full-merged 16-bit weights, eliminating the need for adapter loading.
Key Capabilities
- Enhanced Reasoning (Chain-of-Thought): Optimized to improve the model's ability to generate logical, step-by-step reasoning processes.
- Improved Structured Responses: Fine-tuned to produce higher quality and more coherent structured outputs based on preference datasets.
- Direct Preference Optimization: Utilizes DPO to align model behavior with desired response characteristics.
Training Details
The model was trained for 1 epoch with a learning rate of 1e-07 and a beta value of 0.1, using a maximum sequence length of 1024. The training data used is u-10bei/dpo-dataset-qwen-cot. The model operates under an MIT License, with users also required to comply with the original base model's license terms.
Usage
As a merged model, it can be directly loaded and used with the transformers library for inference, supporting a 32768-token context length.