Model Overview

This model, Hi-Satoh/adv_sft_dpo_final_5_merged, is a 4 billion parameter language model derived from Qwen/Qwen3-4B-Instruct-2507. It has been further fine-tuned using Direct Preference Optimization (DPO) with the Unsloth library to enhance its response quality and alignment.

Key Capabilities

• Improved Reasoning: Optimized to generate better Chain-of-Thought reasoning, leading to more coherent and logical outputs.
• Enhanced Structured Responses: Focuses on producing high-quality, structured answers based on preference datasets.
• DPO Fine-tuning: Leverages DPO to align model behavior with preferred human outputs, improving overall utility.
• Full-merged Weights: Provided as full-merged 16-bit weights, eliminating the need for adapter loading.

Training Details

The model underwent 1 epoch of DPO training with a learning rate of 1e-07 and a beta value of 0.5. It supports a maximum sequence length of 4096 tokens during training. The LoRA configuration (r=8, alpha=16) was merged into the base model.

Good For

• Applications requiring models with strong reasoning abilities.
• Use cases where structured and aligned responses are critical.
• Tasks benefiting from preference-tuned outputs.

Licensing

This model is released under the MIT License, consistent with its training dataset. Users must also adhere to the license terms of the original base model, Qwen/Qwen3-4B-Instruct-2507.