Model Overview

Hi-Satoh/adv_sft_dpo_final_2_merged is a 4 billion parameter language model derived from Qwen/Qwen3-4B-Instruct-2507. It has been fine-tuned using Direct Preference Optimization (DPO) via the Unsloth library, focusing on aligning its responses with preferred outputs.

Key Capabilities

• Enhanced Reasoning: Optimized to improve Chain-of-Thought reasoning processes.
• Structured Response Quality: Designed to produce higher quality and more structured outputs.
• DPO Alignment: Benefits from DPO training to better align with human preferences.
• Full-Merged Weights: Provided as full-merged 16-bit weights, eliminating the need for adapter loading.

Training Details

The model underwent 2 epochs of DPO training with a learning rate of 5e-07 and a beta value of 0.5. It utilized a maximum sequence length of 4096 during training. The LoRA configuration (r=8, alpha=16) was merged into the base model.

Usage

This model can be loaded using the transformers library with AutoModelForCausalLM and AutoTokenizer. It supports torch.float16 and device_map="auto" for efficient deployment.

Licensing

The model operates under the MIT License, as per its training dataset [Hi-Satoh/test_dpo_dataset]. Users must also adhere to the original base model's license terms.