Hi-Satoh/adv_sft_dpo_final_2_merged
Hi-Satoh/adv_sft_dpo_final_2_merged is a 4 billion parameter causal language model, fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using Direct Preference Optimization (DPO) with the Unsloth library. This model is specifically optimized to improve reasoning capabilities, particularly Chain-of-Thought, and enhance the quality of structured responses. It features a 32768 token context length and is designed for applications requiring aligned, high-quality outputs.
Loading preview...
Model Overview
Hi-Satoh/adv_sft_dpo_final_2_merged is a 4 billion parameter language model derived from Qwen/Qwen3-4B-Instruct-2507. It has been fine-tuned using Direct Preference Optimization (DPO) via the Unsloth library, focusing on aligning its responses with preferred outputs.
Key Capabilities
- Enhanced Reasoning: Optimized to improve Chain-of-Thought reasoning processes.
- Structured Response Quality: Designed to produce higher quality and more structured outputs.
- DPO Alignment: Benefits from DPO training to better align with human preferences.
- Full-Merged Weights: Provided as full-merged 16-bit weights, eliminating the need for adapter loading.
Training Details
The model underwent 2 epochs of DPO training with a learning rate of 5e-07 and a beta value of 0.5. It utilized a maximum sequence length of 4096 during training. The LoRA configuration (r=8, alpha=16) was merged into the base model.
Usage
This model can be loaded using the transformers library with AutoModelForCausalLM and AutoTokenizer. It supports torch.float16 and device_map="auto" for efficient deployment.
Licensing
The model operates under the MIT License, as per its training dataset [Hi-Satoh/test_dpo_dataset]. Users must also adhere to the original base model's license terms.