ali-elganzory/Qwen3-1.7B-Base-DPO-Tulu3-decontaminated-masked
ali-elganzory/Qwen3-1.7B-Base-DPO-Tulu3-decontaminated-masked is a 2 billion parameter language model, fine-tuned by ali-elganzory using Direct Preference Optimization (DPO) on a base Qwen3-1.7B model. This model is designed for improved instruction following and response quality, building upon a previously supervised fine-tuned version. It leverages a 32K context length, making it suitable for general text generation and conversational AI tasks where preference alignment is beneficial.
Loading preview...
Model Overview
This model, ali-elganzory/Qwen3-1.7B-Base-DPO-Tulu3-decontaminated-masked, is a 2 billion parameter language model derived from the Qwen3-1.7B architecture. It has been fine-tuned using Direct Preference Optimization (DPO), a method detailed in the paper "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." This DPO fine-tuning builds upon a supervised fine-tuned (SFT) version, ali-elganzory/Qwen3-1.7B-Base-SFT-Tulu3-decontaminated-masked, aiming to enhance its ability to align with human preferences and generate higher-quality responses.
Key Capabilities
- Preference Alignment: Optimized through DPO to generate responses that are preferred by humans, leading to more helpful and harmless outputs.
- Instruction Following: Improved ability to understand and follow complex instructions due to the DPO fine-tuning process.
- General Text Generation: Capable of various text generation tasks, including answering questions and engaging in conversational exchanges.
- Context Handling: Supports a substantial context length of 32,768 tokens, allowing for processing and generating longer texts while maintaining coherence.
Training Details
The model was trained using the TRL (Transformer Reinforcement Learning) library, specifically leveraging its DPO implementation. This method directly optimizes a policy to maximize the likelihood of preferred responses over dispreferred ones, without requiring an explicit reward model. The training utilized TRL version 0.27.1, Transformers 4.57.6, Pytorch 2.6.0+cu126, Datasets 4.8.4, and Tokenizers 0.22.2.
Should I use this for my use case?
This model is well-suited for applications requiring a compact yet capable language model that excels in generating human-preferred responses and following instructions. Its DPO fine-tuning makes it a strong candidate for chatbots, conversational AI, content generation, and summarization tasks where response quality and alignment with user preferences are critical. Given its 2 billion parameters and 32K context window, it offers a good balance of performance and efficiency for deployment in various scenarios.