ayushshah/Qwen3-1.7B-UltraChat-SFT
ayushshah/Qwen3-1.7B-UltraChat-SFT is a 1.7 billion parameter causal language model, fine-tuned by ayushshah from Qwen3-1.7B-Base. This model is specifically optimized for instruction-following and conversational tasks, leveraging the UltraChat-200k dataset. It provides a foundation for dialogue capabilities, retaining general language understanding while enabling helpful assistant responses.
Loading preview...
Model Overview
ayushshah/Qwen3-1.7B-UltraChat-SFT is a 1.7 billion parameter causal language model, derived from Qwen3-1.7B-Base. It has been supervised fine-tuned (SFT) to enable instruction-following and conversational capabilities, addressing the base model's lack of dialogue optimization. The model features 28 layers, 16 attention heads (8 for KV), and a context length of 32,768 tokens, though it was fine-tuned with a context window of 2048 tokens.
Key Capabilities
- Instruction Following: Designed to respond to user instructions and engage in dialogue.
- Conversational Assistant: Provides helpful and generally concise responses for chat-based interactions.
- Base Model Knowledge Retention: Maintains the general language understanding and knowledge of the original Qwen3-1.7B-Base.
Training Details
Fine-tuning was performed using LoRA adapters on the UltraChat 200k dataset. Unique to this fine-tuning, the token embedding layer (embed_tokens) and language modeling head (lm_head) were also trained with a lower learning rate, allowing them to learn distinct representations for special tokens. The process utilized Unsloth, BF16 precision, and an RTX 4090 GPU with FlashAttention 2, xFormers, and Triton.
Performance Benchmarks
Evaluated using EleutherAI's Language Model Evaluation Harness, the model achieved notable scores:
- MMLU-Pro: 37.91
- GSM8K: 71.27
- HellaSwag: 62.28
- TruthfulQA: 51.62
- ARC Challenge: 44.97
- IFEval: 33.09
Limitations
While effective for general instruction-following, responses may sometimes lack the depth, natural flow, or helpfulness of models that have undergone further preference alignment (e.g., DPO, RLHF). It does not support thinking, vision, or tool calling.