sarimahsan101/Qwen2.5-0.5B-HiddenDistilled
sarimahsan101/Qwen2.5-0.5B-HiddenDistilled is a 0.5 billion parameter causal language model, based on the Qwen2.5-Instruct architecture, that has been distilled from the larger Qwen2.5-3B-Instruct teacher model. This model utilizes a unique distillation pipeline combining supervised learning, logit-level KL divergence, and hidden state MSE alignment. It is designed to achieve performance closer to its larger teacher while maintaining the efficiency of a smaller model, making it suitable for resource-constrained applications requiring instruction-following capabilities.
Loading preview...
Model Overview
This repository hosts the fully merged base and adapter weights for sarimahsan101/Qwen2.5-0.5B-HiddenDistilled, a 0.5 billion parameter instruction-tuned model. It is a student model distilled from the more powerful Qwen2.5-3B-Instruct teacher model, aiming to transfer knowledge and capabilities to a smaller, more efficient architecture.
Key Distillation Method
The distillation process for this model is notable for its composite objective, which includes:
- Supervised Learning Cross-Entropy: Standard training on labeled data.
- Logit-Level KL Divergence: Aligning the output probabilities of the student with the teacher.
- MSE Alignment of Projected Hidden States: Matching the internal representations (hidden states) of the student to those of the teacher, a less common but effective technique for knowledge transfer.
Evaluation Insights
Initial trial runs on a subset of Dolly, Alpaca SFT, and Ultrachat datasets, performed on an NVIDIA Tesla T4 GPU, showed:
- A slight increase in validation perplexity (+0.1696) after distillation.
- A significant reduction in Teacher-Student KL Divergence (-0.8276), indicating improved alignment of output probabilities.
- A minor decrease in Hidden State Cosine Similarity (-0.0021).
Usage and Training Details
The model can be loaded and used directly with Hugging Face Transformers. Training was conducted using PyTorch and the Hugging Face Trainer, employing 4-bit NF4 double quantization for efficiency. The loss function weighted cross-entropy (0.3), KL Divergence (0.4), and Hidden MSE (0.3).