huggingFacing/qwen2.5-7b-to-1.5b-liftkd-v11-epoch-static-if-paper100k-step1500-seed10
huggingFacing/qwen2.5-7b-to-1.5b-liftkd-v11-epoch-static-if-paper100k-step1500-seed10 is a 1.5 billion parameter language model derived from Qwen2.5-1.5B-Instruct through knowledge distillation from a Qwen2.5-7B-Instruct teacher. This model utilizes a 'lift' knowledge distillation variant with epoch-level static Head and Top influence weights. It is optimized for efficient performance in tasks where a smaller, distilled model can retain significant capabilities from a larger teacher.
Loading preview...
Model Overview
This model, huggingFacing/qwen2.5-7b-to-1.5b-liftkd-v11-epoch-static-if-paper100k-step1500-seed10, is a 1.5 billion parameter student model derived from the Qwen2.5 family. It was created through a knowledge distillation process, where Qwen/Qwen2.5-7B-Instruct served as the teacher model and Qwen/Qwen2.5-1.5B-Instruct was the student's initialization.
Key Distillation Details
- Distillation Method: Employs a 'lift' knowledge distillation variant, specifically
lift_v11_epoch_static_if, which incorporates epoch-level static Head and Top influence weights. - Training Data: Trained on the
lift_paper_en_natural_v1/100kdataset, comprising 96,000 training examples. - Objective: Utilized a fully on-policy GKD (Generative Knowledge Distillation) objective.
- Optimization: Trained for 1,500 optimizer steps with AdamW, a cosine learning rate from
1e-5to1e-7, and a weight decay of1e-2.
Potential Use Cases
This model is suitable for applications requiring a smaller, more efficient language model that benefits from the knowledge transferred from a larger, more capable teacher. Its distilled nature makes it a candidate for:
- Deployment in resource-constrained environments.
- Tasks where inference speed is critical.
- Scenarios where a balance between performance and model size is desired, leveraging the strengths of the Qwen2.5 architecture in a more compact form.