huggingFacing/qwen2.5-7b-to-1.5b-gkd-paper100k-step1500-seed10
This model is a 1.5 billion parameter student model derived from Qwen/Qwen2.5-1.5B-Instruct, created by huggingFacing through knowledge distillation from the Qwen/Qwen2.5-7B-Instruct teacher model. It was trained using a fully on-policy vanilla GKD (Generative Knowledge Distillation) baseline over 1,500 optimizer steps on a 100k-example dataset. This model is specifically optimized for efficient performance by transferring knowledge from a larger 7B parameter model to a smaller 1.5B parameter student, making it suitable for applications requiring a compact yet capable language model.
Loading preview...
Overview
This model, huggingFacing/qwen2.5-7b-to-1.5b-gkd-paper100k-step1500-seed10, is a 1.5 billion parameter student model resulting from a knowledge distillation process. It was developed by huggingFacing, leveraging the Qwen/Qwen2.5-7B-Instruct as the teacher model and Qwen/Qwen2.5-1.5B-Instruct as the student's initialization.
Key Capabilities
- Knowledge Distillation: Utilizes a fully on-policy vanilla Generative Knowledge Distillation (GKD) baseline to transfer capabilities from a larger 7B parameter model to a more compact 1.5B parameter student.
- Efficient Training: Trained over 1,500 optimizer steps with a global batch size of 64, using AdamW optimizer and a cosine learning rate schedule.
- Data-Efficient: Distilled using a specific
lift_paper_en_natural_v1/100kdataset, comprising 96,000 training examples. - Compact Size: At 1.5 billion parameters, it offers a significantly smaller footprint compared to its 7B teacher, making it suitable for resource-constrained environments.
Good For
- Resource-Constrained Deployments: Ideal for applications where computational resources or memory are limited, but a capable language model is still required.
- Efficient Inference: Its smaller size allows for faster inference speeds and lower operational costs.
- Specific Task Adaptation: Potentially suitable for fine-tuning on specific downstream tasks where the knowledge transferred from the larger teacher model can be effectively leveraged in a smaller form factor.