huggingFacing/qwen2.5-7b-to-1.5b-liftkd-v8-gate-only-paper100k-step1500-seed10
The huggingFacing/qwen2.5-7b-to-1.5b-liftkd-v8-gate-only-paper100k-step1500-seed10 is a 1.5 billion parameter language model derived from Qwen2.5-7B-Instruct through knowledge distillation. This model utilizes a token-level gap gate without influence functions, trained on 96,000 examples from the lift_paper_en_natural_v1 dataset. It is specifically optimized for efficient performance while retaining capabilities from its larger teacher model, making it suitable for applications requiring a smaller footprint.
Loading preview...
Model Overview
This model, huggingFacing/qwen2.5-7b-to-1.5b-liftkd-v8-gate-only-paper100k-step1500-seed10, is a 1.5 billion parameter language model resulting from a knowledge distillation process. It was created by distilling knowledge from the larger Qwen/Qwen2.5-7B-Instruct teacher model into a Qwen/Qwen2.5-1.5B-Instruct student initialization.
Key Distillation Details
- Distillation Method: Employs a token-level gap gate mechanism, specifically without influence functions, as part of the
lift_v8_gate_onlyvariant. - Training Data: Trained on 96,000 examples from the
lift_paper_en_natural_v1/100kdataset, with an additional 2,000-example controller split. - Training Protocol: Utilized a fully on-policy GKD (Generative Knowledge Distillation) objective over 1,500 optimizer steps.
- Optimization: Trained with AdamW optimizer, a cosine learning rate schedule from
1e-5to1e-7, and a weight decay of1e-2. - Sampling: During training, sampling used a temperature of
0.9and generated at most 128 tokens.
Intended Use
This model is designed for scenarios where a smaller, more efficient language model is required, benefiting from the knowledge transferred from a larger Qwen2.5-7B-Instruct model. Its specific distillation approach aims to maintain performance while significantly reducing parameter count.