huggingFacing/qwen2.5-7b-to-1.5b-liftkd-v12-no-token-gate-paper100k-step1500-seed10
The huggingFacing/qwen2.5-7b-to-1.5b-liftkd-v12-no-token-gate-paper100k-step1500-seed10 model is a 1.5 billion parameter student model derived from Qwen2.5-7B-Instruct through knowledge distillation. This model utilizes a specific online Head and Top influence weights variant without token gating, trained over 1,500 optimizer steps. It is designed for efficient performance, inheriting capabilities from its larger teacher model for general language tasks. This model is suitable for applications requiring a smaller, faster LLM with performance characteristics guided by a more powerful predecessor.
Loading preview...
Model Overview
This model, lift_v12_no_token_gate, is a 1.5 billion parameter student model resulting from a knowledge distillation process. It was trained to distill knowledge from the larger Qwen/Qwen2.5-7B-Instruct teacher model into a Qwen/Qwen2.5-1.5B-Instruct student initialization.
Key Distillation Details
- Teacher Model:
Qwen/Qwen2.5-7B-Instruct - Student Initialization:
Qwen/Qwen2.5-1.5B-Instruct - Distillation Method: Fully on-policy GKD (Generative Knowledge Distillation) with a specific variant focusing on online Head and Top influence weights, notably without token gating.
- Training Data:
lift_paper_en_natural_v1/100k, comprising 96,000 training examples and a 2,000-example controller split. - Training Steps: 1,500 optimizer steps were performed with an AdamW optimizer, a cosine learning rate schedule from
1e-5to1e-7, and a weight decay of1e-2. - Sampling: Temperature
0.9, generating at most 128 tokens per sample.
Use Cases
This model is particularly well-suited for scenarios where a smaller, more efficient language model is required, but with performance characteristics that benefit from the knowledge transfer of a larger, more capable teacher model. It can be applied to general language understanding and generation tasks where resource constraints are a factor, leveraging its 1.5 billion parameters and 32K context length for efficient inference.