huggingFacing/qwen2.5-7b-to-1.5b-liftkd-v12-head-only-paper100k-step1500-seed10

Model Overview

This model, huggingFacing/qwen2.5-7b-to-1.5b-liftkd-v12-head-only-paper100k-step1500-seed10, is a 1.5 billion parameter student model resulting from a knowledge distillation process. It was distilled from the larger Qwen2.5 7B-Instruct teacher model, with Qwen2.5-1.5B-Instruct serving as the student's initialization.

Key Characteristics

• Knowledge Distillation: Employs a specific lift_v12_head_only checkpoint, which focuses on a 'gap gate with step-level online LM-head influence weights only' variant of the distillation method.
• Training Data: Trained on 96,000 examples from the lift_paper_en_natural_v1/100k dataset, with an additional 2,000-example controller split.
• Training Protocol: Utilizes a fully on-policy GKD (Generative Knowledge Distillation) objective over 1,500 optimizer steps, with a global batch size of 64.
• Efficiency: As a 1.5B parameter model, it offers a significantly smaller footprint compared to its 7B parameter teacher, making it suitable for applications requiring lower computational resources.

Use Cases

This model is particularly well-suited for scenarios where the performance of a larger Qwen2.5 model is desired but with the efficiency and reduced resource requirements of a 1.5 billion parameter model. Its distillation method suggests potential for retaining key capabilities of the teacher model in a more compact form, making it ideal for deployment in resource-constrained environments or for tasks where a smaller, faster model is advantageous.