Harvard-DCML/ADAPT-Qwen3-2.3B-Instruct
The Harvard-DCML/ADAPT-Qwen3-2.3B-Instruct is a 4 billion parameter instruction-tuned causal language model, distilled from Qwen3-4B-Instruct-2507 using the ADAPT technique. This model is specifically designed for size interpolation, allowing developers to create intermediate model sizes between the student and teacher models. It is optimized for scenarios requiring flexible model sizing while maintaining performance characteristics of its larger teacher model.
Loading preview...
Model Overview
The Harvard-DCML/ADAPT-Qwen3-2.3B-Instruct is a 4 billion parameter student model, distilled from the Qwen3-4B-Instruct-2507 teacher model. It leverages the ADAPT technique, which enables size interpolation across different post-trained variants of the same base model. This model was created by copying every other layer and the last two layers from its teacher, then distilled on 0.5 billion tokens from The Pile and 0.5 billion tokens from the Llama Nemotron Post Training Dataset's math split.
Key Capabilities
- Size Interpolation: Facilitates the creation of intermediate model sizes between the student and teacher models using the
build_intermediate_modelfunction from the ADAPT GitHub repository. - Efficient Distillation: Achieved through a combination of cross-entropy, KL, and cosine loss to match the activations of the larger Qwen3-4B-Instruct-2507.
- Flexible Deployment: Offers a smaller footprint while retaining characteristics of a larger model, suitable for environments with varying computational constraints.
When to Use This Model
This model is particularly useful for researchers and developers who need to:
- Experiment with different model sizes without retraining from scratch.
- Optimize model deployment for specific hardware or latency requirements by interpolating to an ideal size.
- Explore the trade-offs between model size and performance in a controlled manner.