Harvard-DCML/ADAPT-Qwen3-8.5B
Harvard-DCML/ADAPT-Qwen3-8.5B is a 14 billion parameter student model distilled from Qwen3-14B, developed by Harvard-DCML. This model utilizes the ADAPT technique for size interpolation across post-trained variants, allowing for flexible model sizing. It was trained on 2 billion tokens from The deduplicated Pile and 2 billion tokens from the math split of the Llama Nemotron Post Training Dataset. Its primary use case is to serve as a base for creating intermediate-sized models through layer patching, offering a unique approach to model deployment and optimization.
Loading preview...
Model Overview
HARVARD-DCML/ADAPT-Qwen3-8.5B is a 14 billion parameter language model derived from Qwen3-14B. It leverages the ADAPT technique, which enables size interpolation across different post-trained variants of the same base model. This model is specifically designed as a student model, distilled from its larger Qwen3-14B counterpart.
Key Capabilities & Training
- Size Interpolation: The core innovation is the ability to create intermediate model sizes between the student (ADAPT-Qwen3-8.5B) and teacher (Qwen3-14B) models using a layer patching mechanism. This is facilitated by the
build_intermediate_modelfunction from the associated GitHub repository. - Distillation Process: The model was initialized by copying every other layer and the last two layers from Qwen3-14B. It underwent distillation on a dataset comprising 2 billion tokens from The deduplicated Pile and 2 billion tokens from the math split of the Llama Nemotron Post Training Dataset.
- Loss Functions: Distillation involved cross-entropy, KL divergence, and cosine loss to align activations with the Qwen3-14B teacher model.
- Context Length: Supports a maximum sequence length of 4096 tokens during training.
Use Cases
This model is particularly useful for researchers and developers looking to:
- Optimize Model Size: Dynamically adjust model size to balance performance and computational resources without retraining from scratch.
- Experiment with Model Scaling: Explore the performance characteristics of models at various intermediate sizes.
- Leverage Distillation Benefits: Utilize a distilled model that retains key characteristics of a larger teacher model while offering flexibility in deployment.