Harvard-DCML/ADAPT-Qwen3-8.5B-Base
Harvard-DCML/ADAPT-Qwen3-8.5B-Base is an 8.5 billion parameter student model distilled from Qwen3-14B-Base, developed by Harvard-DCML. This model utilizes the ADAPT technique for size interpolation across post-trained variants, making it suitable for creating intermediate models with varying layer counts. It was trained on 4 billion tokens from The Pile and Llama Nemotron Post Training Dataset, focusing on matching activations of the larger Qwen3-14B-Base model. Its primary use is as a base for generating custom-sized models through weight-delta transfer for specific deployment needs.
Loading preview...
Model Overview
The Harvard-DCML/ADAPT-Qwen3-8.5B-Base is an 8.5 billion parameter student model derived from the larger Qwen3-14B-Base. Developed by Harvard-DCML, this model is a key component of the ADAPT technique, which enables size interpolation across different post-trained variants of a base model. This allows developers to create custom-sized models by adjusting the number of layers.
Training and Distillation
The model was initialized by selectively copying layers from Qwen3-14B-Base (every other layer and the last two). It underwent a distillation process on a combined dataset of 2 billion tokens from The deduplicated Pile and 2 billion tokens from the math split of the Llama Nemotron Post Training Dataset. The distillation employed cross-entropy, KL divergence, and cosine loss to align its activations with those of the Qwen3-14B-Base teacher model. Key training hyperparameters included a learning rate of 3e-4, AdamW optimizer, and a maximum sequence length of 4096.
Key Capabilities and Use Cases
This model's primary utility lies in its role within the ADAPT framework for generating intermediate models. Developers can use the provided build_intermediate_model function from the Harvard-DCML GitHub repository to perform weight-delta transfer. This process allows for the creation of models with varying layer counts, offering flexibility in model size and performance trade-offs. It is particularly useful for researchers and practitioners looking to experiment with different model sizes derived from a strong base model without retraining from scratch.