Harvard-DCML/ADAPT-Qwen3-2.3B-Base
Harvard-DCML/ADAPT-Qwen3-2.3B-Base is a 2.3 billion parameter student model distilled from Qwen3-4B-Base, developed by Harvard-DCML. This model utilizes the ADAPT technique for size interpolation across post-trained variants of the same base model. It is specifically designed to enable flexible model sizing and performance tuning through weight-delta transfer, making it suitable for research into efficient LLM deployment and adaptation.
Loading preview...
Model Overview
The Harvard-DCML/ADAPT-Qwen3-2.3B-Base is a 2.3 billion parameter student model derived from the Qwen3-4B-Base using the ADAPT (Amortized Distillation Across Post-Trained LLMs) technique. This method allows for the creation of models with interpolated sizes from existing base models, as detailed in the associated research paper.
Key Capabilities
- Size Interpolation: The primary feature of this model is its ability to be used in conjunction with the ADAPT technique to create intermediate models of varying sizes. This is achieved by patching different numbers of student layers.
- Distillation from Qwen3-4B-Base: The model was initialized by copying specific layers from the Qwen3-4B-Base and then distilled on a diverse dataset including 0.5 billion tokens from The Pile (deduplicated) and 0.5 billion tokens from the math split of the Llama Nemotron Post Training Dataset.
- Weight-Delta Transfer: It is designed to facilitate weight-delta transfer, allowing for interpolation between this student model and post-trained variants of the Qwen3-4B-Base, offering flexibility in model deployment and fine-tuning.
Training Details
The distillation process involved using cross-entropy, KL divergence, and cosine loss to match the activations of the larger Qwen3-4B-Base. Key hyperparameters included a learning rate of 3e-4, cosine learning rate scheduler, and an effective batch size of 2048 during pre-training. The model supports a maximum sequence length of 1024 tokens and uses bf16 mixed precision.
Use Cases
This model is particularly useful for researchers and developers interested in:
- Efficient LLM Deployment: Exploring methods to create smaller, more efficient models from larger, pre-trained ones without significant performance degradation.
- Model Adaptation and Customization: Utilizing the ADAPT technique to fine-tune model size and performance for specific applications by interpolating between different post-trained variants.
- Research into Distillation Techniques: Investigating advanced distillation methods for large language models.