Harvard-DCML/ADAPT-Llama3.1-4.8B-Instruct

Harvard-DCML/ADAPT-Llama3.1-4.8B-Instruct Overview

This model is a 4.8 billion parameter student model, distilled from the larger Llama-3.1-8B-Instruct, developed by Harvard-DCML. It leverages the ADAPT (Amortized Distillation Across Post-Trained LLMs) technique, which allows for flexible size interpolation between different post-trained variants of the same base model. The core innovation lies in its ability to create intermediate models by selectively patching layers from the student and teacher models.

Key Capabilities & Training

• Size Interpolation: The model facilitates the creation of custom-sized LLMs by interpolating between the 4.8B student and the 8B teacher model. This is achieved by adjusting the number of layers patched from the student model.
• Distillation Process: It was initialized by copying every other layer and the last two layers from Llama-3.1-8B-Instruct. The distillation involved 1 billion tokens from The Pile (deduplicated) and 1 billion tokens from the math split of the Llama Nemotron Post Training Dataset.
• Loss Functions: Training utilized a combination of cross-entropy, KL divergence, and cosine loss to align activations with the teacher model.
• Configurable Patching: Users can control the size of the intermediate model by specifying num_layers_to_patch using the provided build_intermediate_model function from the ADAPT GitHub repository.

Ideal Use Cases

• Research in Model Distillation: Excellent for exploring techniques in amortized distillation and understanding the trade-offs between model size and performance.
• Flexible LLM Deployment: Useful for scenarios requiring custom-sized language models that balance computational resources with desired capabilities.
• Experimentation with Llama-3.1 Variants: Provides a structured way to generate and evaluate models of varying sizes derived from the Llama-3.1-8B-Instruct base.