Harvard-DCML/boomerang-qwen3-4.9B

TEXT GENERATIONConcurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Oct 10, 2025License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

Harvard-DCML/boomerang-qwen3-4.9B is an 8 billion parameter student model distilled from Qwen3-8B-Base, developed by Harvard-DCML. This model showcases the 'Boomerang distillation' phenomenon, allowing for the creation of intermediate-sized models without additional training by reincorporating teacher layers. It was distilled on 2.1 billion tokens using a combination of cross entropy, KL, and cosine loss to match the activations of its teacher model. Its primary utility lies in enabling zero-shot model size interpolation, offering flexibility in model deployment.

Loading preview...

Boomerang Qwen3-4.9B: Distilled for Flexible Model Sizing

HARVARD-DCML/boomerang-qwen3-4.9B is an 8 billion parameter student model derived from Qwen3-8B-Base, demonstrating the novel "Boomerang distillation" technique. This method allows for the creation of intermediate-sized models by selectively reincorporating layers from a larger teacher model into a smaller student model, all without requiring further training.

Key Characteristics & Training:

  • Distillation Process: Initialized from Qwen3-8B-Base by copying every other layer and the last two layers.
  • Training Data: Distilled on 2.1 billion tokens from The Pile, which was deduplicated.
  • Loss Functions: Utilized a combination of cross entropy, KL divergence, and cosine loss to align activations with the Qwen3-8B-Base teacher model.
  • Zero-Shot Interpolation: The core innovation is its ability to enable zero-shot model size interpolation. Developers can use the build_intermediate_model function from the dcml-lab/boomerang-distillation GitHub repository to create models of varying sizes between this student model and the Qwen3-8B-Base teacher.

Use Cases:

  • Model Size Optimization: Ideal for scenarios requiring flexible model sizes, allowing users to fine-tune the computational footprint and performance trade-offs without extensive retraining.
  • Research in Distillation: A valuable resource for researchers exploring advanced distillation techniques and model interpolation.

Finetunes
1 models