Savoxism/amid-kd-8xh200-qwen2.5-0.5b-it-amid

Overview

Savoxism/amid-kd-8xh200-qwen2.5-0.5b-it-amid is a 0.5 billion parameter instruction-tuned model, distilled from the base model Qwen/Qwen2.5-0.5B-Instruct. It was created using the AMiD (Adaptive Multi-Teacher Distillation) method, with Qwen/Qwen3-4B-Instruct-2507 serving as the teacher model. The distillation process involved training on 79,751 teacher-generated responses from the VoCuc/UltraInteract-Infer dataset.

Training Details

The model underwent a full fine-tune over 2 epochs, totaling 2,492 steps, using a learning rate of 1e-4 and a global batch size of 64. The training utilized 8 H200 GPUs with DeepSpeed bf16. The maximum sequence length was 1025 tokens, with a maximum prompt length of 512.

Key Characteristics and Limitations

This model is a research checkpoint and exhibits significant limitations. Evaluation metrics show a degradation in performance during training, with dev loss increasing and MMLU-STEM accuracy falling below chance level (0.25). Generations often fall into repetition loops. No baseline evaluation was performed, so direct comparisons to the untrained student or teacher are unavailable. Due to these issues, the model is not recommended for use as a capable assistant.

Usage

For those interested in exploring this research checkpoint, the model can be loaded using the Hugging Face transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Savoxism/amid-kd-8xh200-qwen2.5-0.5b-it-amid", torch_dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("Savoxism/amid-kd-8xh200-qwen2.5-0.5b-it-amid")

This model inherits the Apache 2.0 license from its base model, Qwen/Qwen2.5-0.5B-Instruct.