TinySwallow-1.5B: A Compact Japanese Language Model

TinySwallow-1.5B is a 1.5 billion parameter autoregressive language model developed collaboratively by Sakana AI and the Swallow Team. Its creation leverages a new knowledge distillation technique called Temporally Adaptive Interpolated Distillation (TAID). This method utilized Qwen2.5-32B-Instruct as the teacher model and Qwen2.5-1.5B-Instruct as the student, focusing on efficient knowledge transfer.

Key Characteristics:

• Japanese Language Focus: The model has been specifically pre-trained on extensive Japanese text data to significantly enhance its proficiency in the language.
• Distillation Method: Employs TAID for efficient model compression and performance retention from a larger teacher model.
• Base Architecture: Built upon the Qwen2.5-1.5B-Instruct architecture, further refined for Japanese contexts.

Intended Use:

This model is primarily provided for research and development purposes as an experimental prototype. It is not intended for commercial deployment or use in mission-critical applications. Users should be aware of its experimental nature and use it at their own discretion, understanding that performance and outcomes are not guaranteed. For an instruction-following variant, users are directed to TinySwallow-1.5B-Instruct.