Qwen2.5 7B Instruct for Burmese: Vocabulary Expansion

This model, developed by Atsuki Yamaguchi, is a specialized adaptation of the Qwen2.5 7B Instruct base model, fine-tuned for the Burmese language. It features a 7.6 billion parameter architecture and supports a substantial 131,072 token context length.

Key Capabilities

• Burmese Language Specialization: Optimized for processing and generating text in Burmese.
• Expanded Vocabulary: Includes an additional 10,000 target vocabulary tokens, enhancing its linguistic coverage for Burmese.
• Mean Initialization: The target weights for the embedding and LM head were initialized using a mean initialization strategy.
• Continual Pre-training: Underwent continual pre-training on 500 million Burmese language tokens sampled from the MADLAD-400 dataset.

Good for

• Applications requiring high-quality Burmese language understanding and generation.
• Research and development in low-resource language NLP, specifically for Burmese.
• Tasks such as translation, summarization, and conversational AI in Burmese.

For more technical details, refer to the associated paper: Adapting Chat Language Models Using Only Target Unlabeled Language Data.