pkupie/Qwen2.5-3B-kk-cpt
TEXT GENERATIONConcurrency Cost:1Model Size:3.1BQuant:BF16Ctx Length:32kPublished:Apr 28, 2026License:apache-2.0Architecture:Transformer Open Weights Cold
pkupie/Qwen2.5-3B-kk-cpt is a 3.1 billion parameter Qwen2.5-based language model continually pretrained on the Kazakh (Arabic Script) portion of the MC^2 Corpus. This model is specifically adapted for improved Kazakh language modeling and supports research in low-resource language adaptation. It serves as a specialized base model for tasks like model merging and logit fusion, particularly for Kazakh language applications.
Loading preview...
Model Overview
pkupie/Qwen2.5-3B-kk-cpt is a specialized language model built upon the Qwen2.5-3B architecture. It has undergone continual pretraining (CPT) specifically on the Kazakh (Arabic Script) subset of the MC^2 Corpus.
Key Capabilities & Purpose
- Enhanced Kazakh Language Modeling: The primary goal of this model is to significantly improve performance in Kazakh language understanding and generation, particularly for text written in Arabic script.
- Low-Resource Language Adaptation: It serves as a valuable resource for research focused on adapting large language models to languages with limited available data.
- Base for Further Research: This checkpoint is intended to be used as a foundational model for advanced research in areas such as model merging and logit fusion techniques, as detailed in the paper "Efficient Low-Resource Language Adaptation via Multi-Source Dynamic Logit Fusion" (ACL 2026).
Intended Use Cases
- Academic Research: Ideal for researchers exploring methods for low-resource language adaptation and multilingual model development.
- Kazakh NLP Development: Can be fine-tuned or integrated into applications requiring strong Kazakh language capabilities.
- Model Merging & Fusion Experiments: Provides a robust base for experimenting with combining models or logits for improved performance in specific linguistic contexts.