pkupie/Qwen2.5-3B-mn-cpt
pkupie/Qwen2.5-3B-mn-cpt is a 3.1 billion parameter Qwen2.5-based language model continually pretrained on the Mongolian (Traditional Mongolian Script) portion of the MC^2 Corpus. Developed by pkupie, this model is specifically adapted for improved language modeling in low-resource Mongolian. It is intended for research in low-resource language adaptation, particularly as a base for model merging and logit fusion.
Loading preview...
Overview
pkupie/Qwen2.5-3B-mn-cpt is a specialized language model built upon the Qwen2.5 3B architecture. It has undergone continual pretraining (CPT) using the Mongolian (Traditional Mongolian Script) subset of the MC^2 Corpus. This adaptation aims to enhance its performance and understanding of the Mongolian language, particularly in its traditional script form.
Key Capabilities
- Low-Resource Language Adaptation: Specifically fine-tuned for the Mongolian language, addressing challenges in low-resource linguistic contexts.
- Research Base Model: Designed as a foundational checkpoint for further research, especially in areas like model merging and dynamic logit fusion.
- Qwen2.5 Architecture: Leverages the robust Qwen2.5 3B base model, providing a strong starting point for specialized applications.
Intended Use Cases
- Linguistic Research: Ideal for researchers studying low-resource language modeling and adaptation techniques.
- Model Development: Serves as a base model for developing more advanced Mongolian language processing applications.
- Experimental AI: Suitable for experiments involving model merging and logit fusion methodologies, as detailed in the associated research paper: "Efficient Low-Resource Language Adaptation via Multi-Source Dynamic Logit Fusion".