RefalMachine/RuadaptQwen2.5-1.5B-instruct
RefalMachine/RuadaptQwen2.5-1.5B-instruct is a 1.5 billion parameter instruction-tuned language model based on the Qwen2.5 architecture, adapted for the Russian language by RefalMachine. It features a replaced tokenizer and continued pretraining on a Russian corpus, followed by the application of Learned Embedding Propagation (LEP). This adaptation significantly improves the generation speed of Russian texts by up to 60% compared to the original Qwen-2.5-1.5B-Instruct model, making it optimized for Russian language processing tasks.
Loading preview...
Overview
RefalMachine/RuadaptQwen2.5-1.5B-instruct is an instruction-tuned variant of the Qwen2.5-1.5B model, specifically adapted for the Russian language. Developed by RefalMachine, this model underwent a multi-stage adaptation process to enhance its performance and efficiency for Russian text generation.
Key Adaptations and Features
- Tokenizer Replacement: The original tokenizer was replaced with an extended tiktoken cl100k tokenizer, further enhanced with a unigram tokenizer to include 48,000 tokens relevant to Russian.
- Continued Pretraining: The model was subjected to continued pretraining on a substantial Russian language corpus.
- Learned Embedding Propagation (LEP): This technique was applied post-pretraining to further refine the model's understanding and generation capabilities in Russian.
- Improved Russian Generation Speed: A primary benefit of these adaptations is a reported increase in Russian text generation speed by up to 60% compared to the base Qwen-2.5-1.5B-Instruct model, measured by characters/words per second on identical text sequences.
Research and Citation
The adaptation process involved research on tokenization and embedding propagation, with relevant papers including "Facilitating large language model Russian adaptation with Learned Embedding Propagation" (Tikhomirov M., Chernyshev D., 2024) and "Impact of Tokenization on LLaMa Russian Adaptation" (Tikhomirov M., Chernyshev D., 2023).
Limitations and Usage
As an adapted model, its responses reflect knowledge acquired from its training data. Users should exercise caution, as the control over the original pretrained model's content is not the responsibility of the current authors.