Overview
RefalMachine/RuadaptQwen2.5-14B-Instruct is an instruction-tuned variant of the Qwen2.5-14B model, specifically adapted for the Russian language. Developed by RefalMachine, this model incorporates a new tokenizer and undergoes continued pretraining on a substantial Russian corpus. A key innovation is the application of Learned Embedding Propagation (LEP) to further enhance its capabilities.
Key Adaptations & Features
- Tokenizer Replacement: The original tokenizer was replaced with an extended tiktoken cl100k, utilizing a unigram tokenizer with 48,000 tokens, specifically optimized for Russian.
- Continued Pretraining: The model underwent additional pretraining on a Russian-language dataset to improve its understanding and generation of Russian text.
- Learned Embedding Propagation (LEP): This technique was applied post-pretraining to further refine the model's performance.
- Enhanced Russian Generation Speed: Due to the specialized tokenizer, the model demonstrates up to a 60% increase in Russian text generation speed (characters/words per second) compared to the base Qwen-2.5-14B-Instruct model on identical text sequences.
Current Status & Evaluation
This model is currently a work in progress (v1). Evaluation is planned or ongoing across several benchmarks including Ru-Arena-General, MERA, and llmtf_open. Preliminary measurements on Ru-Arena-General were conducted with a repetition_penalty=1.1. Custom system prompts were prepared for MERA submissions to mitigate issues with code-related tasks.
Good for
- Applications requiring efficient and high-quality Russian text generation.
- Developers looking for a Qwen2.5-based model with strong Russian language capabilities.
- Use cases where generation speed for Russian content is a critical factor.