Overview
RefalMachine/RuadaptQwen2.5-32B-Pro-Beta is a 32.8 billion parameter language model, representing an adaptation of the T-pro-it-1.0 model for the Russian language. Developed by RefalMachine, this model underwent a multi-stage adaptation process. Initially, its tokenizer was replaced with an extended tiktoken cl100k tokenizer, enhanced with a unigram tokenizer for 48,000 tokens. This was followed by continued pretraining on a substantial Russian corpus. The final stage involved the application of Learned Embedding Propagation (LEP) techniques.
Key Adaptations & Performance
The primary differentiator of this model is its specialized optimization for Russian. The new tokenizer and subsequent pretraining have led to a notable improvement in Russian text generation speed, reportedly up to 60% faster compared to the original T-pro-it-1.0 model when generating Russian text. This speedup is measured by the quantity of Russian characters/words generated per second on identical text sequences.
Evaluation
The model's performance has been evaluated on several Russian-specific benchmarks, including Ru-Arena-General, MERA, and llmtf_open. Preliminary results on Ru-Arena-General, measured with repetition_penalty=1.1, indicate its capabilities in Russian language tasks. Further evaluation results for MERA and llmtf_open are pending.
Good for
- Applications requiring high-speed Russian text generation.
- Projects focused on Russian natural language processing where efficiency is critical.
- Developers seeking a Qwen2.5-based model with enhanced Russian language capabilities.