bxod/Llama-3.2-1B-Instruct-uz
The bxod/Llama-3.2-1B-Instruct-uz is an experimental 1.2 billion parameter Llama-3.2-based instruction-tuned model, continually pretrained on a 2048-token context length with 80% English and 20% Uzbek data. It features a customized tokenizer that averages 1.7 tokens per Uzbek word, significantly improving inference speed and effective context length for Uzbek text. This model is optimized for Uzbek language tasks, particularly translation and sentiment analysis, and is designed for efficient deployment on devices with limited VRAM (as low as 2 GB with quantization).
Loading preview...
Model Overview
The bxod/Llama-3.2-1B-Instruct-uz is an experimental 1.2 billion parameter instruction-tuned model based on the Llama-3.2 architecture. It has been continually pretrained with a 2048-token context length, utilizing a dataset composed of 80% English and 20% Uzbek tokens. A key innovation is its customized tokenizer, which achieves an average of 1.7 tokens per Uzbek word, leading to approximately 2x faster inference and a longer effective context length for Uzbek text compared to original Llama models.
Key Capabilities & Performance
This model demonstrates significant improvements in Uzbek language tasks:
- Enhanced Uzbek Translation: Outperforms its base Llama-3.2 1B Instruct counterpart in both Uzbek-to-English and English-to-Uzbek translation benchmarks (BLEU and COMET scores).
- Improved Uzbek Sentiment Analysis: Shows better performance on Uzbek sentiment analysis tasks.
- Efficient Deployment: Designed to run on devices with as little as 2 GB of VRAM when quantized, making it suitable for small GPUs, edge devices, and mobile applications.
Limitations
While excelling in Uzbek-specific tasks, the model exhibits a slight decline in MMLU (English) and Uzbek News Classification scores due to catastrophic forgetting of original English instruction following, as noted by the developers. This model is considered an experimental checkpoint with room for further improvement.
Usage Notes
For optimal performance with Uzbek input, it is recommended to preprocess text by replacing apostrophes (') with the sequence "APST" to leverage the model's lower tokenizer fertility.