5ivatej/qwen2.5-1.5B-india-finetuned
The 5ivatej/qwen2.5-1.5B-india-finetuned model is a 1.5 billion parameter Qwen2.5-based causal language model, fine-tuned by 5ivatej. It is specifically optimized for instruction-following in Indic languages, including Kannada, Hindi, Tamil, Telugu, Marathi, and Gujarati, alongside English instructions. This model leverages LoRA fine-tuning on Indic instruction datasets to provide a standalone checkpoint for language generation tasks. It is designed for applications requiring compact, efficient processing of Indic language prompts.
Loading preview...
Overview
This model, 5ivatej/qwen2.5-1.5B-india-finetuned, is a 1.5 billion parameter variant of the Qwen2.5 architecture, fine-tuned by 5ivatej. It has been specifically adapted using LoRA (Low-Rank Adaptation) on small instruction-following datasets focused on Indic languages. The LoRA adapters are merged into the base weights, resulting in a standalone checkpoint that can be directly utilized, particularly with mlx_lm.
Key Capabilities
- Indic Language Instruction Following: Optimized for understanding and generating responses in multiple Indic languages, including Kannada, Hindi, Tamil, Telugu, Marathi, and Gujarati, in addition to English.
- Compact Size: At 1.5 billion parameters, it offers a relatively small footprint for efficient deployment.
- Standalone Checkpoint: The fine-tuned model is provided as a merged checkpoint, simplifying its integration and usage.
Training Details
The model was fine-tuned using LoRA-SFT, targeting both attention and MLP layers, with specific hyperparameters (r=16, alpha=32, dropout=0.05). Training involved 1500 steps with a batch size of 1 and a maximum sequence length of 1024, utilizing subsets of the ai4bharat/indic-align dataset (Dolly_T and Anudesh) converted into completion-style prompts. The training was performed on Apple Silicon hardware using the mlx_lm framework.
Good For
- Applications requiring instruction-following capabilities in Indic languages.
- Deployment on resource-constrained devices, given its 1.5B parameter count.
- Research and development involving multilingual models with a focus on the Indian subcontinent's languages.