eulogik/Bharat-Tiny-LLM-v3.4
Bharat-Tiny-LLM-v3.4 by eulogik is an experimental 2 billion parameter language model, based on Bharat-Tiny-LLM-v3, with a 32768 token context length. This version is a mixed math and chat SFT, fine-tuned using LoRA, though its quality gates for GSM8K-Hindi and chat performance were not met. It is primarily kept for reproducibility, with eulogik/Bharat-Tiny-LLM-v3 recommended for production use.
Loading preview...
Model Overview
The eulogik/Bharat-Tiny-LLM-v3.4 is an experimental 2 billion parameter language model developed by eulogik, building upon the eulogik/Bharat-Tiny-LLM-v3 base model. This iteration was fine-tuned using a mixed math and chat Supervised Fine-Tuning (SFT) dataset, comprising 43,687 gold rows from eulogik/bharat-v3-mixed-sft-v34.
Training Details
The model was trained using LoRA (r=16, α=32) for one epoch with a learning rate of 2e-5 and a sequence length of 1024. The training process, conducted on 2×T4 DataParallel on Kaggle, showed a reduction in train loss from 1.82 to 1.23 over 611 minutes. The resulting LoRA adapters were merged into an fp16 model.
Current Status and Recommendations
It is important to note that Bharat-Tiny-LLM-v3.4 failed its quality gates, specifically exhibiting significantly weaker performance in GSM8K-Hindi (~13% compared to v3's 56%) and general chat capabilities. Consequently, this version is considered superseded and is maintained primarily for reproducibility purposes. For production environments, eulogik recommends using eulogik/Bharat-Tiny-LLM-v3 and its associated v3.3 chat GGUF models.