eulogik/Bharat-Tiny-LLM-fused
Bharat-Tiny-LLM-fused by eulogik is a 1.5 billion parameter language model, based on the Qwen2.5 architecture, with a 32768 token context length. This full-precision fused model integrates a LoRA adapter into the base Qwen2.5-1.5B weights, provided in PyTorch float16. It is specifically designed for server-side inference, further fine-tuning, or for producing custom quantized builds like GGUF and MLX. The model is optimized for scenarios requiring a balance of performance and flexibility in deployment.
Loading preview...
Overview
This repository hosts the Bharat-Tiny-LLM-fused model, a 1.5 billion parameter language model developed by eulogik. It is a full-precision (PyTorch float16) version of the Bharat-Tiny-LLM, where a LoRA adapter has been merged into the base Qwen2.5-1.5B weights. This fused model is distinct from its quantized counterparts, offering maximum flexibility for developers.
Key Capabilities
- Full-Precision Inference: Suitable for running inference on CPU/CUDA using the
transformerslibrary. - Fine-Tuning Base: Provides a robust foundation for further fine-tuning on specific datasets or tasks.
- Custom Quantization: Ideal for generating custom quantized builds, such as GGUF for
llama.cppor MLX for Apple Silicon, tailored to specific deployment needs. - Qwen2.5 Architecture: Leverages the capabilities of the Qwen2.5 model family.
Good For
- Server-side deployments: When computational resources allow for full-precision models.
- Developers requiring maximum control: For those who need to fine-tune the model or create their own optimized quantized versions.
- Experimentation: Provides a flexible base for research and development.
Important Considerations
When using this model with transformers, specific generation parameters are recommended to avoid garbled output, including temperature ≈ 0.3, repetition_penalty ≥ 1.25, and no_repeat_ngram_size = 3.