gardner/TinyLlama-1.1B-Instruct-3T
gardner/TinyLlama-1.1B-Instruct-3T is a 1.1 billion parameter Llama-derived instruction-tuned model, based on the TinyLlama intermediate step 1431k-3T base model. It was fine-tuned on the OpenHermes instruct dataset for four epochs, making it specifically designed as a foundational model for further fine-tuning on instruction-following tasks. This model offers a compact yet capable base for developing specialized conversational AI applications.
Loading preview...
gardner/TinyLlama-1.1B-Instruct-3T Overview
This model is a compact, instruction-tuned variant of the TinyLlama 1.1B architecture, specifically the intermediate step 1431k-3T base model. It has been fine-tuned for four epochs using the OpenHermes instruct dataset, making it suitable as a starting point for various instruction-following applications.
Key Characteristics
- Base Model: Derived from
TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T. - Parameter Count: 1.1 billion parameters, offering a lightweight solution for deployment and further development.
- Instruction Tuning: Fine-tuned on the
teknium/openhermesdataset, enhancing its ability to understand and respond to instructions. - Training Details: Trained for 4 epochs with a sequence length of 4096, utilizing LoRA (r=32, alpha=16, dropout=0.05) for efficient fine-tuning.
- Development Focus: Primarily intended as a base model for subsequent fine-tuning, allowing developers to build specialized instruction-following models.
Good For
- Further Fine-tuning: Ideal for developers looking for a pre-trained, instruction-aware base to adapt to specific domains or tasks.
- Resource-Constrained Environments: Its small size makes it suitable for applications where computational resources or memory are limited.
- Experimental AI Development: Provides a quick and accessible model for experimenting with instruction-tuned LLMs without the overhead of larger models.