Hugging Face The habanoz/TinyLlama-1.1B-intermediate-step-715k-1.5T-lr-5-4epochs-oasst1-top1-instruct-V1 is a 1.1 billion parameter instruction-tuned causal language model, fine-tuned from TinyLlama-1.1B-intermediate-step-715k-1.5T. Developed by habanoz, this model was instruction-tuned using the OpenAssistant/oasst_top1_2023-08-25 dataset with QLoRA. It is designed for general instruction-following tasks, offering a compact solution for applications requiring a smaller, efficient language model.
Model Overview
This model, habanoz/TinyLlama-1.1B-intermediate-step-715k-1.5T-lr-5-4epochs-oasst1-top1-instruct-V1, is a 1.1 billion parameter instruction-tuned language model. It is based on the TinyLlama-1.1B-intermediate-step-715k-1.5T base model and has been fine-tuned using the OpenAssistant/oasst_top1_2023-08-25 dataset.
Training Details
The instruction tuning was performed using QLoRA (Quantized Low-Rank Adapters) with the adapter merged into the base model. Key training parameters included 4 epochs, a learning rate of 1e-5, and a model_max_len of 1024 tokens. The training utilized 4-bit quantization with nf4 type and double quantization.
Performance Benchmarks
Evaluated on the Open LLM Leaderboard, the model achieved an average score of 35.28. Specific scores include:
- AI2 Reasoning Challenge (25-Shot): 31.14
- HellaSwag (10-Shot): 54.31
- MMLU (5-Shot): 25.42
- TruthfulQA (0-shot): 41.72
- Winogrande (5-shot): 57.77
- GSM8k (5-shot): 1.29
Use Cases
Given its instruction-tuned nature and compact size, this model is suitable for applications requiring efficient, general-purpose instruction following, particularly where computational resources are limited. Its performance metrics suggest utility in tasks like common sense reasoning and question answering, though its mathematical reasoning (GSM8k) is limited.