Hugging Face Model Overview
The habanoz/TinyLlama-1.1B-intermediate-step-715k-1.5T-lr-5-1epch-airoboros3.1-1k-instruct-V1 is a 1.1 billion parameter language model derived from the TinyLlama-1.1B-intermediate-step-715k-1.5T base model. It has been instruction-tuned using the airoboros-3.1-no-mathjson-max-1k dataset, leveraging QLoRA for efficient fine-tuning, with the adapter subsequently merged into the base model.
Key Capabilities & Performance
This model demonstrates general instruction-following abilities, as indicated by its performance on the Open LLM Leaderboard. Key evaluation metrics include:
- Avg. Score: 34.98
- AI2 Reasoning Challenge (25-Shot): 30.72
- HellaSwag (10-Shot): 54.32
- MMLU (5-Shot): 24.78
- TruthfulQA (0-shot): 41.67
- Winogrande (5-shot): 57.62
- GSM8k (5-shot): 0.76
Training Details
The model was fine-tuned for 1 epoch with a learning rate of 1e-5, utilizing 4-bit quantization (nf4) and double quantization. It employs a model_max_len of 1024 tokens and incorporates Flash Attention 2 for improved efficiency. The training process involved a per_device_train_batch_size of 4 and gradient_accumulation_steps of 4.
Use Cases
Given its instruction-tuned nature and performance on various benchmarks, this model is suitable for general-purpose conversational AI, text generation, and tasks requiring basic reasoning and common sense understanding, particularly in resource-constrained environments due to its small size.