Hugging Face Model Overview
This model, habanoz/TinyLlama-1.1B-intermediate-step-715k-1.5T-lr-5-3epochs-oasst1-top1-instruct-V1, is a 1.1 billion parameter instruction-tuned language model. It was fine-tuned by habanoz from the TinyLlama-1.1B-intermediate-step-715k-1.5T base model using the OpenAssistant/oasst_top1_2023-08-25 dataset. The fine-tuning process utilized QLoRA, and the adapter was subsequently merged into the base model.
Key Capabilities
- Instruction Following: Optimized for understanding and responding to user instructions, leveraging the Oasst1 dataset for alignment.
- General Reasoning: Exhibits capabilities in various reasoning tasks, as reflected in its performance on benchmarks like AI2 Reasoning Challenge and HellaSwag.
- Compact Size: At 1.1 billion parameters, it offers a balance between performance and computational efficiency, making it suitable for resource-constrained environments.
Performance Metrics
The model's performance on the Open LLM Leaderboard indicates an average score of 35.42. Specific benchmark results include:
- AI2 Reasoning Challenge (25-Shot): 31.40
- HellaSwag (10-Shot): 54.24
- MMLU (5-Shot): 25.36
- TruthfulQA (0-shot): 42.47
- Winogrande (5-shot): 57.70
- GSM8k (5-shot): 1.36
Training Details
The model was trained for 3 epochs with a learning rate of 1e-5, using a batch size of 4 and gradient accumulation steps of 4. It incorporates 4-bit quantization with NF4 and double quantization, along with Flash Attention 2 for efficiency. The full training script and parameters are available in the associated GitHub repository.