habanoz/TinyLlama-1.1B-intermediate-step-715k-1.5T-lr-5-4epochs-oasst1-top1-instruct-V1

Model Overview

This model, habanoz/TinyLlama-1.1B-intermediate-step-715k-1.5T-lr-5-4epochs-oasst1-top1-instruct-V1, is a 1.1 billion parameter instruction-tuned language model. It is based on the TinyLlama-1.1B-intermediate-step-715k-1.5T base model and has been fine-tuned using the OpenAssistant/oasst_top1_2023-08-25 dataset.

Training Details

The instruction tuning was performed using QLoRA (Quantized Low-Rank Adapters) with the adapter merged into the base model. Key training parameters included 4 epochs, a learning rate of 1e-5, and a model_max_len of 1024 tokens. The training utilized 4-bit quantization with nf4 type and double quantization.

Performance Benchmarks

Evaluated on the Open LLM Leaderboard, the model achieved an average score of 35.28. Specific scores include:

• AI2 Reasoning Challenge (25-Shot): 31.14
• HellaSwag (10-Shot): 54.31
• MMLU (5-Shot): 25.42
• TruthfulQA (0-shot): 41.72
• Winogrande (5-shot): 57.77
• GSM8k (5-shot): 1.29

Use Cases

Given its instruction-tuned nature and compact size, this model is suitable for applications requiring efficient, general-purpose instruction following, particularly where computational resources are limited. Its performance metrics suggest utility in tasks like common sense reasoning and question answering, though its mathematical reasoning (GSM8k) is limited.