habanoz/TinyLlama-1.1B-intermediate-step-715k-1.5T-lr-5-2.2epochs-oasst1-top1-instruct-V1

Model Overview

habanoz/TinyLlama-1.1B-intermediate-step-715k-1.5T-lr-5-2.2epochs-oasst1-top1-instruct-V1 is a compact 1.1 billion parameter instruction-tuned language model. Developed by habanoz, this model is a fine-tuned version of the TinyLlama-1.1B-intermediate-step-715k-1.5T base model, specifically trained on the high-quality OpenAssistant/oasst_top1_2023-08-25 dataset.

Key Characteristics

• Architecture: Based on the TinyLlama 1.1B parameter model.
• Training: Instruction-tuned for 2 epochs using QLoRA, with an adapter merged into the final model.
• Dataset: Utilizes the oasst1-top1 dataset, focusing on high-quality human-annotated conversational data.
• Context Length: Supports a maximum context length of 1024 tokens during training, with the base model supporting 2048 tokens.
• Performance: Achieves an average score of 35.45 on the Open LLM Leaderboard, with specific scores including:
  • HellaSwag (10-Shot): 54.40
  • TruthfulQA (0-shot): 42.34
  • Winogrande (5-shot): 57.54
  • AI2 Reasoning Challenge (25-Shot): 31.48
  • MMLU (5-Shot): 25.47
  • GSM8k (5-shot): 1.44

Use Cases

This model is suitable for applications requiring a small, efficient instruction-following model. Its training on the OpenAssistant dataset suggests capabilities in:

• General conversational AI.
• Responding to diverse prompts and instructions.
• Lightweight deployment where computational resources are limited.