TinyLlama-1.1B-Chat-v1.0 Overview

TinyLlama-1.1B-Chat-v1.0 is a compact, 1.1 billion parameter conversational model built on the Llama 2 architecture. Developed by the TinyLlama project, its primary goal was to pretrain a Llama model on an extensive 3 trillion token dataset. The model maintains the exact architecture and tokenizer of Llama 2, ensuring compatibility with existing open-source projects.

Key Capabilities & Training

• Compact Design: With only 1.1 billion parameters, it is optimized for applications with limited computational and memory resources.
• Llama 2 Compatibility: Adopts the Llama 2 architecture and tokenizer, allowing for seamless integration into Llama-based ecosystems.
• Chat Fine-tuning: This specific version is fine-tuned for chat applications, following the HF's Zephyr training recipe.
• Instruction Tuning: Initially fine-tuned on a variant of the UltraChat dataset, which provides diverse synthetic dialogues.
• Alignment with DPO: Further aligned using 🤗 TRL's DPOTrainer on the openbmb/UltraFeedback dataset, which includes 64k prompts and GPT-4 ranked model completions.

Ideal Use Cases

• Resource-Constrained Environments: Suitable for deployment where computational power or memory is limited.
• Chatbot Development: Designed specifically for conversational AI tasks due to its instruction tuning and alignment.
• Llama Ecosystem Integration: Easily integrates into projects already utilizing the Llama 2 framework.