Hugging Face TinyLlama-1.1B-Chat-v1.0 is a 1.1 billion parameter Llama 2-based conversational model developed by the TinyLlama project, trained on 3 trillion tokens with a 2048-token context length. This compact model is fine-tuned using the Zephyr training recipe, leveraging UltraChat for initial instruction tuning and UltraFeedback with DPO for alignment. It is designed for chat applications requiring a restricted computation and memory footprint.
TinyLlama-1.1B-Chat-v1.0 Overview
TinyLlama-1.1B-Chat-v1.0 is a compact, 1.1 billion parameter conversational model built on the Llama 2 architecture. Developed by the TinyLlama project, its primary goal was to pretrain a Llama model on an extensive 3 trillion token dataset. The model maintains the exact architecture and tokenizer of Llama 2, ensuring compatibility with existing open-source projects.
Key Capabilities & Training
- Compact Design: With only 1.1 billion parameters, it is optimized for applications with limited computational and memory resources.
- Llama 2 Compatibility: Adopts the Llama 2 architecture and tokenizer, allowing for seamless integration into Llama-based ecosystems.
- Chat Fine-tuning: This specific version is fine-tuned for chat applications, following the HF's Zephyr training recipe.
- Instruction Tuning: Initially fine-tuned on a variant of the
UltraChatdataset, which provides diverse synthetic dialogues. - Alignment with DPO: Further aligned using 🤗 TRL's
DPOTraineron the openbmb/UltraFeedback dataset, which includes 64k prompts and GPT-4 ranked model completions.
Ideal Use Cases
- Resource-Constrained Environments: Suitable for deployment where computational power or memory is limited.
- Chatbot Development: Designed specifically for conversational AI tasks due to its instruction tuning and alignment.
- Llama Ecosystem Integration: Easily integrates into projects already utilizing the Llama 2 framework.