TinyLLama_0.6_Chat_BF16 is a 1.1 billion parameter Llama 2-based chat model developed by the TinyLlama project, fine-tuned using the Zephyr training recipe. This compact model is designed for conversational AI, leveraging a variant of the UltraChat dataset for initial fine-tuning and further alignment with DPO on the UltraFeedback dataset. Its small size makes it suitable for applications with restricted computational and memory footprints.
Loading preview...
TinyLLama_0.6_Chat_BF16: A Compact Llama 2-based Chat Model
This model, TinyLLama_0.6_Chat_BF16, is a chat-finetuned version of the TinyLlama 1.1B model, which was pretrained on 3 trillion tokens. Developed by the TinyLlama project, it adopts the exact architecture and tokenizer of Llama 2, ensuring compatibility with existing Llama-based open-source projects.
Key Capabilities & Training
- Architecture: Based on the Llama 2 architecture with 1.1 billion parameters, making it highly compact.
- Pretraining: The base model was pretrained on an extensive 3 trillion tokens.
- Fine-tuning: This specific chat model was fine-tuned following the Hugging Face Zephyr training recipe.
- Dataset Utilization:
- Initially fine-tuned on a variant of the
UltraChatdataset, comprising synthetic dialogues generated by ChatGPT. - Further aligned using 🤗 TRL's
DPOTraineron theopenbmb/UltraFeedbackdataset, which includes 64k prompts and GPT-4 ranked model completions.
- Initially fine-tuned on a variant of the
Good For
- Conversational AI: Optimized for generating human-like responses in chat-based applications.
- Resource-Constrained Environments: Its compact 1.1B parameter size makes it ideal for deployment where computational power and memory are limited.
- Llama 2 Ecosystem Integration: Seamlessly integrates into projects built upon the Llama 2 framework due to its identical architecture and tokenizer.