TinyLlama/TinyLlama-1.1B-Chat-v0.6
TinyLlama/TinyLlama-1.1B-Chat-v0.6 is a 1.1 billion parameter Llama 2-based chat model developed by the TinyLlama project. It was fine-tuned using the Zephyr training recipe, initially on a variant of the UltraChat dataset and further aligned with DPO on the UltraFeedback dataset. This compact model is designed for chat applications requiring restricted computation and memory footprints, leveraging the Llama 2 architecture and tokenizer for broad compatibility.
Loading preview...
TinyLlama-1.1B-Chat-v0.6 Overview
TinyLlama-1.1B-Chat-v0.6 is a compact, 1.1 billion parameter language model developed by the TinyLlama project. It is built upon the Llama 2 architecture and tokenizer, ensuring compatibility with many existing open-source projects. The model was pretrained on 3 trillion tokens and subsequently fine-tuned for chat applications.
Key Capabilities
- Llama 2 Architecture: Adopts the exact architecture and tokenizer of Llama 2, allowing for seamless integration into Llama-based workflows.
- Chat Fine-tuning: Initially fine-tuned on a variant of the UltraChat dataset (synthetic dialogues generated by ChatGPT).
- DPO Alignment: Further aligned using 🤗 TRL's
DPOTraineron the openbmb/UltraFeedback dataset, which contains 64k prompts and GPT-4 ranked model completions. - Compact Size: With only 1.1 billion parameters, it is optimized for applications with limited computational resources and memory.
Good For
- Resource-constrained environments: Ideal for deployment where memory and computational power are restricted.
- Llama 2 ecosystem integration: Easily plugs into projects built around the Llama 2 framework.
- Chatbot development: Specifically fine-tuned for conversational AI tasks, following the Zephyr training recipe.
Top 3 parameter combinations used by Featherless users for this model. Click a tab to see each config.