amachree/TinyLlama-1.1B-Chat-v1.0
amachree/TinyLlama-1.1B-Chat-v1.0 is a 1.1 billion parameter Llama-architecture chat model, fine-tuned from the TinyLlama base model. It was developed by amachree, following the Zephyr training recipe, and is designed for conversational applications with a compact computational and memory footprint. This model is optimized for chat interactions, leveraging synthetic dialogues and ranked model completions for alignment.
Loading preview...
TinyLlama-1.1B-Chat-v1.0 Overview
This model is a chat-tuned variant of the TinyLlama project, which aims to pretrain a 1.1 billion parameter Llama model on 3 trillion tokens. Developed by amachree, it adopts the exact same architecture and tokenizer as Llama 2, ensuring compatibility with existing Llama-based open-source projects. Its compact size makes it suitable for applications with restricted computational and memory resources.
Key Capabilities & Training
- Llama 2 Architecture: Inherits the Llama 2 architecture and tokenizer, allowing for seamless integration into many existing projects.
- Chat Fine-tuning: Fine-tuned using a recipe similar to Hugging Face's Zephyr, initially on a variant of the UltraChat dataset (synthetic dialogues generated by ChatGPT).
- Alignment: Further aligned using 🤗 TRL's
DPOTraineron the openbmb/UltraFeedback dataset, which contains 64k prompts and GPT-4 ranked model completions. - Compact Size: With only 1.1 billion parameters, it is designed for efficiency and deployment in environments with limited resources.
Good For
- Conversational AI: Excels in chat-based interactions due to its specific fine-tuning on dialogue datasets.
- Resource-Constrained Environments: Ideal for applications requiring a small model footprint and efficient inference.
- Llama Ecosystem Integration: Easily integrates into projects built around the Llama architecture.