dogma-black/TinyLlama-1.1B-Chat-v1.0 is a 1.1 billion parameter Llama-based chat model, pretrained on 3 trillion tokens and fine-tuned using the Zephyr training recipe. It utilizes the same architecture and tokenizer as Llama 2, making it compatible with existing Llama-based projects. This compact model is designed for applications requiring restricted computation and memory footprints, excelling in conversational tasks.
Loading preview...
TinyLlama-1.1B-Chat-v1.0 Overview
This model is a chat-finetuned version of the TinyLlama 1.1 billion parameter language model, developed by the TinyLlama project. The base model was pretrained on an extensive 3 trillion tokens, adopting the exact architecture and tokenizer of Llama 2. This design choice ensures broad compatibility with open-source projects built upon the Llama ecosystem.
Key Capabilities
- Compact Size: With only 1.1 billion parameters, it is optimized for environments with limited computational resources and memory.
- Llama 2 Compatibility: Shares the same architecture and tokenizer as Llama 2, allowing for seamless integration into existing Llama-based workflows.
- Chat Fine-tuning: The model was fine-tuned following the Hugging Face Zephyr training recipe. This involved initial fine-tuning on a variant of the
UltraChatdataset, which contains diverse synthetic dialogues generated by ChatGPT. - Alignment with DPO: Further aligned using
DPOTraineron theopenbmb/UltraFeedbackdataset, which includes 64k prompts and model completions ranked by GPT-4, enhancing its conversational quality.
Good for
- Resource-constrained applications: Ideal for deployment where computational power or memory is limited.
- Conversational AI: Suited for chatbot applications and generating human-like dialogue.
- Llama-based project integration: Easily integrates into projects already utilizing Llama 2 due to architectural consistency.