Overview
This model, TinyLlama-1.1B-intermediate-step-480k-1T-chat-llama-style, is a 1.1 billion parameter language model based on the Llama 2 architecture. It is an intermediate checkpoint from the larger TinyLlama project, which aims to pretrain a 1.1B Llama model on 3 trillion tokens. This specific version has been fine-tuned by Trelis for chat purposes, using an adapted and filtered Openassistant dataset. It maintains the same architecture and tokenizer as Llama 2, ensuring compatibility with existing Llama-based open-source projects.
Key Characteristics
- Architecture: Llama 2-compatible, allowing for seamless integration into existing Llama ecosystems.
- Parameter Count: 1.1 billion parameters, making it compact and suitable for environments with limited computation and memory.
- Training Data: Fine-tuned on an adapted filtered Openassistant dataset, with the base model trained on 1 trillion tokens over 480,000 steps.
- Prompt Format: Uses the Llama 2-style instruction format:
f"[INST] {prompt} [INST]".
Use Cases
- Chat Applications: Specifically fine-tuned for conversational AI and chat-based interactions.
- Resource-Constrained Environments: Its compact size makes it ideal for deployment where computational power or memory is limited.
- Llama 2 Ecosystems: Can be "plug and played" into projects built upon the Llama 2 framework due to architectural and tokenizer compatibility.