TinyLlama/TinyLlama-1.1B-intermediate-step-955k-token-2T
TEXT GENERATIONConcurrency Cost:1Model Size:1.1BQuant:BF16Ctx Length:2kPublished:Nov 19, 2023License:apache-2.0Architecture:Transformer0.0K Open Weights Cold
TinyLlama-1.1B-intermediate-step-955k-token-2T is a 1.1 billion parameter Llama-architecture language model developed by the TinyLlama project. This intermediate checkpoint has been pretrained on 2 trillion tokens across 995,000 steps, utilizing the same architecture and tokenizer as Llama 2. Its compact size and Llama 2 compatibility make it suitable for applications requiring restricted computation and memory footprints.
Loading preview...
TinyLlama-1.1B-intermediate-step-955k-token-2T Overview
This model is an intermediate checkpoint from the TinyLlama project, which aims to pretrain a 1.1 billion parameter Llama-architecture model on 3 trillion tokens. Developed by the TinyLlama project, this specific version has completed 995,000 training steps, processing 2 trillion tokens.
Key Characteristics
- Architecture: Employs the exact same architecture and tokenizer as Llama 2, ensuring compatibility with existing Llama-based open-source projects.
- Parameter Count: Features a compact 1.1 billion parameters, making it efficient for deployment in environments with limited computational resources.
- Training Progress: Represents a significant milestone in the ongoing 90-day training initiative, which began on 2023-09-01.
Use Cases
- Resource-Constrained Environments: Ideal for applications where memory and computational power are limited, due to its small size.
- Llama 2 Ecosystem Integration: Can be easily integrated into projects and workflows already utilizing Llama 2 models, leveraging its architectural compatibility.
- Research and Development: Suitable for researchers and developers exploring the capabilities of smaller, Llama-compatible models at various stages of pretraining.