jeff31415/TinyLlama-1.1B-1T-OpenOrca
jeff31415/TinyLlama-1.1B-1T-OpenOrca is a 1.1 billion parameter language model based on the TinyLlama architecture, fine-tuned on the OpenOrca GPT-4 subset. This model is optimized for instruction-following tasks, leveraging its training on high-quality conversational data. It offers a compact yet capable solution for applications requiring efficient natural language understanding and generation.
Loading preview...
Model Overview
This model, jeff31415/TinyLlama-1.1B-1T-OpenOrca, is a 1.1 billion parameter language model built upon the PY007/TinyLlama-1.1B-intermediate-step-480k-1T base. It has been specifically fine-tuned for one epoch on the OpenOrca GPT-4 subset, utilizing the CHATML format for instruction-following capabilities.
Key Characteristics
- Base Model: TinyLlama 1.1B, known for its compact size and efficiency.
- Fine-tuning: Enhanced with the OpenOrca GPT-4 dataset, aiming to improve its ability to follow instructions and engage in conversational tasks.
- License: Adheres to the Apache 2.0 license, consistent with its TinyLlama base.
- Quantizations: Available in various quantized formats for optimized deployment, including GPTQ, AWQ, and GGUF.
Training Details
The fine-tuning process was conducted on a single RTX A5000 GPU, taking approximately 16 hours to complete one epoch. Further details regarding the training run can be found on Weights & Biases.
Ideal Use Cases
This model is well-suited for applications where a smaller, efficient model with good instruction-following capabilities is required, such as:
- Lightweight chatbots or conversational agents.
- Text generation tasks requiring adherence to specific prompts.
- Edge device deployment where computational resources are limited.