acalatrava/TinyLlama-1.1B-orca-gpt4 is a 1.1 billion parameter language model, fine-tuned from TinyLlama-1.1B-intermediate-step-240k-503b. It leverages the ORCA-GPT4 dataset in ChatML format, making it optimized for instruction-following and conversational tasks. This model offers a compact yet capable solution for applications requiring efficient, instruction-tuned language generation within a 2048-token context window.
Loading preview...
Model Overview
acalatrava/TinyLlama-1.1B-orca-gpt4 is a compact 1.1 billion parameter language model, building upon the base of TinyLlama-1.1B-intermediate-step-240k-503b. This model has been specifically fine-tuned using the sam-mosaic/orca-gpt4-chatml dataset, formatted for ChatML.
Key Characteristics
- Base Model: Fine-tuned from TinyLlama-1.1B-intermediate-step-240k-503b.
- Fine-tuning Dataset: Utilizes the ORCA-GPT4 dataset, known for its high-quality, instruction-following examples.
- Format: Implements the ChatML format, making it suitable for chat-based applications and instruction-following.
- Training Method: Employed QLORA for efficient fine-tuning.
- Quantization: Trained with fp16 quantization.
- Context Length: Supports a context window of 2048 tokens.
Use Cases
This model is particularly well-suited for scenarios where a small, efficient, and instruction-following language model is required. Its fine-tuning on the ORCA-GPT4 dataset suggests strong capabilities in:
- Instruction Following: Generating responses that adhere to specific user instructions.
- Conversational AI: Engaging in coherent and contextually relevant dialogue.
- Resource-Constrained Environments: Its small size makes it ideal for deployment on hardware with limited resources.