bhenrym14/airoboros-3_1-yi-34b-200k
The bhenrym14/airoboros-3_1-yi-34b-200k is a 34 billion parameter instruction-tuned language model based on the 01-ai/Yi-34B-200k architecture, utilizing Llama2 model definitions and tokenizer. Fine-tuned with the Airoboros-3.1 dataset, this model is designed for general instruction-following tasks. It offers a substantial 32,768 token context length, making it suitable for processing longer inputs and generating coherent, extended responses.
Loading preview...
Overview
This model, bhenrym14/airoboros-3_1-yi-34b-200k, is an instruction-tuned variant of the 01-ai/Yi-34B-200k base model, specifically using the larryvrh/Yi-34B-200K-Llamafied version which incorporates Llama2 model definitions and tokenizer to eliminate remote code dependencies. It features 34 billion parameters and supports a significant 32,768 token context window.
Key Capabilities
- Instruction Following: Fine-tuned using the
jondurbin/airoboros-3.1dataset, it is optimized for general instruction-following tasks. - Llama2 Compatibility: Utilizes Llama2 model definitions and tokenizer, ensuring broad compatibility with existing Llama2-based workflows and tools.
- Extended Context: Benefits from the 200k context length of its base model, allowing for processing and generation of longer texts.
Training Details
The model was fine-tuned using a QLoRA (rank 64) method. The training process involved truncating prompts to 4096 tokens for efficiency and was performed on an RTX 6000 Ada GPU, taking approximately 80 hours to reach its current checkpoint.
Prompting
It is designed to be used with the Llama-2 chat prompt format, consistent with models like jondurbin/airoboros-l2-13b-3.1.1.