Llama-3-Giraffe-70B-Instruct: Extended Context Llama 3

Abacus.AI presents Llama-3-Giraffe-70B-Instruct, an instruction-tuned variant of the Llama 3 70B model, specifically engineered for significantly longer context windows. This model boasts an effective context length of approximately 128k tokens, a substantial increase over its base model.

Key Capabilities & Features

• Extended Context Window: Achieves an effective context of ~128k tokens, enabling processing of very long documents and conversations.
• Context Extension Methodology: Utilizes Positional Skip-wise Training (PoSE) with 5 chunks and a max position ID of 32768, combined with dynamic-NTK interpolation (scale factor 4).
• Performance Preservation: MT-Bench evaluations indicate that the context extension process did not significantly degrade performance on instruct tasks, with scores closely matching the base Meta-Llama-3-70B-Instruct.
• Training Data: Trained on approximately 1.5 billion tokens, using ~8K long samples from the RedPajama dataset.
• Adapter Transfer: The context extension techniques were first applied to Llama-3-70B-Base using LoRA on Q and K weights, then transferred to Llama-3-70B-Instruct.

Ideal Use Cases

• Long Document Analysis: Excellent for tasks requiring understanding and summarization of extensive texts, such as legal documents, research papers, or books.
• Extended Conversational AI: Suitable for chatbots or virtual assistants that need to maintain context over very long dialogues.
• Information Retrieval: Can process large amounts of information to extract specific details or answer complex queries from vast knowledge bases.