qihoo360/llama3-8B-360Zhinao-360k-Instruct

TEXT GENERATIONPricing:Input $0.37 / Cached $0.074 / Output $0.38Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:8kTool Calling:SupportedPublished:May 19, 2024License:apache-2.0Architecture:Transformer0.0K Open Weights Featherless Exclusive Cold

The qihoo360/llama3-8B-360Zhinao-360k-Instruct model, developed by 360Zhinao, is an 8 billion parameter instruction-tuned language model based on Llama3-8B-Instruct. It features an extended context window of 360,000 tokens, significantly enhancing its ability to process and recall information from very long inputs. This model excels in long-context understanding and retrieval tasks, achieving high accuracy on various Needle-In-A-Haystack benchmarks.

Loading preview...

Overview

qihoo360/llama3-8B-360Zhinao-360k-Instruct is an 8 billion parameter model developed by 360Zhinao, extending the context window of Meta's Llama3-8B-Instruct to an impressive 360,000 tokens. This significant context extension is achieved by modifying the RoPE base and training on a combined SFT dataset including LWM's open-sourced data and internal long-context data in both Chinese and English.

Key Capabilities

  • Exceptional Long-Context Retrieval: Demonstrates 100% perfect recall on the "value retrieval" variant of Needle-In-A-Haystack (NIAH) and 99.75% near-perfect recall on the original NIAH and its Chinese counterpart. This indicates strong performance in extracting specific information from very long documents.
  • Robust Long-Context Understanding: Unlike some models that perform well only on simpler NIAH variants, this model maintains high accuracy on more complex, context-dependent NIAH tasks.
  • Instruction Following: As an instruction-tuned model, it is designed to follow user commands effectively within its extended context.

Good For

  • Applications requiring extensive context: Ideal for tasks like summarizing long documents, analyzing large codebases, processing lengthy conversations, or any scenario where maintaining context over hundreds of thousands of tokens is crucial.
  • Information Retrieval: Particularly strong in tasks that involve finding specific pieces of information embedded within very long texts.
  • Research and Development: Useful for researchers exploring the limits of long-context language models and their practical applications.

Finetunes
1 models