self-long/SelfLong-Llama3.2-3B-Instruct-1M

TEXT GENERATIONConcurrent Unit Cost:1Model Size:3.2BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Mar 17, 2025License:mitArchitecture:Transformer Open Weights Featherless Exclusive Cold

SelfLong-Llama3.2-3B-Instruct-1M is a 3.2 billion parameter instruction-tuned large language model from the SelfLong series, initialized from the Llama-3.2 architecture. Developed by Wang et al., this model is specifically designed to handle extremely long contexts, supporting up to 1 million tokens. It excels in tasks requiring extensive context understanding, as demonstrated by its performance on the RULER-1M benchmark, making it suitable for applications needing deep contextual analysis.

Loading preview...

SelfLong-Llama3.2-3B-Instruct-1M: Extreme Context Length LLM

This model is part of the SelfLong series, developed by Wang, Yang, Zhang, Huang, and Wei, focusing on exceptionally long context handling. Initialized from the Llama-3.2 architecture, this 3.2 billion parameter instruction-tuned model is engineered to process and understand contexts up to an impressive 1 million tokens.

Key Capabilities

  • Ultra-Long Context Processing: Designed to manage and reason over context lengths up to 1 million tokens, significantly surpassing typical LLM capabilities.
  • Llama-3.2 Base: Leverages the robust Llama-3.2 architecture for its foundational language understanding.
  • Instruction-Tuned: Optimized for following instructions, making it suitable for a wide range of NLP tasks.
  • RULER-1M Benchmark Performance: Evaluated on the RULER-1M benchmark, which assesses performance across 13 tasks at various support lengths, demonstrating its proficiency in long-context scenarios. The SelfLong-3B-1M model achieves a RULER-1M score of 38.8 at 1M support length.

When to Use This Model

This model is particularly well-suited for use cases that demand processing and understanding very large documents or extensive conversational histories. Consider using SelfLong-Llama3.2-3B-Instruct-1M for:

  • Document Analysis: Summarizing, querying, or extracting information from extremely long texts like legal documents, research papers, or books.
  • Long-form Content Generation: Creating coherent and contextually relevant long-form articles, reports, or creative writing pieces.
  • Complex Reasoning: Tasks requiring the model to synthesize information from vast amounts of input to answer questions or solve problems.
  • Conversational AI: Maintaining context over very long dialogues or chat histories.

Finetunes
1 models