Xirui1208/selective-readtwice-14b-sft-epoch1-step2944-20260925
Xirui1208/selective-readtwice-14b-sft-epoch1-step2944-20260925 is a 14.8 billion parameter model developed by Xirui1208, representing a selective gate SFT checkpoint built upon the ReadAll reader architecture. This model incorporates a trained selective LoRA adapter and gate bias to enable selective reading capabilities, distinguishing it from standard language models. It is specifically designed for tasks requiring efficient navigation and processing of long documents, leveraging a 32,768 token native context length and recurrent pages for extended content.
Loading preview...
Overview
This model, Xirui1208/selective-readtwice-14b-sft-epoch1-step2944-20260925, is a 14.8 billion parameter selective gate Supervised Fine-Tuning (SFT) checkpoint. It is built on the Xirui1208/readall-readtwice-14b-rl-step15-20260925 base, featuring a frozen ReadAll reader and a trained selective LoRA adapter with gate bias. The core innovation lies in its "selective reading" mechanism, which allows the model to efficiently process and navigate long documents by selectively focusing on relevant sections.
Key Capabilities
- Selective Reading: Utilizes a LoRA adapter and gate bias to enable selective processing of document sections, improving efficiency for long contexts.
- Extended Context Handling: Supports a native context length of 32,768 tokens and employs recurrent pages for documents exceeding this limit.
- Optimized for Document Navigation: Trained with SFT views including short and very long documents (up to 224K tokens) to refine navigation outlines.
- Modular Architecture: The ReadAll reader weights are frozen, with the selective gate implemented as an adapter, ensuring a specific processing pipeline.
Good For
- Information Retrieval from Long Documents: Ideal for applications requiring precise extraction or summarization from extensive texts where not all content is equally relevant.
- Question Answering over Large Corpora: Can be used to efficiently locate and process information pertinent to a query within large document sets.
- Research and Analysis: Suitable for tasks involving the analysis of lengthy reports, articles, or technical specifications by focusing on key sections.