LargeWorldModel/LWM-Text-1M
TEXT GENERATIONConcurrency Cost:1Model Size:7BQuant:FP8Ctx Length:4kPublished:Jan 27, 2024Architecture:Transformer0.0K Cold
LWM-Text-1M is an open-source 7 billion parameter auto-regressive language model, based on the transformer architecture and trained from LLaMA-2. Developed by LargeWorldModel, it was specifically trained on a subset of Books3 data containing documents with over 1 million tokens. This model is designed for tasks requiring processing of very long text sequences, leveraging its specialized training on extensive document lengths.
Loading preview...
LWM-Text-1M Overview
LWM-Text-1M is a 7 billion parameter open-source language model developed by LargeWorldModel. It is built upon the LLaMA-2 architecture and functions as an auto-regressive transformer model. The model was trained in December 2023 with a specific focus on handling exceptionally long text inputs.
Key Capabilities
- Long Context Processing: Uniquely trained on a subset of Books3 documents, specifically those exceeding 1 million tokens in length, indicating a specialization in processing very long textual data.
- LLaMA-2 Foundation: Benefits from the robust architecture and pre-training of the LLaMA-2 model.
- Open-Source: Available under the LLAMA 2 Community License, encouraging broad use and further development.
Good For
- Applications requiring extensive document analysis: Ideal for tasks that involve understanding, summarizing, or generating content from very long texts, such as books, research papers, or legal documents.
- Research and development: Provides a foundation for exploring long-context language modeling based on a LLaMA-2 derivative.