ibm-granite/granite-3.1-1b-a400m-base
The Granite-3.1-1B-A400M-Base is a 1.3 billion parameter decoder-only Mixture of Experts (MoE) language model developed by IBM, featuring an effective 400 million active parameters. It extends its context length to 128K tokens through progressive training, utilizing approximately 500 billion tokens during this long-context pre-training stage. This model is designed for general text-to-text generation tasks such as summarization, classification, and question-answering, and serves as a baseline for specialized applications.
Loading preview...
Overview
IBM's Granite-3.1-1B-A400M-Base is a 1.3 billion parameter decoder-only Mixture of Experts (MoE) language model, part of the Granite 3.1 series. It distinguishes itself with an extended context length of 128K tokens, achieved through a progressive training strategy that incrementally increased context length and adjusted RoPE theta. This long-context pre-training phase involved approximately 500 billion tokens.
Key Capabilities
- Extended Context Window: Supports a 128K token context length, enabling processing of significantly longer inputs and generating more coherent, extended outputs.
- Mixture of Experts Architecture: Utilizes a sparse MoE design with 32 experts and a TopK of 8, resulting in 400 million active parameters for efficient inference while maintaining a larger total parameter count.
- Multilingual Support: Trained to support 12 languages including English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese, with potential for fine-tuning in additional languages.
- General Purpose Base Model: Designed to handle a wide array of text-to-text generation tasks such as summarization, text classification, extraction, and question-answering.
Intended Use Cases
This model is suitable for developers looking for a compact yet capable base model for various generative AI tasks. Its extended context window makes it particularly useful for applications requiring processing or generating long documents, conversations, or code. It can also serve as a strong foundation for creating specialized models through fine-tuning for specific domains or tasks.