ibm-granite/granite-3.0-3b-a800m-base
Granite-3.0-3B-A800M-Base is a 3.3 billion parameter decoder-only language model developed by IBM, based on a sparse Mixture of Experts (MoE) transformer architecture. It is trained on 10 trillion tokens across two stages, designed to support a variety of text-to-text generation tasks such as summarization, classification, and question-answering. This model is notable for its MoE architecture with 40 experts and 8 active experts, enabling efficient processing for its size. It supports 12 languages including English, German, and Japanese, and serves as a versatile baseline for specialized applications.
Loading preview...
Overview
Granite-3.0-3B-A800M-Base is a 3.3 billion parameter decoder-only language model developed by the IBM Granite Team. It utilizes a sparse Mixture of Experts (MoE) transformer architecture, featuring 40 experts with 8 active experts, and is trained on a substantial 10 trillion tokens. The training process involves two stages: an initial stage on 8 trillion tokens from diverse domains (web, code, academic, books, math), followed by a second stage on 2 trillion tokens of high-quality, curated data, including multilingual and instruction data, to enhance task-specific performance.
Key Capabilities
- Text-to-Text Generation: Designed to handle various text generation tasks, including summarization, text classification, information extraction, and question-answering.
- Multilingual Support: Supports 12 languages out-of-the-box, including English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese, with potential for fine-tuning for additional languages.
- MoE Architecture: Employs a decoder-only sparse Mixture of Experts (MoE) transformer architecture, incorporating Fine-grained Experts, Dropless Token Routing, and Load Balancing Loss for efficient processing.
- Baseline Model: Intended to serve as a robust baseline for creating more specialized models tailored to specific application scenarios.
Intended Use Cases
- General Text Generation: Suitable for a broad range of generative AI tasks.
- Specialized Model Development: Can be fine-tuned to create custom models for particular domains or tasks.
Limitations and Ethical Considerations
As a base model, Granite-3.0-3B-A800M-Base has not undergone safety alignment and may produce problematic outputs. Users should be aware of potential risks such as bias, misinformation, and hallucination, especially given its smaller size. Responsible and ethical use is strongly encouraged.