ibm-ai-platform/Bamba-9B-2T
Bamba-9B-2T is a 9.78 billion parameter decoder-only language model developed by IBM, based on the Mamba-2 architecture. It was trained from scratch using a two-stage approach, with the first stage involving 2 trillion tokens from the Dolma v1.7 dataset. This model is designed for a wide range of text generation tasks, leveraging its Mamba-2 architecture for efficient sequence processing.
Loading preview...
Bamba-9B-2T: A Mamba-2 Architecture Language Model
Bamba-9B-2T is a 9.78 billion parameter decoder-only language model developed by IBM, built upon the Mamba-2 architecture. This model is designed for diverse text generation tasks and was trained using a two-stage process. The initial stage involved pre-training on 2 trillion tokens from the Dolma v1.7 dataset, followed by a second stage with an additional 200 billion tokens from a high-quality data blend to enhance performance and output quality.
Key Capabilities
- Efficient Text Generation: Leverages the Mamba-2 architecture for effective sequence processing.
- Two-Stage Training: Benefits from extensive pre-training on 2.2 trillion tokens for robust language understanding.
- Quantization Support: Can be optimized for storage and inference efficiency using FP8 quantization via
fms-model-optimizer, significantly reducing memory footprint. - Hugging Face Integration: Fully integrated with Hugging Face
transformersfor easy inference and fine-tuning.
Good for
- General Text Generation: Suitable for a broad spectrum of text-based tasks.
- Research and Development: Provides a strong base model for further fine-tuning and experimentation with the Mamba-2 architecture.
- Resource-Constrained Deployment: Quantized versions offer reduced memory usage, making them viable for environments with limited resources.