geodesic-research/hybrid-moe-30b-a3b-base
The geodesic-research/hybrid-moe-30b-a3b-base is a 31.6 billion parameter base language model developed by geodesic-research, featuring a unique hybrid architecture combining Mamba2, attention, and mixture-of-experts layers. Trained from scratch on 553.8 billion tokens, it utilizes approximately 3 billion active parameters per token and supports a context length of 32,768 tokens. This model is designed for text completion tasks, serving as a foundational model without instruction tuning or preference optimization.
Loading preview...
Model Overview
The geodesic-research/hybrid-moe-30b-a3b-base is a 31.6 billion parameter base language model developed by geodesic-research. It stands out due to its novel hybrid architecture, which integrates Mamba2, attention, and mixture-of-experts (MoE) layers, designated as nemotron_h.
Key Characteristics
- Hybrid Architecture: Combines Mamba2, attention, and MoE layers for potentially enhanced efficiency and performance.
- Parameter Efficiency: While having a total of 31.6 billion parameters, it utilizes roughly 3 billion active parameters per token, leveraging the MoE design.
- Extensive Training: Trained from random initialization on over 553 billion tokens (553.8B).
- Long Context Window: Supports a substantial context length of 32,768 tokens, having been continued from an initial 8,192 tokens.
- Base Model: This is a foundational model, meaning it has not undergone instruction tuning or preference optimization.
Intended Use Cases
This model is primarily suited for text completion tasks where a raw, untuned language model is required. Due to its base nature, it is not optimized for direct chat applications or instruction-following, but rather for tasks that benefit from its strong generative capabilities and long context understanding.