Dohyeon1/ERNIE-M-SMoE-ngroups48-maxcls2
Dohyeon1/ERNIE-M-SMoE-ngroups48-maxcls2 is a 21 billion parameter language model. This model is based on the ERNIE-M architecture, utilizing a Sparse Mixture of Experts (SMoE) with 48 expert groups, and supports a context length of 32768 tokens. The specific differentiators and primary use cases are not detailed in the provided model card, which indicates that more information is needed regarding its development, training, and intended applications.
Loading preview...
Model Overview
This model, Dohyeon1/ERNIE-M-SMoE-ngroups48-maxcls2, is a 21 billion parameter language model based on the ERNIE-M architecture. It incorporates a Sparse Mixture of Experts (SMoE) design, featuring 48 expert groups. The model supports a substantial context length of 32768 tokens, which is beneficial for processing longer sequences of text.
Key Characteristics
- Architecture: ERNIE-M with Sparse Mixture of Experts (SMoE).
- Parameters: 21 billion.
- Expert Groups: 48 expert groups within the SMoE architecture.
- Context Length: Supports up to 32768 tokens.
Current Status
The provided model card indicates that significant details regarding its development, funding, specific model type, language support, license, and fine-tuning origins are currently marked as "More Information Needed." Consequently, specific direct or downstream use cases, known biases, risks, limitations, training data, and evaluation results are not yet available. Users are advised that further information is required to understand the model's intended applications and performance characteristics fully.