Dohyeon1/ERNIE-Sub-MoE-ngroups48-maxcls2
Dohyeon1/ERNIE-Sub-MoE-ngroups48-maxcls2 is a 21 billion parameter Mixture-of-Experts (MoE) language model with a context length of 32768 tokens. This model is part of the ERNIE family, designed for general language understanding and generation tasks. Its MoE architecture with 48 groups and 2 experts per token aims to enhance efficiency and performance for various natural language processing applications.
Loading preview...
Model Overview
The Dohyeon1/ERNIE-Sub-MoE-ngroups48-maxcls2 is a 21 billion parameter language model utilizing a Mixture-of-Experts (MoE) architecture. It is designed to process long sequences with a substantial context length of 32768 tokens. The model's architecture incorporates 48 expert groups, with 2 experts activated per token, which typically allows for more efficient computation while maintaining high performance compared to dense models of similar parameter count.
Key Characteristics
- Architecture: Mixture-of-Experts (MoE) with 48 expert groups and 2 experts per token.
- Parameter Count: 21 billion parameters.
- Context Length: Supports a long context window of 32768 tokens.
Current Status
As indicated by the model card, specific details regarding its development, funding, training data, evaluation metrics, and intended use cases are currently marked as "More Information Needed." Users should be aware that comprehensive information on its performance, biases, risks, and limitations is not yet available. Further details are required to fully understand its capabilities and appropriate applications.