Dohyeon1/ERNIE-HC-SMoE-ngroups48

TEXT GENERATIONPricing:Input $0.32 / Cached $0.016 / Output $1.6Concurrent Unit Cost:1Model Size:21BQuant:FP8Context Size:32kPublished:Sep 19, 2026Architecture:Transformer Featherless Exclusive Cold

Dohyeon1/ERNIE-HC-SMoE-ngroups48 is a 21 billion parameter language model with a context length of 32768 tokens. This model is based on the ERNIE architecture, utilizing a Sparse Mixture of Experts (SMoE) design with 48 expert groups. Its primary differentiator lies in its SMoE architecture, which allows for efficient scaling and potentially enhanced performance by selectively activating experts for different tasks. This model is suitable for general language understanding and generation tasks where a large parameter count and efficient processing are beneficial.

Loading preview...

Model Overview

Dohyeon1/ERNIE-HC-SMoE-ngroups48 is a large language model featuring 21 billion parameters and a substantial context length of 32768 tokens. This model is built upon the ERNIE architecture and incorporates a Sparse Mixture of Experts (SMoE) design, specifically configured with 48 expert groups. The SMoE architecture is a key differentiator, enabling the model to efficiently handle diverse tasks by activating only a subset of its experts, which can lead to improved computational efficiency and performance compared to dense models of similar size.

Key Capabilities

  • Large Scale: With 21 billion parameters, it is designed for complex language understanding and generation.
  • Extended Context: A 32768-token context window allows for processing and generating longer texts, maintaining coherence over extended conversations or documents.
  • Efficient Architecture: The Sparse Mixture of Experts (SMoE) with 48 groups aims to provide high performance while potentially reducing computational costs during inference by dynamically selecting relevant experts.

Good For

  • General-purpose language tasks: Suitable for a wide range of applications requiring robust language understanding and generation.
  • Applications requiring long context: Ideal for tasks like summarization of lengthy documents, detailed question answering, or extended conversational AI.
  • Research into SMoE architectures: Provides a substantial model for exploring the benefits and performance characteristics of Sparse Mixture of Experts designs.

Finetunes
3 models