unsloth/ERNIE-4.5-21B-A3B-PT
ERNIE-4.5-21B-A3B-PT is a 21 billion total parameter, 3 billion activated parameter text-based Mixture-of-Experts (MoE) model developed by Baidu. This post-trained model features a 32768 token context length and is optimized for general-purpose language understanding and generation. It leverages multimodal heterogeneous MoE pre-training and scaling-efficient infrastructure for high-performance inference.
Loading preview...
ERNIE-4.5-21B-A3B-PT Overview
ERNIE-4.5-21B-A3B-PT is a text-based Mixture-of-Experts (MoE) model from Baidu, featuring 21 billion total parameters and 3 billion activated parameters per token. It is a post-trained variant, optimized for general-purpose language understanding and generation. The model incorporates a multimodal heterogeneous MoE pre-training approach, designed to effectively process both textual and visual modalities without mutual interference, utilizing techniques like modality-isolated routing and router orthogonal loss.
Key Technical Innovations
- Multimodal Heterogeneous MoE Pre-Training: Joint training on text and visual data with a specialized MoE structure to capture multimodal nuances. This includes modality-isolated routing and multimodal token-balanced loss for effective representation.
- Scaling-Efficient Infrastructure: Employs heterogeneous hybrid parallelism, hierarchical load balancing, and FP8 mixed-precision training for high pre-training throughput. For inference, it uses multi-expert parallel collaboration and convolutional code quantization for 4-bit/2-bit lossless quantization.
- Modality-Specific Post-Training: Fine-tuned using methods like Supervised Fine-tuning (SFT), Direct Preference Optimization (DPO), or Unified Preference Optimization (UPO) to meet diverse application requirements.
Model Configuration
- Modality: Text
- Parameters: 21B (Total) / 3B (Activated)
- Context Length: 131072 tokens (Note: The provided model info states 32768, but the README table indicates 131072. Following README table for detailed summary).
- Experts: 64 Text Experts (6 activated), 64 Vision Experts (6 activated), 2 Shared Experts.
Usage
This model can be easily integrated using the transformers library (version 4.54.0 or newer) for text completion tasks. It also supports vLLM inference for efficient serving.