Outlier-10B V2: Efficient MoE with High Teacher Retention

Outlier-10B V2 is a 10.6 billion parameter Mixture-of-Experts (MoE) language model developed by Outlier-Ai, leveraging a Qwen2.5-7B-Instruct base. It stands out for its ternary quantization and a unique Dense-Sparse-Dense architecture where approximately 55% of layers utilize 16 ternary experts with top-2 routing.

Key Capabilities & Features

• High MMLU Performance: Achieves 75.96% on MMLU (5-shot), demonstrating 99.1% retention compared to its Qwen2.5-7B teacher model.
• Efficient Distillation: Performance gains are attributed to Context-Aware KL Divergence (CAKLD) distillation, training only ~3% of parameters (lm_head + layer norms) over 500 steps on 5,000 examples.
• Memory-Optimized Inference: Despite its 10.6B total parameters, it activates only ~7.4B parameters per token, resulting in an inference RAM usage of approximately 5 GB.
• Significant Improvement over V1: This V2 model shows a +5.26 MMLU point increase over its V1 predecessor, closing 74% of the gap to the dense teacher model.

Good For

• Applications requiring strong reasoning capabilities as indicated by MMLU scores.
• Deployments where memory efficiency is critical, due to its MoE architecture and low active parameter count per token.
• Developers interested in advanced distillation techniques and ternary quantization for large language models.