shisa-ai/Ornith-1.5-35B-A3B-MTP
shisa-ai/Ornith-1.5-35B-A3B-MTP is a 35.1 billion parameter language model developed by shisa-ai, built upon the Ornith-1.5-35B-A3B architecture with a 32K context length. This model features a custom, re-aligned MTP (Multi-Token Prediction) head, specifically optimized for speculative decoding performance, particularly excelling in code-related tasks. It offers significantly improved draft token acceptance and throughput compared to the original Ornith-1.5 MTP head, making it suitable for high-efficiency inference scenarios.
Loading preview...
Ornith-1.5-35B-A3B-MTP Overview
This model is a 35.1 billion parameter variant of the Ornith-1.5-35B-A3B architecture, developed by shisa-ai. Its primary distinction is a custom-trained MTP (Multi-Token Prediction) head, which replaces the poorly performing native MTP head found in the official Ornith-1.5 checkpoint. This replacement head is initialized from Qwen3.6-35B-A3B's MTP head and then re-aligned to Ornith-1.5's hidden states using full-vocabulary KL distillation.
Key Differentiators & Performance
- Enhanced Speculative Decoding: The custom MTP head significantly improves speculative draft acceptance rates. It achieves 69.27% code acceptance and 60.51% overall suite acceptance, a substantial improvement over the native Ornith-1.5 head (37.20% code, 32.19% overall).
- Code-Optimized: The training data for the MTP head is 50% code-heavy, leading to superior performance on code-related prompts. This optimization results in higher accepted token lengths and better per-position acceptance for speculative decoding in code tasks.
- Improved Throughput: Benchmarks show this model achieving 273.09 client output tok/s on code gates, compared to 220.47 tok/s with the native head, indicating faster inference.
- Efficient Training: The MTP head training process was optimized for speed, achieving a 4.25x faster update time (2.82 seconds/update) through techniques like packing multiple rows per forward pass and using a fused full-vocabulary KL divergence.
Use Cases & Limitations
- Ideal for high-throughput inference where speculative decoding is crucial, especially for code generation and related tasks.
- Not an across-the-board replacement; while strong on code, ShareGPT-style acceptance did not improve over the 5K head.
- Experimental status: It's noted as experimental, with observed differences between AR and speculative modes in vLLM, indicating ongoing correctness investigations.
- Text-only evaluation: Multimodal MTP behavior was not tested. Users should refer to the official Ornith-1.5 model card for base model capabilities and risks.