microperceptron/Kimi-Linear-48B-A3B-Instruct

Overview

Kimi Linear is a 48 billion parameter instruction-tuned language model developed by Moonshot-AI, featuring a novel hybrid linear attention architecture. It introduces Kimi Delta Attention (KDA), a refined version of Gated DeltaNet, which optimizes finite-state RNN memory usage through an efficient gating mechanism. This architecture allows Kimi Linear to outperform traditional full attention methods across various contexts, including short, long, and reinforcement learning scaling regimes.

Key Capabilities

• Superior Long-Context Performance: Achieves high performance on tasks requiring up to 1M tokens, with a 3:1 KDA-to-global MLA ratio that reduces memory usage while maintaining quality.
• Enhanced Hardware Efficiency: Reduces KV cache requirements by up to 75% and boosts decoding throughput by up to 6x for long sequences, offering significant speedups in time per output token (TPOT).
• Robust Attention Mechanism: KDA refines the gated delta rule with fine-grained gating, contributing to its superior performance and efficiency.

Good For

• Applications requiring extensive context processing: Ideal for tasks that benefit from very long input sequences, such as document analysis, complex reasoning, or summarization of large texts.
• Resource-constrained environments: Its reduced memory footprint and higher decoding throughput make it suitable for deployment where hardware efficiency is critical.
• Research and development in attention mechanisms: Provides an open-source KDA kernel for further exploration and integration into other models.