hxia7/qwen3-4b-blockdist
The hxia7/qwen3-4b-blockdist model is a block-attention distilled Qwen3-4B variant designed for efficient long-context and RAG serving. It features a novel decode-side router that dynamically re-selects active blocks for every generated token, optimizing KV-read bandwidth. This model is specifically engineered to maintain full-attention quality while significantly improving efficiency for long-context applications by reusing document chunk KV states across requests.
Loading preview...
Overview
hxia7/qwen3-4b-blockdist is a specialized Qwen3-4B model utilizing block-attention distillation for enhanced efficiency in long-context and Retrieval Augmented Generation (RAG) scenarios. A key innovation is its decode-side router, which dynamically re-selects active blocks for each generated token, allowing for efficient sparse serving. This approach aims to preserve the quality of full-attention models while drastically reducing KV-read bandwidth, particularly beneficial for batched serving and prefill KV reuse of document chunks.
Key Capabilities & Findings
- Block Attention Quality: Achieves performance comparable to full causal attention, with previous "block costs" identified as measurement artifacts (e.g., token-F1 penalizing verbosity). When evaluated with subspan-EM and a no-merge protocol, block attention matches or surpasses full attention.
- Dynamic Rerouting: The per-step reroute mechanism is crucial for long-form and multi-document generation, where the relevant context block shifts during decoding. For short-answer QA, it performs similarly to static block selection.
- Multi-hop Reasoning: The model demonstrates that the block-attention deficit in multi-hop QA is training-distribution-limited, not structural. Distilling on multi-hop data (like HotpotQA, 2WikiMQA, Musique) effectively closes this gap, with the block model even outperforming the full teacher on some multi-hop datasets after specialized distillation.
- KV-Read Bandwidth Savings: The primary efficiency gain comes from reduced KV-read bandwidth (approximately
k/nblocks active plus resident summaries) and the ability to reuse prefill KV states of document chunks across multiple requests.
When to Use This Model
- Long-Context Applications: Ideal for use cases requiring processing and generating responses based on very long input contexts.
- RAG Systems: Highly optimized for RAG architectures where efficient document retrieval and context integration are critical.
- Efficient Serving: When serving large language models with long contexts, especially in batched scenarios, to reduce memory and computational overhead related to KV cache management.
- Multi-hop Question Answering: Particularly strong for complex QA tasks that require assembling information from multiple distinct blocks of text, especially after multi-hop distillation.