Level of Detail Attention: Allocating Resolution Instead of Sparsity

Subquadratic attention for pretrained transformers, no changes needed!

Code · Paper (link pending) · LinkedIn (link pending) · X (link pending)

The Main Idea

By this point everyone knows that attention can be slow when your context length gets long. Looking back at every token to figure out what’s relevant is inefficient at best. Sparsity methods attempt to combat this problem by removing tokens. But this actually changes attention, ignoring things that your model was trained to see and depend on!

Our key idea is that instead of ignoring the less important parts, you should just view them in a lower resolution. By evaluating attention in this way, we directly get subquadratic inference speeds on models like Qwen, because we keep our attention approximation close to what the model already expects. High quality, fast speed, no training necessary - pick three!

Still from the explainer animation. Play the 10 second clip

How It Works

We create a multi-resolution view of the KV cache in an online manner as inference proceeds. After every chunk of 256 tokens, similar keys are grouped into centroid clusters. Within each cluster, keys are divided into pages of 16 keys each. We now have a three-tier view of the KV cache, starting from a zoomed out set of keys averaged within each cluster and moving into further detail until we reach the original individual keys and values.

Queries start by scoring themselves against the low resolution centroids. We take the top eight of these and “open” them to reveal the pages within. We then score the query against those page summaries. We take the top two pages within each group, and “open” those, revealing the individual keys and values of each page. We apply exact attention to these, as well as the most recent block sliding window of 256-512 tokens, and one or more sink tokens.

We combine the exactly attended KV pairs with a coarse remainder in a single softmax. Within this coarse remainder, there are two parts: The unopened groups are represented by the group’s average key and value, repeated by the number of keys in the group. The opened groups still have unopened pages, and the same averaging method is used to add these into the softmax.

The result is a mixture of exact attention applied to the local window, sinks, and highest detail pages, and averaged attention applied to the groups and pages everywhere else. When the groups have zero variance, this ends up being equivalent to full attention. As variance increases, it becomes a conservative estimate of attention mass.

Results

We developed fast kernels for AMD hardware and tested LoD on vLLM for both Qwen3.8-27B and K2 Horizon 32B. At 128K tokens context length, LoD accelerated Qwen3.8 attention computation by over 6x for both prefill and decode.

Graph 1: Qwen3.8-27B attention timings.

We also tested quality by evaluating perplexity on various ProLong documents, and ran LongBench v2 and RULER.

Table 1: Perplexity on 64k ProLong documents

Table 2: LongBench v2

Table 3: RULER 64k

Additional Benefits

We also tested using our centroids to enable highly compressed KV Cache storage. Because the centroids keep similar keys together, it turns out that we get good results even at INT4! And in our three-tier resolution mode, only the selected pages need to be decompressed for any given token.

Video LoD

Our results on text got us excited to see if LoD could be used for other modalities! So we developed a video version of the idea, and implemented it against MinMax H3. Because video has a lot of spatiotemporal locality, we leveraged that by keeping queries and KV’s in 4x4x4 blocks to keep GPU efficiency high. Long story short, it works great, and rivals other concurrent work like Nvidia’s recent SoL Attention. This is a comparison between videos generated from the same prompt with roughly 3x end-to-end speedup:

Left to right: dense BF16, LoD BF16 (c=16), SoL BF16 matched. Play the comparison

Summary

Level of Detail Attention offers a way to accelerate attention by looking at important regions in exact detail, while preserving approximated attention on the coarse remainder. It can be applied to pretrained models with no training or changes to their weights, and we provide example code and inference kernels for Qwen and K2 Horizon via vLLM, and MiniMax H3 via ComfyUI.

Please see our paper for lots more details and experiments!

Citation

[MUST FILL THIS IN WITH THE FINAL ONE]

@misc{goldstein2026lod,
  title  = {Level of Detail Attention: Allocating Resolution Instead of Sparsity},
  author = {Daniel Goldstein and Masahiro Shimose and Eugene Cheah},
  year   = {2026},
  eprint = {},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url    = {https://arxiv.org/abs/}
}