The Pulse
NVIDIA details sparse-attention trade-offs for long-context AI
NVIDIA published measurements and design guidance for sparse attention on its GPUs. In one 128K-token prefill example, the indexer and selection stages accounted for 47% of a DeepSeek V3.2 layer's time.

Sparse attention can reduce the work an AI model does on a long input, but finding which parts of that input to use can itself consume substantial time: in one NVIDIA example, the indexer and selection stages accounted for 47% of a DeepSeek V3.2 layer's prefill time. NVIDIA published the measurement and design guidance for its GPUs in a technical blog post on Nov. 17.
The comparison in the post uses an input sequence length of 128K tokens. It examines prefill, the phase in which a model processes its input, as well as decode, when it generates output.
Where the work goes
Dense attention scores each query against every key in the context. Sparse attention instead runs core attention over a smaller set of keys selected for each query, reducing the attention computation and memory traffic on long contexts.
That selection requires its own work. An indexer scores every key, a Top-k stage retains the highest-scoring keys, and core attention operates on the selected keys and values. The indexer's work still scales with the full context length.
The company's primer focuses on sparsity trained into a model, rather than methods applied after training. It uses a production kernel to examine indexer choices, selection costs and the use of individual tokens or blocks in core attention.
Measuring the indexer
An indexer uses query heads to score against a cache of keys, then processes the scores into a logit for each key. The post examines how the number of heads, their dimensions and speculative decoding affect the work.
Unless otherwise noted, the measurements use an FP8 indexer kernel with 64K indexer keys and a compression ratio of 4. The other default settings include a head dimension of 128, one key head, a prefill batch size of 1 and a decode batch size of 128.
In the measurements, decode runtime moved only 2% as the group size—the number of query heads scoring against a shared key head—grew 16-fold, according to the post. Prefill behaved differently: doubling the group size from 8 to 16 increased runtime by 1.66 times.
Fitting work into a tile
Speculative decoding lets the indexer score draft tokens against the same key cache in a single step. NVIDIA says the extra work can be relatively inexpensive while the query rows fit within one pass over that cache.
A Tensor Core tile can hold up to 256 query rows, but with an FP32 epilogue, register pressure limits the practical tile to about 200 rows. The post reports that this accommodates six query tokens at a group size of 32, or 26 at a group size of 8. Beyond that point, the rows no longer fit in one tile and runtime steps up.
With an FP16 epilogue, the tile widens to 256 rows; the post reports a 1.31-times decode indexer speedup at a group size of 64 and four query tokens per step. Using FP16 for accumulation as well reached 1.45 times, but only when the indexer's query-key scaling kept the matrix multiplication outputs within FP16 range.