Sparse attention still has a storage bill

With the default settings, 32,768 stored positions at 16 KiB per token occupy 512 MiB. Reading 2,048 positions fetches an idealised 32 MiB of cache state per decode step. It is one sixteenth of the read traffic, but still 512 MiB of storage. A 24 GiB cache budget holds 48 such sessions under either read policy.

Increase context to 131,072 tokens with the same read budget. The read total stays at 32 MiB while storage grows to 2 GiB. The cache budget now holds only 12 sessions. Fixed per-query attention does not imply fixed per-session memory.

Lower physical retention to 50%. Now storage falls, so session capacity rises. If more than 2,048 entries remain, the sparse query can still read 2,048. Compression has moved the storage ledger without changing this query’s read count.

A sparse indexer routes a particular query to a subset of positions. A different query can need a different subset. A position with no observed attention might have been excluded by routing rather than found irrelevant. Treating every unselected position as disposable confuses today’s route with tomorrow’s requirements.

Ramp’s GLM 5.3 experiments describe a maximum of 2,048 positions per query, with selected positions shared within groups of layers. Their reconstruction scoring uses dense attention so excluded positions can still receive a score. The lab illustrates that distinction, without pretending to reproduce the learned indexer.

Read fewer positions. Sparse selection changes which cached positions contribute to a query. The positions can remain available for later queries.

Store fewer positions. Eviction or compaction removes entries or builds a smaller representation. Future queries must work with what survives.

Use fewer bytes per position. Quantization and latent representations can reduce the size of stored state. That changes storage even if the position count stays fixed.

Architectures can combine these. Sparse attention is a family of designs; some also store compressed summaries or bound their cache with a sliding window. This page isolates sparse selection over a stored token cache, rather than claiming that every sparse model has the same memory behaviour.

The read count times bytes per token is an ideal cache payload. It excludes indexer state, selection overhead, weights, workspace, repeated reads and cache-line effects. A gather kernel does not necessarily deliver the same bandwidth as a contiguous read. Smaller payloads alone do not predict token latency.

The coloured routes and retained sets are invented, and no accuracy score is attached to them. Real selection has to preserve information across heads, layers and future queries. The point of changing the query here is to see that the route moves while storage remains.