feed/tags/memory
memory
4 explainers you can run, 8 glossary terms, and 2 readings ordered as a path, each with one line on why it earns your time.
- beginnerThe KV cache is the thing you are actually renting
Fill a GPU with weights and per-request cache until it will not take another user. 320 KiB per token adds up faster than anyone expects.
- beginnerReserve for the worst case, or page
Forty requests of unknown length against one memory, under two allocators at once. The PagedAttention paper's 20 to 40% utilisation figure, reproduced rather than quoted.
- beginnerSmaller weights run faster, a smaller cache does not
Two different things are called quantization, and on your machine one is worth 3x and the other is worth nothing. Move the weight precision, the cache precision and the context, and watch which one moves the clock.
- intermediateThe hit rate is the workload
Agent sessions taking turns against one card's memory. Add sessions and watch each turn's cost jump from nothing to ten seconds as its history is evicted.
- Preemption
Evicting a running request from the batch when the KV cache runs out, so the others can keep growing.
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.
- Block table
The per-request list that says which physical block of KV cache holds each run of its tokens.
- Cache eviction
Dropping stored KV cache to make room for new requests.
- High bandwidth memory
The stacked DRAM on the GPU package that holds the weights and the KV cache.
- KV cache offloading
Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.
- Working set
The amount of KV cache the active sessions need resident to keep hitting.
Current work, at the depth practitioners actually argue about.
- articleScaling KV Cache Compression to Frontier Models
Evicting cache entries gets harder when sparse attention and shared latent KV force several heads and layers to keep the same token set. Ramp adapts KVzip and Attention Matching to GLM 5.3: normalizing each head's scores before selection improves ComplexMCP accuracy by 5 percentage points at 30% prefix compression, while fitting new biases and shared values recovers behavior eviction alone loses. On five QuALITY articles and 90 questions across four runs, Attention Matching reaches 83.9% accuracy with an 80% smaller article cache, against 91.8% for full context. At 95% compression it reaches 77.8%, versus 67.8% for eviction alone. The tradeoff includes preparation: the 80% setting takes about 257 seconds per article for key selection alone. A useful account of what shared-cache compression preserves, what it loses and what it costs, without a measured end-to-end serving speedup.
Jeff Zheng and Alex Chen#kv-cache#attention#long-context#memory#agents - paperSliding-window beats linear attention
A useful corrective: the older, simpler mechanism holds up against linear attention on long context, without the post-training bill.
Alexia Jolicoeur-Martineau and colleagues#attention#long-context#memory