attention

Evicting cache entries gets harder when sparse attention and shared latent KV force several heads and layers to keep the same token set. Ramp adapts KVzip and Attention Matching to GLM 5.3: normalizing each head's scores before selection improves ComplexMCP accuracy by 5 percentage points at 30% prefix compression, while fitting new biases and shared values recovers behavior eviction alone loses. On five QuALITY articles and 90 questions across four runs, Attention Matching reaches 83.9% accuracy with an 80% smaller article cache, against 91.8% for full context. At 95% compression it reaches 77.8%, versus 67.8% for eviction alone. The tradeoff includes preparation: the 80% setting takes about 257 seconds per article for key selection alone. A useful account of what shared-cache compression preserves, what it loses and what it costs, without a measured end-to-end serving speedup.

A decoder that carries its final hidden state across every token, prompt and response alike, so depth grows with sequence length instead of being fixed by layer count. Worth reading here for what it costs rather than what it saves: recurrence prevents the decoder from processing all positions in parallel, so the prompt is consumed token by token and prefill as we know it stops existing, with only the causal encoder keeping sequence-level parallelism and a sliding window bounding the decoder cache. The author is explicit that this is a specification, reporting no throughput, latency or memory, and not claiming to reduce arithmetic or latency against a standard transformer. Read it as a design to argue with, not a result.

Long-horizon agents make the workload input-heavy, so the bytes of cache per token are the bill. V4.1-Flash attacks that in the architecture: the lower 20 of 40 layers run as an encoder and the decoder's global KV is projected from the encoder state, so prefill does about half the layer work; CSA2 shares main KV and indexer keys across layers, and Reindex layers rescore only a candidate pool the first full layer chose, so per-query cost stops growing with context; main KV is MXFP4 and the sliding-window cache is replayed on resume rather than stored. The result is 890 bytes per token of global KV in HBM at 1M context, a quarter of V4-Flash, an eighth in persistent storage. The absolute V4-Flash figure is not in the paper, so the ratios are what you get.

A useful corrective: the older, simpler mechanism holds up against linear attention on long context, without the post-training bill.