How model serving actually behaves once it meets production traffic.

An agent's context becomes a file it can edit, not just a history its harness periodically summarizes. Existing models learn to keep notes, remove irrelevant tool output and update state in place. On BrowseComp-Plus with Qwen3.6-27B and a 32K context budget, that beats Codex-style summarization by 11.4% relative accuracy with 21.5% fewer prefix-reuse FLOPs. The serving consequence is the part to read here: editing the middle breaks prefix reuse and forces the suffix through prefill again. Suffix Cache Reuse keeps surviving tokens' old states instead, cutting compute by another 35% in the paper's SGLang comparison at matched task accuracy. Those states still encode the previous prefix, so this is approximate reuse, not an exact cache hit, and FLOPs are not a latency or billing measurement.

Evicting cache entries gets harder when sparse attention and shared latent KV force several heads and layers to keep the same token set. Ramp adapts KVzip and Attention Matching to GLM 5.3: normalizing each head's scores before selection improves ComplexMCP accuracy by 5 percentage points at 30% prefix compression, while fitting new biases and shared values recovers behavior eviction alone loses. On five QuALITY articles and 90 questions across four runs, Attention Matching reaches 83.9% accuracy with an 80% smaller article cache, against 91.8% for full context. At 95% compression it reaches 77.8%, versus 67.8% for eviction alone. The tradeoff includes preparation: the 80% setting takes about 257 seconds per article for key selection alone. A useful account of what shared-cache compression preserves, what it loses and what it costs, without a measured end-to-end serving speedup.

A production stack for a 1M-context multimodal model brought up from scratch on 100,000 Chinese-made accelerators in under two weeks, tripling end-to-end throughput, then serving 62 trillion tokens in six days. The framing is that an agent did much of the work; the substance is the three debugging cases, and they are worth the read on their own. A TF32 default inside tl.dot quietly accumulating error through long-context state merges. An acceptance criterion of 5% on the prefill-to-decode handoff that measured over 20%, because a held interpreter lock kept the transfer thread from being scheduled, until releasing it took the gap under 1%.

A bet that a structured decision should not be generated one token at a time. Jev samples its whole output in a single parallel query rather than autoregressively, and trains for calibrated confidence so the score can be thresholded by software instead of read by a person. The 70 ms to 500 ms end-to-end latency, the 40x to 200x speedup and the free output tokens are the vendor's own figures on an early-access model and nobody outside has audited them. The architectural claim is the part worth arguing with: that most of what an agent pays for is decoding text which was only ever going to be parsed.

A decoder that carries its final hidden state across every token, prompt and response alike, so depth grows with sequence length instead of being fixed by layer count. Worth reading here for what it costs rather than what it saves: recurrence prevents the decoder from processing all positions in parallel, so the prompt is consumed token by token and prefill as we know it stops existing, with only the causal encoder keeping sequence-level parallelism and a sliding window bounding the decoder cache. The author is explicit that this is a specification, reporting no throughput, latency or memory, and not claiming to reduce arithmetic or latency against a standard transformer. Read it as a design to argue with, not a result.