agents

A bet that a structured decision should not be generated one token at a time. Jev samples its whole output in a single parallel query rather than autoregressively, and trains for calibrated confidence so the score can be thresholded by software instead of read by a person. The 70 ms to 500 ms end-to-end latency, the 40x to 200x speedup and the free output tokens are the vendor's own figures on an early-access model and nobody outside has audited them. The architectural claim is the part worth arguing with: that most of what an agent pays for is decoding text which was only ever going to be parsed.

An agent's context becomes a file it can edit, not just a history its harness periodically summarizes. Existing models learn to keep notes, remove irrelevant tool output and update state in place. On BrowseComp-Plus with Qwen3.6-27B and a 32K context budget, that beats Codex-style summarization by 11.4% relative accuracy with 21.5% fewer prefix-reuse FLOPs. The serving consequence is the part to read here: editing the middle breaks prefix reuse and forces the suffix through prefill again. Suffix Cache Reuse keeps surviving tokens' old states instead, cutting compute by another 35% in the paper's SGLang comparison at matched task accuracy. Those states still encode the previous prefix, so this is approximate reuse, not an exact cache hit, and FLOPs are not a latency or billing measurement.

Evicting cache entries gets harder when sparse attention and shared latent KV force several heads and layers to keep the same token set. Ramp adapts KVzip and Attention Matching to GLM 5.3: normalizing each head's scores before selection improves ComplexMCP accuracy by 5 percentage points at 30% prefix compression, while fitting new biases and shared values recovers behavior eviction alone loses. On five QuALITY articles and 90 questions across four runs, Attention Matching reaches 83.9% accuracy with an 80% smaller article cache, against 91.8% for full context. At 95% compression it reaches 77.8%, versus 67.8% for eviction alone. The tradeoff includes preparation: the 80% setting takes about 257 seconds per article for key selection alone. A useful account of what shared-cache compression preserves, what it loses and what it costs, without a measured end-to-end serving speedup.

A production stack for a 1M-context multimodal model brought up from scratch on 100,000 Chinese-made accelerators in under two weeks, tripling end-to-end throughput, then serving 62 trillion tokens in six days. The framing is that an agent did much of the work; the substance is the three debugging cases, and they are worth the read on their own. A TF32 default inside tl.dot quietly accumulating error through long-context state merges. An acceptance criterion of 5% on the prefill-to-decode handoff that measured over 20%, because a held interpreter lock kept the transfer thread from being scheduled, until releasing it took the gap under 1%.

Long-horizon agents make the workload input-heavy, so the bytes of cache per token are the bill. V4.1-Flash attacks that in the architecture: the lower 20 of 40 layers run as an encoder and the decoder's global KV is projected from the encoder state, so prefill does about half the layer work; CSA2 shares main KV and indexer keys across layers, and Reindex layers rescore only a candidate pool the first full layer chose, so per-query cost stops growing with context; main KV is MXFP4 and the sliding-window cache is replayed on resume rather than stored. The result is 890 bytes per token of global KV in HBM at 1M context, a quarter of V4-Flash, an eighth in persistent storage. The absolute V4-Flash figure is not in the paper, so the ratios are what you get.

An agent turn is 88K tokens in and 413 out, so the benchmark is really a KV cache hit-rate benchmark: 91% on B300 at 384 sessions, 73% on B200 with the rest spilling to DRAM. Nvidia, AMD, five open models, 70 upstream PRs. Partly paywalled.

An agent that pauses for a tool call gets its cache evicted and pays to rebuild it. Scheduling the whole program instead of each request is the fix.