Feed

Four things routinely called 'caching' that solve four different problems. Untangling them is most of the way to reading anyone else's cost numbers.

Batching is what makes serving efficient and is exactly what lets one tenant ruin another's latency. A clear account of scheduling against that tension.

A bet that a structured decision should not be generated one token at a time. Jev samples its whole output in a single parallel query rather than autoregressively, and trains for calibrated confidence so the score can be thresholded by software instead of read by a person. The 70 ms to 500 ms end-to-end latency, the 40x to 200x speedup and the free output tokens are the vendor's own figures on an early-access model and nobody outside has audited them. The architectural claim is the part worth arguing with: that most of what an agent pays for is decoding text which was only ever going to be parsed.

Which provider fits your workload? 12,795 calls across 14 serving routes show how cost, first-token latency, generation speed and reliability change with prompt size and caching. Repeated prompts can change the cost ranking, including the comparison between automatic routing and pinned hosts. Explore the interactive charts, download the measurements, and run your own experiments with Compound.

About 130 links from CUDA basics to disaggregated serving, and every one is the paper, the spec, the repo or a worklog with measurements. Sections cap at five links, so adding one means arguing which to drop. The eight-item start-here path is the shortest route to the vocabulary the rest of this feed assumes.

Rotate the die and hover the parts. Memory bandwidth stops being a number you quote and starts being a place on the chip.

Latency, cost, quality: you get two. The clearest statement of the trade every other decision on this site is downstream of.

The KV cache is the scarce resource, not the weights. Everything about batching downstream of this follows from that one reframing.

Where continuous batching comes from. Read it before you tune a max-batch-size you do not understand.

An agent's context becomes a file it can edit, not just a history its harness periodically summarizes. Existing models learn to keep notes, remove irrelevant tool output and update state in place. On BrowseComp-Plus with Qwen3.6-27B and a 32K context budget, that beats Codex-style summarization by 11.4% relative accuracy with 21.5% fewer prefix-reuse FLOPs. The serving consequence is the part to read here: editing the middle breaks prefix reuse and forces the suffix through prefill again. Suffix Cache Reuse keeps surviving tokens' old states instead, cutting compute by another 35% in the paper's SGLang comparison at matched task accuracy. Those states still encode the previous prefix, so this is approximate reuse, not an exact cache hit, and FLOPs are not a latency or billing measurement.

Evicting cache entries gets harder when sparse attention and shared latent KV force several heads and layers to keep the same token set. Ramp adapts KVzip and Attention Matching to GLM 5.3: normalizing each head's scores before selection improves ComplexMCP accuracy by 5 percentage points at 30% prefix compression, while fitting new biases and shared values recovers behavior eviction alone loses. On five QuALITY articles and 90 questions across four runs, Attention Matching reaches 83.9% accuracy with an 80% smaller article cache, against 91.8% for full context. At 95% compression it reaches 77.8%, versus 67.8% for eviction alone. The tradeoff includes preparation: the 80% setting takes about 257 seconds per article for key selection alone. A useful account of what shared-cache compression preserves, what it loses and what it costs, without a measured end-to-end serving speedup.

A production stack for a 1M-context multimodal model brought up from scratch on 100,000 Chinese-made accelerators in under two weeks, tripling end-to-end throughput, then serving 62 trillion tokens in six days. The framing is that an agent did much of the work; the substance is the three debugging cases, and they are worth the read on their own. A TF32 default inside tl.dot quietly accumulating error through long-context state merges. An acceptance criterion of 5% on the prefill-to-decode handoff that measured over 20%, because a held interpreter lock kept the transfer thread from being scheduled, until releasing it took the gap under 1%.

A decoder that carries its final hidden state across every token, prompt and response alike, so depth grows with sequence length instead of being fixed by layer count. Worth reading here for what it costs rather than what it saves: recurrence prevents the decoder from processing all positions in parallel, so the prompt is consumed token by token and prefill as we know it stops existing, with only the causal encoder keeping sequence-level parallelism and a sliding window bounding the decoder cache. The author is explicit that this is a specification, reporting no throughput, latency or memory, and not claiming to reduce arithmetic or latency against a standard transformer. Read it as a design to argue with, not a result.

Long-horizon agents make the workload input-heavy, so the bytes of cache per token are the bill. V4.1-Flash attacks that in the architecture: the lower 20 of 40 layers run as an encoder and the decoder's global KV is projected from the encoder state, so prefill does about half the layer work; CSA2 shares main KV and indexer keys across layers, and Reindex layers rescore only a candidate pool the first full layer chose, so per-query cost stops growing with context; main KV is MXFP4 and the sliding-window cache is replayed on resume rather than stored. The result is 890 bytes per token of global KV in HBM at 1M context, a quarter of V4-Flash, an eighth in persistent storage. The absolute V4-Flash figure is not in the paper, so the ratios are what you get.

A decode step on one H100 moves 6.6 GB of weights and 0.5 GB of cache, so its speed of light is about 470 tokens a second; vLLM gets 185 of them and the gaps between kernels are where the rest go. One persistent kernel with counter barriers, weight prefetch across tasks and idle SMs backfilled from the next layer gets 292, 1.58x at batch 1 and 1.32x at batch 8, with the code published. The clearest account yet of what a memory-bound step loses to launch boundaries.

The roofline on one laptop. Decode hits 90% of the chip's sustained weight-read rate and removing the arithmetic changes nothing, so speculative decoding comes out 18% slower at batch 1.

Disaggregation only pays if the KV cache moves between pools fast enough. This is the transfer layer underneath Dynamo doing that job.

A useful corrective: the older, simpler mechanism holds up against linear attention on long context, without the post-training bill.

Cold start is the tax on serving many models from few GPUs. Splitting weights from the skeleton takes frontier-model wake-up from minutes to seconds.

An agent turn is 88K tokens in and 413 out, so the benchmark is really a KV cache hit-rate benchmark: 91% on B300 at 384 sessions, 73% on B200 with the rest spilling to DRAM. Nvidia, AMD, five open models, 70 upstream PRs. Partly paywalled.

Disaggregation is usually argued for on principle. This argues it on conditions (scale, network, and traffic balance) and is honest that below them it does not pay.

Every model swap in a cascade or a router throws the prefill away, because the bigger sibling cannot read the smaller one's cache. This paper fits a per-head ridge regression from the small model's keys and values to the large one's, with RoPE stripped so the map is position-free, on 500 calibration sequences. Four of six pairs keep 73 to 98% of standalone accuracy and the map runs 2.7 to 25x faster than re-prefill; two pairs fall apart, which is the honest part. Prefix caching across models, not just across requests.

Nobody profiles the tokenizer. A hardware-aware rewrite finds 37x, which says something about where else nobody is looking.

An agent that pauses for a tool call gets its cache evicted and pays to rebuild it. Scheduling the whole program instead of each request is the fix.

Temperature zero is not determinism. A thousand identical requests gave eighty different completions, because kernel output depends on batch size, and batch size depends on who else is on the server.

You are billed for a model you cannot inspect. A 258-byte proof per 32 tokens that catches a swapped model, an altered prompt, or quietly reduced precision.

Names the generation stall and removes it: prefill in chunks that ride inside decode steps under a per-step token budget, so a long prompt never freezes the batch. 2.6x the serving capacity of vLLM for Mistral 7B on one A100 under tail-latency SLOs, 3.7x for Yi-34B on two. OSDI 2024; every engine has it now.

Prefix caching as a first-class scheduling primitive. If your agent workload is 80% cache hits, this is the paper explaining your bill.

Prefill and decode want different hardware. The clearest statement of why disaggregation keeps winning.

The same idea arrived at independently at DeepMind, on Chinchilla 70B with a 4B draft across a pod: two to two and a half times faster in a distributed setting where the draft has to be worth its own communication. The clearest write-up of the modified rejection sampling.

The paper that made a small model guessing ahead respectable: the target verifies the guesses in one pass and a rejection-sampling step keeps its output distribution exactly. Two to three times faster decode on T5-XXL with no retraining, and the expected tokens-per-step formula every later variant is measured against.