Explainers
The concepts serving actually turns on, as things you can operate. Every page here is a simulation with the inputs exposed. Change them and watch the system misbehave, which is the part a diagram cannot show you.
- beginnerA batch is only as fast as its slowest member
Run the same fourteen requests under static and continuous batching, and watch where a GPU's idle time actually comes from.
- beginnerThe KV cache is the thing you are actually renting
Fill a GPU with weights and per-request cache until it will not take another user. 320 KiB per token adds up faster than anyone expects.
- beginnerReserve for the worst case, or page
Forty requests of unknown length against one memory, under two allocators at once. The PagedAttention paper's 20 to 40% utilisation figure, reproduced rather than quoted.
- intermediateTwo phases, two bottlenecks
Move sequence length and batch size across a roofline and watch the limit flip from compute to memory bandwidth, the reason disaggregation exists.
- beginnerSmaller weights run faster, a smaller cache does not
Two different things are called quantization, and on your machine one is worth 3x and the other is worth nothing. Move the weight precision, the cache precision and the context, and watch which one moves the clock.
- beginnerCompute the system prompt once
Eight users, one system prompt, several turns each. Step through the requests and watch what each one actually computes once the cache is addressed by prefix.
- intermediateThe hit rate is the workload
Agent sessions taking turns against one card's memory. Add sessions and watch each turn's cost jump from nothing to ten seconds as its history is evicted.
- intermediateVerifying five tokens costs the same as making one
A small model guesses ahead, the large one checks every guess in one pass. Turn the draft, the lookahead, the acceptance rate and the batch, and watch the speedup come from idle compute and vanish once the batch uses it.
- intermediateA long prompt should not stall everyone else
Sixteen streams mid-reply when a 16K prompt lands. Run its prefill alone and they freeze for two seconds; chunk it into decode steps and the longest gap is 30 milliseconds. The right chunk size is the ridge point.
- advancedSplit the phases, and the cache has to travel
Prefill on one pool, decode on another, and the whole KV cache in between. Send it after the prefill and it costs 15%; send it layer by layer and the same bytes over the same wire cost 0.19%.
- advancedCross-Model KV Cache Transfer in LLM Families
Escalate a request from a 14B model to its 32B sibling and price the handoff both ways: re-prefill the context, or map the small model's cache across with a per-head linear fit.
- intermediateEvery layer stops to agree, twice
Split a model across 1 to 8 GPUs and watch the decode step: each shard reads less, and 160 all-reduces a token cost the same whether the batch is 1 or 256.
- intermediateThe load balancer decides the hit rate
The same sessions against the same fleet under round-robin, sticky sessions and a cache-aware router. One of them throws the prefix cache away.
- intermediateCut every matrix, or cut the stack
The same model, the same eight GPUs, the same requests in flight, cut three ways. Rows are GPUs on one clock: who is busy, who is waiting, and how often a request gets a token.
- intermediateLatency, cost, quality: pick two(not built yet)
Drag a point inside the triangle and see which real configurations land there, priced from the nightly catalogue.