feed/tags/batching
batching
3 explainers you can run, 4 glossary terms, and 5 readings ordered as a path, each with one line on why it earns your time.
- beginnerA batch is only as fast as its slowest member
Run the same fourteen requests under static and continuous batching, and watch where a GPU's idle time actually comes from.
- intermediateVerifying five tokens costs the same as making one
A small model guesses ahead, the large one checks every guess in one pass. Turn the draft, the lookahead, the acceptance rate and the batch, and watch the speedup come from idle compute and vanish once the batch uses it.
- intermediateA long prompt should not stall everyone else
Sixteen streams mid-reply when a 16K prompt lands. Run its prefill alone and they freeze for two seconds; chunk it into decode steps and the longest gap is 30 milliseconds. The right chunk size is the ridge point.
- Throughput
Tokens produced per second across all requests in flight, or requests completed per second.
- Continuous batching
Scheduling at the granularity of a single decode step instead of a whole batch.
- Static batching
Collecting a batch of requests, running them together until every one has finished, then collecting the next.
- Request scheduling
Deciding, each step, which waiting requests join the batch and which running ones stay.
Start here. No serving experience assumed.
The mental models the rest of the field takes for granted.
- talkProducing the World's Cheapest Tokens: A How-to Guide
Latency, cost, quality: you get two. The clearest statement of the trade every other decision on this site is downstream of.
Meryem Arik#cost#latency#inference-triangle#batching - paperEfficient Memory Management for Large Language Model Serving with PagedAttention
The KV cache is the scarce resource, not the weights. Everything about batching downstream of this follows from that one reframing.
#kv-cache#vllm#batching - paperOrca: A Distributed Serving System for Transformer-Based Generative Models
Where continuous batching comes from. Read it before you tune a max-batch-size you do not understand.
#batching#scheduling
Current work, at the depth practitioners actually argue about.