glossary/serving/request-scheduling
Request scheduling
also scheduler, admission control
Deciding, each step, which waiting requests join the batch and which running ones stay. The scheduler balances the memory the KV cache needs against the latency each request was promised. First-come-first-served is the default; it is fair and it lets one long prompt delay everyone. Admitting greedily fills the cache and forces preemptions later. A watermark of free memory held back is the usual compromise.
4 versus 32
Preemptions over forty requests in the PagedAttention explainer, with and without a free-block watermark.
See it happen
Related
- glossary/
- Continuous batching
Scheduling at the granularity of a single decode step instead of a whole batch.
- Preemption
Evicting a running request from the batch when the KV cache runs out, so the others can keep growing.
- Chunked prefill
Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.
- Service level objective
A latency promise stated as a percentile: the first token within 500 ms for 99% of requests, or each following token within 50 ms.
- sources/
- topics/