feed/tags/scheduling
scheduling
2 explainers you can run, 5 glossary terms, and 3 readings ordered as a path, each with one line on why it earns your time.
- beginnerA batch is only as fast as its slowest member
Run the same fourteen requests under static and continuous batching, and watch where a GPU's idle time actually comes from.
- beginnerReserve for the worst case, or page
Forty requests of unknown length against one memory, under two allocators at once. The PagedAttention paper's 20 to 40% utilisation figure, reproduced rather than quoted.
- Continuous batching
Scheduling at the granularity of a single decode step instead of a whole batch.
- Static batching
Collecting a batch of requests, running them together until every one has finished, then collecting the next.
- Chunked prefill
Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.
- Request scheduling
Deciding, each step, which waiting requests join the batch and which running ones stay.
- Preemption
Evicting a running request from the batch when the KV cache runs out, so the others can keep growing.
Start here. No serving experience assumed.
The mental models the rest of the field takes for granted.
Current work, at the depth practitioners actually argue about.