feed/tags/scheduling
scheduling
4 explainers you can run, 7 glossary terms, and 4 readings ordered as a path, each with one line on why it earns your time.
- beginnerA batch is only as fast as its slowest member
Run the same fourteen requests under static and continuous batching, and watch where a GPU's idle time actually comes from.
- beginnerReserve for the worst case, or page
Forty requests of unknown length against one memory, under two allocators at once. The PagedAttention paper's 20 to 40% utilisation figure, reproduced rather than quoted.
- intermediateA long prompt should not stall everyone else
Sixteen streams mid-reply when a 16K prompt lands. Run its prefill alone and they freeze for two seconds; chunk it into decode steps and the longest gap is 30 milliseconds. The right chunk size is the ridge point.
- intermediateThe load balancer decides the hit rate
The same sessions against the same fleet under round-robin, sticky sessions and a cache-aware router. One of them throws the prefix cache away.
- Continuous batching
Scheduling at the granularity of a single decode step instead of a whole batch.
- Static batching
Collecting a batch of requests, running them together until every one has finished, then collecting the next.
- Chunked prefill
Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.
- Generation stall
A pause in every decoding stream in a batch while the scheduler runs another request's prefill as its own iteration.
- Request scheduling
Deciding, each step, which waiting requests join the batch and which running ones stay.
- Preemption
Evicting a running request from the batch when the KV cache runs out, so the others can keep growing.
- KV-cache-aware routing
A load balancer that knows which replica's memory holds which prefixes, and routes a request to the one with the longest match instead of the next one in line.
Start here. No serving experience assumed.
The mental models the rest of the field takes for granted.
Current work, at the depth practitioners actually argue about.
- articleThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale
An agent that pauses for a tool call gets its cache evicted and pays to rebuild it. Scheduling the whole program instead of each request is the fix.
Hao Kang and colleagues#agents#kv-cache#scheduling - paperTaming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Names the generation stall and removes it: prefill in chunks that ride inside decode steps under a per-step token budget, so a long prompt never freezes the batch. 2.6x the serving capacity of vLLM for Mistral 7B on one A100 under tail-latency SLOs, 3.7x for Yi-34B on two. OSDI 2024; every engine has it now.
Amey Agrawal, Nitin Kedia, Ashish Panwar and others#chunked-prefill#prefill#scheduling#latency