glossary/serving/preemption
Preemption
also swap, recompute
Evicting a running request from the batch when the KV cache runs out, so the others can keep growing. Its cache is either swapped to host memory and copied back later, or dropped and recomputed from the prompt when the request resumes. Either way the user sees their stream pause. Preemptions are the symptom of admitting more than the memory can carry to completion.
youngest first
The request usually chosen, because it has the least cache to throw away and the least work to redo.
See it happen
Related
- glossary/
- Request scheduling
Deciding, each step, which waiting requests join the batch and which running ones stay.
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- Cache eviction
Dropping stored KV cache to make room for new requests.
- KV cache offloading
Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.
- sources/
- topics/