glossary/memory/block-table
Block table
also page table
The per-request list that says which physical block of KV cache holds each run of its tokens. The attention kernel reads it to find the keys and values for position n, so a request's cache can be scattered across memory and still be read in order. It is also where sharing lives: two requests whose tables point at the same block are reading one copy of the same prefix.
16 tokens
The default block size in vLLM; smaller blocks waste less at the tail of a request and cost more table lookups.
See it happen
Related
- glossary/
- PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- sources/
- topics/