glossary/compute/prefill
Prefill
also prompt processing, context encoding
The forward pass over the whole prompt at once, producing the KV cache for every token and the first output token. All the prompt's tokens go through the weights together, so the arithmetic per byte of weight read is high and the GPU is compute-bound above a few hundred tokens. Its duration is the time to first token. Prefix caching skips the part already stored; chunking spreads it over steps.
208 tokens
The prompt length above which prefill on an A100 becomes compute-bound, the roofline's ridge point for BF16.
See it happen
Related
- glossary/
- Decode
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.
- Time to first token
The delay between a request arriving and the first token of its answer leaving.
- Compute-bound
Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck.
- Chunked prefill
Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.
- sources/
- topics/