glossary/compute/decode
Decode
also generation, autoregressive decoding
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token. The arithmetic per byte is close to one, so decode is memory-bound at any batch a cache can hold, and its step time is set by bandwidth. Batching more requests per step raises throughput almost for free; longer contexts slow it by adding cache to read.
1 FLOP per byte
At batch 1 in BF16: two operations per parameter over two bytes per parameter, two hundred times below the A100's ridge.
See it happen
Related
- glossary/
- Prefill
The forward pass over the whole prompt at once, producing the KV cache for every token and the first output token.
- Inter-token latency
The gap between one output token and the next, once streaming has started.
- Memory-bound
Limited by how fast bytes can be read, not by how fast they can be operated on.
- Speculative decoding
Having a small draft model guess several tokens ahead, then checking all of them with the large model in one forward pass.
- sources/
- topics/