glossary/serving/ttft
Time to first token
also TTFT
The delay between a request arriving and the first token of its answer leaving. It is dominated by prefill, the single forward pass over the whole prompt, so it grows with prompt length and shrinks with prefix cache hits. It is the latency a user feels most, because it is the part spent staring at nothing. A queue in front of the model adds to it directly.
23.3 ms
The floor on four A100s for Llama 3 70B: the time to read 140 GB of weights once, which no cache hit removes.
See it happen
Related
- glossary/
- Prefill
The forward pass over the whole prompt at once, producing the KV cache for every token and the first output token.
- Inter-token latency
The gap between one output token and the next, once streaming has started.
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- Service level objective
A latency promise stated as a percentile: the first token within 500 ms for 99% of requests, or each following token within 50 ms.
- sources/
- topics/