glossary/serving/speculative-decoding

Speculative decoding

also speculative sampling, draft model

Having a small draft model guess several tokens ahead, then checking all of them with the large model in one forward pass. Accepted tokens are free; the first rejected one is replaced by what the large model would have said, so the output distribution is unchanged. It works because a decode step is bound by reading the weights, so verifying five tokens costs about the same as producing one. The gain is the acceptance rate.

See it happen

• explainers/
• intermediateTwo phases, two bottlenecks

  Move sequence length and batch size across a roofline and watch the limit flip from compute to memory bandwidth, the reason disaggregation exists.

Related

• glossary/
• Decode

  Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.

• Memory-bound

  Limited by how fast bytes can be read, not by how fast they can be operated on.

• Inter-token latency

  The gap between one output token and the next, once streaming has started.

• Arithmetic intensity

  Floating-point operations performed per byte moved from memory.

• sources/
• Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2023)
• Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al., 2023)
• topics/
• #decode#latency