glossary/serving/acceptance-rate

Acceptance rate

also acceptance length, draft acceptance

The share of a draft model's guessed tokens that the target model accepts during speculative decoding. It sets how many tokens come out of each verify pass: with per-token acceptance a and lookahead k, about (1 - a^(k+1)) / (1 - a). It is a property of the draft and of the text rather than of the technique; code and templated output accept high, open-ended prose accepts low, and a deployment measures it rather than assumes it.

See it happen

• explainers/
• intermediateVerifying five tokens costs the same as making one

  A small model guesses ahead, the large one checks every guess in one pass. Turn the draft, the lookahead, the acceptance rate and the batch, and watch the speedup come from idle compute and vanish once the batch uses it.

Related

• glossary/
• Speculative decoding

  Having a small draft model guess several tokens ahead, then checking all of them with the large model in one forward pass.

• Decode

  Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.

• Inter-token latency

  The gap between one output token and the next, once streaming has started.

• sources/
• Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2022)
• topics/
• #speculative-decoding#decode#latency