feed/tags/speculative-decoding
speculative decoding
One explainer you can run, 2 glossary terms, and 2 readings ordered as a path, each with one line on why it earns your time.
Current work, at the depth practitioners actually argue about.
- paperAccelerating Large Language Model Decoding with Speculative Sampling
The same idea arrived at independently at DeepMind, on Chinchilla 70B with a 4B draft across a pod: two to two and a half times faster in a distributed setting where the draft has to be worth its own communication. The clearest write-up of the modified rejection sampling.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving and others#decode#latency#speculative-decoding - paperFast Inference from Transformers via Speculative Decoding
The paper that made a small model guessing ahead respectable: the target verifies the guesses in one pass and a rejection-sampling step keeps its output distribution exactly. Two to three times faster decode on T5-XXL with no retraining, and the expected tokens-per-step formula every later variant is measured against.
Yaniv Leviathan, Matan Kalman and Yossi Matias#decode#latency#speculative-decoding