feed/tags/speculative-decoding
speculative decoding
One explainer you can run, 2 glossary terms, and 3 readings ordered as a path, each with one line on why it earns your time.
Current work, at the depth practitioners actually argue about.
- toolDeepInfer: DeepSeek's inference framework
The engine behind DeepSeek's own production service, open-sourced for DeepSeek-V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950. It is one place to see FlashMLA, DeepGEMM and DeepEP wired together with attention data parallelism, expert parallelism with load balancing, and a scheduler that overlaps batch preparation with the GPU. Two details are worth reading closely. Speculative decoding verifies drafts on a confidence schedule instead of a fixed length. Batch-invariant kernels with deterministic sampling make output independent of what else is in the batch, on NVIDIA only. This release runs offline only. The KV cache system, prefill and decode disaggregation and load balancing across replicas are left out, so it shows the inside of one replica, not the whole service.
#kernels#moe#speculative-decoding#determinism#scheduling - paperAccelerating Large Language Model Decoding with Speculative Sampling
The same idea arrived at independently at DeepMind, on Chinchilla 70B with a 4B draft across a pod: two to two and a half times faster in a distributed setting where the draft has to be worth its own communication. The clearest write-up of the modified rejection sampling.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving and others#decode#latency#speculative-decoding - paperFast Inference from Transformers via Speculative Decoding
The paper that made a small model guessing ahead respectable: the target verifies the guesses in one pass and a rejection-sampling step keeps its output distribution exactly. Two to three times faster decode on T5-XXL with no retraining, and the expected tokens-per-step formula every later variant is measured against.
Yaniv Leviathan, Matan Kalman and Yossi Matias#decode#latency#speculative-decoding