feed/tags/latency
latency
8 explainers you can run, 10 glossary terms, and 6 readings ordered as a path, each with one line on why it earns your time.
- beginnerA batch is only as fast as its slowest member
Run the same fourteen requests under static and continuous batching, and watch where a GPU's idle time actually comes from.
- intermediateTwo phases, two bottlenecks
Move sequence length and batch size across a roofline and watch the limit flip from compute to memory bandwidth, the reason disaggregation exists.
- beginnerCompute the system prompt once
Eight users, one system prompt, several turns each. Step through the requests and watch what each one actually computes once the cache is addressed by prefix.
- intermediateVerifying five tokens costs the same as making one
A small model guesses ahead, the large one checks every guess in one pass. Turn the draft, the lookahead, the acceptance rate and the batch, and watch the speedup come from idle compute and vanish once the batch uses it.
- intermediateA long prompt should not stall everyone else
Sixteen streams mid-reply when a 16K prompt lands. Run its prefill alone and they freeze for two seconds; chunk it into decode steps and the longest gap is 30 milliseconds. The right chunk size is the ridge point.
- intermediateEvery layer stops to agree, twice
Split a model across 1 to 8 GPUs and watch the decode step: each shard reads less, and 160 all-reduces a token cost the same whether the batch is 1 or 256.
- intermediateThe load balancer decides the hit rate
The same sessions against the same fleet under round-robin, sticky sessions and a cache-aware router. One of them throws the prefix cache away.
- intermediateCut every matrix, or cut the stack
The same model, the same eight GPUs, the same requests in flight, cut three ways. Rows are GPUs on one clock: who is busy, who is waiting, and how often a request gets a token.
- Time to first token
The delay between a request arriving and the first token of its answer leaving.
- Inter-token latency
The gap between one output token and the next, once streaming has started.
- Time per output token
The average time each output token took, computed as total generation time divided by tokens generated, excluding the first.
- Goodput
Throughput counted only for requests that met their latency targets.
- Service level objective
A latency promise stated as a percentile: the first token within 500 ms for 99% of requests, or each following token within 50 ms.
- Chunked prefill
Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.
- Generation stall
A pause in every decoding stream in a batch while the scheduler runs another request's prefill as its own iteration.
- Speculative decoding
Having a small draft model guess several tokens ahead, then checking all of them with the large model in one forward pass.
- Acceptance rate
The share of a draft model's guessed tokens that the target model accepts during speculative decoding.
- Kernel launch overhead
The fixed cost of asking the GPU to run a kernel: the driver call, the scheduling, the queue.
The mental models the rest of the field takes for granted.
- articleIntroducing System One Models and Jev
A bet that a structured decision should not be generated one token at a time. Jev samples its whole output in a single parallel query rather than autoregressively, and trains for calibrated confidence so the score can be thresholded by software instead of read by a person. The 70 ms to 500 ms end-to-end latency, the 40x to 200x speedup and the free output tokens are the vendor's own figures on an early-access model and nobody outside has audited them. The architectural claim is the part worth arguing with: that most of what an agent pays for is decoding text which was only ever going to be parsed.
Diogo Almeida#latency#cost#agents#decode - interactiveDeepSeek V4 Flash across 14 providers: cost, speed and caching
Which provider fits your workload? 12,795 calls across 14 serving routes show how cost, first-token latency, generation speed and reliability change with prompt size and caching. Repeated prompts can change the cost ranking, including the comparison between automatic routing and pinned hosts. Explore the interactive charts, download the measurements, and run your own experiments with Compound.
Batuhan Aktas#benchmark#prefix-cache#cost#latency - talkProducing the World's Cheapest Tokens: A How-to Guide
Latency, cost, quality: you get two. The clearest statement of the trade every other decision on this site is downstream of.
Meryem Arik#cost#latency#inference-triangle#batching
Current work, at the depth practitioners actually argue about.
- paperTaming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Names the generation stall and removes it: prefill in chunks that ride inside decode steps under a per-step token budget, so a long prompt never freezes the batch. 2.6x the serving capacity of vLLM for Mistral 7B on one A100 under tail-latency SLOs, 3.7x for Yi-34B on two. OSDI 2024; every engine has it now.
Amey Agrawal, Nitin Kedia, Ashish Panwar and others#chunked-prefill#prefill#scheduling#latency - paperAccelerating Large Language Model Decoding with Speculative Sampling
The same idea arrived at independently at DeepMind, on Chinchilla 70B with a 4B draft across a pod: two to two and a half times faster in a distributed setting where the draft has to be worth its own communication. The clearest write-up of the modified rejection sampling.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving and others#decode#latency#speculative-decoding - paperFast Inference from Transformers via Speculative Decoding
The paper that made a small model guessing ahead respectable: the target verifies the guesses in one pass and a rejection-sampling step keeps its output distribution exactly. Two to three times faster decode on T5-XXL with no retraining, and the expected tokens-per-step formula every later variant is measured against.
Yaniv Leviathan, Matan Kalman and Yossi Matias#decode#latency#speculative-decoding