Glossary

The delay between a request arriving and the first token of its answer leaving.

The gap between one output token and the next, once streaming has started.

The average time each output token took, computed as total generation time divided by tokens generated, excluding the first.

Tokens produced per second across all requests in flight, or requests completed per second.

Throughput counted only for requests that met their latency targets.

A latency promise stated as a percentile: the first token within 500 ms for 99% of requests, or each following token within 50 ms.

Scheduling at the granularity of a single decode step instead of a whole batch.

Collecting a batch of requests, running them together until every one has finished, then collecting the next.

Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.

Having a small draft model guess several tokens ahead, then checking all of them with the large model in one forward pass.

Running prefill and decode on separate pools of GPUs and shipping the KV cache between them.

Deciding, each step, which waiting requests join the batch and which running ones stay.

Evicting a running request from the batch when the KV cache runs out, so the others can keep growing.

The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.

Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.

The per-request list that says which physical block of KV cache holds each run of its tokens.

Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.

The share of prompt tokens found already in the cache rather than prefilled.

SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder.

Dropping stored KV cache to make room for new requests.

The stacked DRAM on the GPU package that holds the weights and the KV cache.

Bytes per second the GPU can move between its HBM and its compute units.

Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.

The amount of KV cache the active sessions need resident to keep hitting.

The forward pass over the whole prompt at once, producing the KV cache for every token and the first output token.

Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.

A chart with arithmetic intensity on the x axis and attainable performance on the y, bounded by a sloped line from memory bandwidth and a flat line from peak compute.

Floating-point operations performed per byte moved from memory.

The arithmetic intensity at which a machine's memory ceiling and compute ceiling meet: peak FLOPS divided by bandwidth.

Limited by how fast bytes can be read, not by how fast they can be operated on.

Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck.

The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores.

General matrix multiply, the operation the projections and feed-forward layers of a transformer reduce to.

An attention kernel that never writes the full attention matrix to HBM.

Recording a sequence of kernel launches once and replaying the whole sequence with a single call.

Combining several operations into one kernel so intermediate results stay in registers or on-chip memory instead of making a round trip to HBM.

The fixed cost of asking the GPU to run a kernel: the driver call, the scheduling, the queue.

A 16-bit float with the same 8-bit exponent as FP32 and only 7 bits of mantissa.

An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients.

A 4-bit float with sixteen representable values, usable only with fine-grained scaling: a shared scale factor per small block of values restores the range the format lacks.

Storing the weights in four or eight bits and converting them back to 16 on the way into the multiply, leaving activations at full precision.

Quantizing the activations flowing between layers as well as the weights, so the matmul itself runs in low precision on the tensor cores.

Storing the cached keys and values in 8 or 4 bits instead of 16.

Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.

Putting different layers on different GPUs and passing activations along the chain.

Running full copies of the model on separate GPUs or nodes and sending each request to one of them.

Placing the experts of a mixture-of-experts model on different GPUs, so each card holds a few experts rather than a slice of all of them.

Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient.

In a mixture-of-experts layer, a small gating network scores each token against every expert and sends it to the top few.

The collective in which every GPU sends a different piece of data to every other GPU.

NVIDIA's GPU-to-GPU interconnect, an order of magnitude faster than PCIe, and the switch fabric that joins every GPU in a node or rack to every other at full rate.

Bytes per second between GPUs, whether over NVLink inside a node, InfiniBand or Ethernet between nodes, or PCIe to the host.

The open-source serving engine from UC Berkeley that introduced PagedAttention and made continuous batching with paged KV cache the default architecture.

A serving engine and frontend language built around RadixAttention, its radix-tree prefix cache, and a compressed finite-state machine for fast constrained decoding.

NVIDIA's inference library for its own GPUs: compiled engines with hand-tuned kernels, the first place FP8 and FP4 land, in-flight batching, paged KV cache, and the wide expert parallelism and disaggregated serving that its newest racks are built for.

A kernel library for the operations serving engines share: attention over paged KV cache, in prefill and decode shapes, plus sampling, GEMM and MoE routines.

A Python-embedded language and compiler for writing GPU kernels at the level of blocks of data rather than individual threads.

NVIDIA's C++ template library for building GEMM and related kernels that hit tensor core peak, and CuTe, its layer for describing the layouts and tilings that make that possible.

NVIDIA's profilers.

The HTTP interface every serving engine exposes: chat completions with a list of messages, streaming over server-sent events, the same request and response shapes OpenAI defined.