hardware

Move sequence length and batch size across a roofline and watch the limit flip from compute to memory bandwidth, the reason disaggregation exists.

A small model guesses ahead, the large one checks every guess in one pass. Turn the draft, the lookahead, the acceptance rate and the batch, and watch the speedup come from idle compute and vanish once the batch uses it.

Split a model across 1 to 8 GPUs and watch the decode step: each shard reads less, and 160 all-reduces a token cost the same whether the batch is 1 or 256.

The same model, the same eight GPUs, the same requests in flight, cut three ways. Rows are GPUs on one clock: who is busy, who is waiting, and how often a request gets a token.

The stacked DRAM on the GPU package that holds the weights and the KV cache.

Bytes per second the GPU can move between its HBM and its compute units.

A chart with arithmetic intensity on the x axis and attainable performance on the y, bounded by a sloped line from memory bandwidth and a flat line from peak compute.

Floating-point operations performed per byte moved from memory.

The arithmetic intensity at which a machine's memory ceiling and compute ceiling meet: peak FLOPS divided by bandwidth.

Limited by how fast bytes can be read, not by how fast they can be operated on.

Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck.

The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores.

An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients.

A 4-bit float with sixteen representable values, usable only with fine-grained scaling: a shared scale factor per small block of values restores the range the format lacks.

Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.

Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient.

The collective that tensor parallelism runs twice per transformer layer: once after the attention output projection and once after the MLP down projection, so that each GPU's partial product of its weight shard becomes the full activation on every GPU.

NVIDIA's GPU-to-GPU interconnect, an order of magnitude faster than PCIe, and the switch fabric that joins every GPU in a node or rack to every other at full rate.

Bytes per second between GPUs, whether over NVLink inside a node, InfiniBand or Ethernet between nodes, or PCIe to the host.

NVIDIA's inference library for its own GPUs: compiled engines with hand-tuned kernels, the first place FP8 and FP4 land, in-flight batching, paged KV cache, and the wide expert parallelism and disaggregated serving that its newest racks are built for.