GPU

Split a model across 1 to 8 GPUs and watch the decode step: each shard reads less, and 160 all-reduces a token cost the same whether the batch is 1 or 256.

The same model, the same eight GPUs, the same requests in flight, cut three ways. Rows are GPUs on one clock: who is busy, who is waiting, and how often a request gets a token.

The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores.

General matrix multiply, the operation the projections and feed-forward layers of a transformer reduce to.

Recording a sequence of kernel launches once and replaying the whole sequence with a single call.

Combining several operations into one kernel so intermediate results stay in registers or on-chip memory instead of making a round trip to HBM.

The fixed cost of asking the GPU to run a kernel: the driver call, the scheduling, the queue.

A single GPU kernel that runs an entire decode step, with every operation expressed as tasks that the streaming multiprocessors pull from a queue and fine-grained counters standing in for kernel boundaries.

The collective that tensor parallelism runs twice per transformer layer: once after the attention output projection and once after the MLP down projection, so that each GPU's partial product of its weight shard becomes the full activation on every GPU.

A Python-embedded language and compiler for writing GPU kernels at the level of blocks of data rather than individual threads.

NVIDIA's C++ template library for building GEMM and related kernels that hit tensor core peak, and CuTe, its layer for describing the layouts and tilings that make that possible.

NVIDIA's profilers.