glossary/scale/all-reduce

All-reduce

also allreduce, NCCL all-reduce, ring all-reduce

The collective that tensor parallelism runs twice per transformer layer: once after the attention output projection and once after the MLP down projection, so that each GPU's partial product of its weight shard becomes the full activation on every GPU. A ring implementation moves 2(N-1)/N of the tensor per GPU in 2N-2 steps; NVSwitch multicast does it in two. For decode the message is batch x hidden x 2 bytes, 16 KB at batch 1 on a 70B model, so the cost is the fixed latency of starting the collective, about 12.5 microseconds on NVSwitch, paid 160 times a token.

See it happen

• explainers/
• intermediateEvery layer stops to agree, twice

  Split a model across 1 to 8 GPUs and watch the decode step: each shard reads less, and 160 all-reduces a token cost the same whether the batch is 1 or 256.

Related

• glossary/
• Tensor parallelism

  Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.

• NVLink

  NVIDIA's GPU-to-GPU interconnect, an order of magnitude faster than PCIe, and the switch fabric that joins every GPU in a node or rack to every other at full rate.

• All-to-all

  The collective in which every GPU sends a different piece of data to every other GPU.

• Interconnect bandwidth

  Bytes per second between GPUs, whether over NVLink inside a node, InfiniBand or Ethernet between nodes, or PCIe to the host.

• sources/
• TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference (Gond et al., 2025)
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (Shoeybi et al., 2019)
• topics/
• #hardware#gpu#kernels