glossary/scale/pipeline-parallelism
Pipeline parallelism
also PP
Putting different layers on different GPUs and passing activations along the chain. It needs only a small point-to-point transfer between stages rather than an all-reduce, so it spans slower links. The cost is that a stage sits idle while the others work on the same request, so it needs several requests in flight to keep every stage busy. For serving it is usually combined with tensor parallelism within a node.
one activation
Per token per stage boundary, a few kilobytes and one send, against two all-reduces per layer under tensor parallelism.
See it happen
Related
- glossary/
- Tensor parallelism
Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.
- Pipeline bubble
The time pipeline stages sit idle because the first micro-batch has not reached them yet or the last has already left.
- Expert parallelism
Placing the experts of a mixture-of-experts model on different GPUs, so each card holds a few experts rather than a slice of all of them.
- All-reduce
The collective that tensor parallelism runs twice per transformer layer: once after the attention output projection and once after the MLP down projection, so that each GPU's partial product of its weight shard becomes the full activation on every GPU.
- Interconnect bandwidth
Bytes per second between GPUs, whether over NVLink inside a node, InfiniBand or Ethernet between nodes, or PCIe to the host.
- sources/
- topics/