glossary/scale/pipeline-parallelism
Pipeline parallelism
also PP
Putting different layers on different GPUs and passing activations along the chain. It needs only a small point-to-point transfer between stages rather than an all-reduce, so it spans slower links. The cost is that a stage sits idle while the others work on the same request, so it needs several requests in flight to keep every stage busy. For serving it is usually combined with tensor parallelism within a node.
one activation
Per token per stage boundary, kilobytes rather than the gigabytes of weights tensor parallelism synchronises over.
Related
- glossary/
- Tensor parallelism
Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.
- Expert parallelism
Placing the experts of a mixture-of-experts model on different GPUs, so each card holds a few experts rather than a slice of all of them.
- Interconnect bandwidth
Bytes per second between GPUs, whether over NVLink inside a node, InfiniBand or Ethernet between nodes, or PCIe to the host.
- sources/
- topics/