Cut every matrix, or cut the stack

A pipeline stage reads its weights once per micro-batch, not once per token. Decode is bound by reading the weights. Under tensor parallelism each GPU reads its eighth of them once and every request in the batch gets a token. A pipeline of eight stages has eight micro-batches in flight, so each stage reads its eighth of the weights eight times per round, once for each micro-batch passing through. The round produces the same tokens for eight times the weight traffic, and the time between a request’s tokens is the whole round. That is why the pipeline rows in the lab are all busy and still slow: busy re-reading. The gap closes as the requests in flight grow, because the cache traffic and the arithmetic are the same under every layout and come to dwarf the re-reads, but at small batch it is nearly the full factor of eight. Topcu and others measured it plainly on eight GPUs: pipeline parallelism yields no latency benefit at any depth.

What the pipeline buys is silence on the link. A stage boundary moves one activation per micro-batch, batch times hidden size times two bytes, and it moves it once. The same layers under tensor parallelism all-reduce that tensor twice per layer across every GPU in the group. On NVLink the difference is a rounding error against the weight re-reads. On PCIe, or across servers on InfiniBand, or across servers without it, it is the deployment. vLLM’s Llama 3.1 405B launch measured two H100 nodes without InfiniBand: eight-way tensor parallelism inside each node with a two-stage pipeline across them ran 6.6 times faster than sixteen-way tensor parallelism spanning both, and with InfiniBand the two were similar. Megatron’s rule from 2021 is still the rule: tensor parallelism up to the number of GPUs in a server, pipeline parallelism beyond it.

The hybrid is not a compromise, it is the shape of the hardware. Tensor parallelism within the NVLink island, where the all-reduce is cheap, and a pipeline across islands, where the link is slow, is what the eight-GPU row labelled TP 4 x PP 2 stands for. It pays half the all-reduce tax and one send. In the lab it sits between the two on NVLink and overtakes pure tensor parallelism on PCIe once the batch is large enough for the all-reduce bytes to matter, which is the same crossover the multi-GPU communication page draws from the other side. The pipeline also asks something of the workload: enough requests in flight to fill every stage’s micro-batch. vLLM implements each stage as its own virtual engine with its own scheduler and cache, so a max_num_seqs of 256 under a four-stage pipeline means four batches of 256, and the memory for all of them.