scale

The same model, the same eight GPUs, the same requests in flight, cut three ways. Rows are GPUs on one clock: who is busy, who is waiting, and how often a request gets a token.

Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.

Putting different layers on different GPUs and passing activations along the chain.

The time pipeline stages sit idle because the first micro-batch has not reached them yet or the last has already left.

Running full copies of the model on separate GPUs or nodes and sending each request to one of them.

Placing the experts of a mixture-of-experts model on different GPUs, so each card holds a few experts rather than a slice of all of them.

Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient.

In a mixture-of-experts layer, a small gating network scores each token against every expert and sends it to the top few.

The collective in which every GPU sends a different piece of data to every other GPU.