feed/tags/scale
scale
7 glossary terms, and one reading ordered as a path, each with one line on why it earns your time.
- Tensor parallelism
Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.
- Pipeline parallelism
Putting different layers on different GPUs and passing activations along the chain.
- Data parallelism
Running full copies of the model on separate GPUs or nodes and sending each request to one of them.
- Expert parallelism
Placing the experts of a mixture-of-experts model on different GPUs, so each card holds a few experts rather than a slice of all of them.
- Wide expert parallelism
Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient.
- MoE routing
In a mixture-of-experts layer, a small gating network scores each token against every expert and sends it to the top few.
- All-to-all
The collective in which every GPU sends a different piece of data to every other GPU.
Current work, at the depth practitioners actually argue about.