glossary/scale/data-parallelism
Data parallelism
also replicas, DP
Running full copies of the model on separate GPUs or nodes and sending each request to one of them. It is how a deployment scales past the batch one copy can serve, and it is the simplest kind of scale: no communication inside a step, just a router in front. Its cost is memory, since every replica holds the weights, and its subtlety is the router, which decides which replica's prefix cache a request lands on.
140 GB per replica
Of Llama 3 70B weights repeated for every copy, which is why replicas are traded against tensor parallelism.
Related
- glossary/
- Tensor parallelism
Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.
- Prefix cache hit rate
The share of prompt tokens found already in the cache rather than prefilled.
- Request scheduling
Deciding, each step, which waiting requests join the batch and which running ones stay.
- sources/
- topics/