glossary/memory/memory-bandwidth
Memory bandwidth
Bytes per second the GPU can move between its HBM and its compute units. For decode it is the whole story: each step reads every weight and every cached key and value once, and the arithmetic per byte read is tiny, so step time is bytes divided by bandwidth. Doubling bandwidth halves inter-token latency at small batch; doubling compute does nothing for it.
140 GB per step
The weight read for Llama 3 70B in BF16, so 6 TB/s across four A100s gives a 23.3 ms floor per token.
See it happen
Related
- glossary/
- High bandwidth memory
The stacked DRAM on the GPU package that holds the weights and the KV cache.
- Memory-bound
Limited by how fast bytes can be read, not by how fast they can be operated on.
- Arithmetic intensity
Floating-point operations performed per byte moved from memory.
- Roofline model
A chart with arithmetic intensity on the x axis and attainable performance on the y, bounded by a sloped line from memory bandwidth and a flat line from peak compute.
- sources/
- topics/