glossary/compute/roofline
Roofline model
A chart with arithmetic intensity on the x axis and attainable performance on the y, bounded by a sloped line from memory bandwidth and a flat line from peak compute. Any workload sits under the lower of the two; where the lines meet is the ridge. It says in one picture why prefill and decode, running the same weights, hit different limits, and what a faster card would actually change for each.
312 TFLOPS and 1.5 TB/s
The A100's two ceilings for dense BF16, meeting at a ridge of 208 FLOP per byte.
See it happen
Related
- glossary/
- Arithmetic intensity
Floating-point operations performed per byte moved from memory.
- Ridge point
The arithmetic intensity at which a machine's memory ceiling and compute ceiling meet: peak FLOPS divided by bandwidth.
- Memory-bound
Limited by how fast bytes can be read, not by how fast they can be operated on.
- Compute-bound
Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck.
- sources/
- topics/