glossary/compute/arithmetic-intensity
Arithmetic intensity
also operational intensity, FLOPs per byte
Floating-point operations performed per byte moved from memory. It is the property of a workload that decides which ceiling it hits: below the ridge, bandwidth; above it, compute. For a transformer, intensity scales with the number of tokens sharing one read of the weights, so prefill of a long prompt is intense and decode at batch 1 is not. Batching decode raises it linearly until the cache read dominates.
2 FLOPs per parameter per token
The multiply and add each weight contributes, the numerator every intensity on this site is built from.
See it happen
Related
- glossary/
- Roofline model
A chart with arithmetic intensity on the x axis and attainable performance on the y, bounded by a sloped line from memory bandwidth and a flat line from peak compute.
- Ridge point
The arithmetic intensity at which a machine's memory ceiling and compute ceiling meet: peak FLOPS divided by bandwidth.
- Memory-bound
Limited by how fast bytes can be read, not by how fast they can be operated on.
- Compute-bound
Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck.
- sources/
- topics/