glossary/compute/tensor-cores
Tensor cores
The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores. Nearly all of a transformer's arithmetic is matrix multiplication, so peak inference FLOPS means tensor core FLOPS. Each generation adds lower-precision formats with higher rates, which is where FP8 and FP4 speedups come from. A kernel that is not feeding tensor cores is leaving most of the chip idle.
312 versus 19.5 TFLOPS
The A100's dense BF16 rate on tensor cores against its FP32 rate on the general cores, a factor of 16.
See it happen
Related
- glossary/
- GEMM
General matrix multiply, the operation the projections and feed-forward layers of a transformer reduce to.
- Compute-bound
Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck.
- FP8
An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients.
- CUTLASS
NVIDIA's C++ template library for building GEMM and related kernels that hit tensor core peak, and CuTe, its layer for describing the layouts and tilings that make that possible.
- sources/
- topics/