glossary/compute/gemm
GEMM
also general matrix multiply, matmul
General matrix multiply, the operation the projections and feed-forward layers of a transformer reduce to. Its shape decides its efficiency: a tall input of many tokens against a weight matrix reuses each weight many times and runs near peak; a single-token input reads the whole matrix for one row of output and runs at memory speed. That shape difference is prefill versus decode, seen from inside one kernel.
M = tokens in the step
The dimension batching grows; at M = 1 a GEMM is a matrix-vector product and bandwidth-bound.
See it happen
Related
- glossary/
- Tensor cores
The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores.
- Arithmetic intensity
Floating-point operations performed per byte moved from memory.
- CUTLASS
NVIDIA's C++ template library for building GEMM and related kernels that hit tensor core peak, and CuTe, its layer for describing the layouts and tilings that make that possible.
- Kernel fusion
Combining several operations into one kernel so intermediate results stay in registers or on-chip memory instead of making a round trip to HBM.
- sources/
- topics/