glossary/compute/kernel-fusion
Kernel fusion
Combining several operations into one kernel so intermediate results stay in registers or on-chip memory instead of making a round trip to HBM. A layer norm followed by a projection, or an attention score followed by its softmax, each written separately costs a full write and read of the activations. Fused, the bytes never leave the chip. Most of the speedups attributed to custom kernels are fusions.
one write, one read
Saved per fused pair, on a machine where moving a byte costs more than operating on it.
Related
- glossary/
- FlashAttention
An attention kernel that never writes the full attention matrix to HBM.
- Memory-bound
Limited by how fast bytes can be read, not by how fast they can be operated on.
- Triton
A Python-embedded language and compiler for writing GPU kernels at the level of blocks of data rather than individual threads.
- GEMM
General matrix multiply, the operation the projections and feed-forward layers of a transformer reduce to.
- sources/
- topics/