feed/tags/kernels
kernels
10 glossary terms, and 2 readings ordered as a path, each with one line on why it earns your time.
- Tensor cores
The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores.
- GEMM
General matrix multiply, the operation the projections and feed-forward layers of a transformer reduce to.
- FlashAttention
An attention kernel that never writes the full attention matrix to HBM.
- CUDA graphs
Recording a sequence of kernel launches once and replaying the whole sequence with a single call.
- Kernel fusion
Combining several operations into one kernel so intermediate results stay in registers or on-chip memory instead of making a round trip to HBM.
- Kernel launch overhead
The fixed cost of asking the GPU to run a kernel: the driver call, the scheduling, the queue.
- FlashInfer
A kernel library for the operations serving engines share: attention over paged KV cache, in prefill and decode shapes, plus sampling, GEMM and MoE routines.
- Triton
A Python-embedded language and compiler for writing GPU kernels at the level of blocks of data rather than individual threads.
- CUTLASS
NVIDIA's C++ template library for building GEMM and related kernels that hit tensor core peak, and CuTe, its layer for describing the layouts and tilings that make that possible.
- Nsight
NVIDIA's profilers.
Current work, at the depth practitioners actually argue about.
- articleOptimizing on-device inference for Apple silicon
The roofline on one laptop. Decode hits 90% of the chip's sustained weight-read rate and removing the arithmetic changes nothing, so speculative decoding comes out 18% slower at batch 1.
Perplexity Engineering#decode#prefill#quantization#kernels#hardware - articleDefeating Nondeterminism in LLM Inference
Temperature zero is not determinism. A thousand identical requests gave eighty different completions, because kernel output depends on batch size, and batch size depends on who else is on the server.
Horace He#determinism#kernels#batching