feed/tags/kernels
kernels
One explainer you can run, 12 glossary terms, and 5 readings ordered as a path, each with one line on why it earns your time.
- Tensor cores
The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores.
- GEMM
General matrix multiply, the operation the projections and feed-forward layers of a transformer reduce to.
- FlashAttention
An attention kernel that never writes the full attention matrix to HBM.
- CUDA graphs
Recording a sequence of kernel launches once and replaying the whole sequence with a single call.
- Kernel fusion
Combining several operations into one kernel so intermediate results stay in registers or on-chip memory instead of making a round trip to HBM.
- Kernel launch overhead
The fixed cost of asking the GPU to run a kernel: the driver call, the scheduling, the queue.
- Megakernel
A single GPU kernel that runs an entire decode step, with every operation expressed as tasks that the streaming multiprocessors pull from a queue and fine-grained counters standing in for kernel boundaries.
- All-reduce
The collective that tensor parallelism runs twice per transformer layer: once after the attention output projection and once after the MLP down projection, so that each GPU's partial product of its weight shard becomes the full activation on every GPU.
- FlashInfer
A kernel library for the operations serving engines share: attention over paged KV cache, in prefill and decode shapes, plus sampling, GEMM and MoE routines.
- Triton
A Python-embedded language and compiler for writing GPU kernels at the level of blocks of data rather than individual threads.
- CUTLASS
NVIDIA's C++ template library for building GEMM and related kernels that hit tensor core peak, and CuTe, its layer for describing the layouts and tilings that make that possible.
- Nsight
NVIDIA's profilers.
The mental models the rest of the field takes for granted.
Current work, at the depth practitioners actually argue about.
- articleToward Recursive Self-Improvement: How GLM Built Its Own Inference Infrastructure
A production stack for a 1M-context multimodal model brought up from scratch on 100,000 Chinese-made accelerators in under two weeks, tripling end-to-end throughput, then serving 62 trillion tokens in six days. The framing is that an agent did much of the work; the substance is the three debugging cases, and they are worth the read on their own. A TF32 default inside tl.dot quietly accumulating error through long-context state merges. An acceptance criterion of 5% on the prefill-to-decode handoff that measured over 20%, because a held interpreter lock kept the transfer thread from being scheduled, until releasing it took the gap under 1%.
#disaggregation#kernels#kv-cache#agents#hardware - articleInside the megakernel serving engine for North Mini Code
A decode step on one H100 moves 6.6 GB of weights and 0.5 GB of cache, so its speed of light is about 470 tokens a second; vLLM gets 185 of them and the gaps between kernels are where the rest go. One persistent kernel with counter barriers, weight prefetch across tasks and idle SMs backfilled from the next layer gets 292, 1.58x at batch 1 and 1.32x at batch 8, with the code published. The clearest account yet of what a memory-bound step loses to launch boundaries.
Xiaochun Tong, Conway Zhu and Donglu Wang#kernels#decode#gpu#hbm#moe - articleOptimizing on-device inference for Apple silicon
The roofline on one laptop. Decode hits 90% of the chip's sustained weight-read rate and removing the arithmetic changes nothing, so speculative decoding comes out 18% slower at batch 1.
Perplexity Engineering#decode#prefill#quantization#kernels#hardware - articleDefeating Nondeterminism in LLM Inference
Temperature zero is not determinism. A thousand identical requests gave eighty different completions, because kernel output depends on batch size, and batch size depends on who else is on the server.
Horace He#determinism#kernels#batching