feed/tags/hbm

HBM

One explainer you can run, 2 glossary terms, and 2 readings ordered as a path, each with one line on why it earns your time.

explainers/ 1 page

beginnerThe KV cache is the thing you are actually renting
Fill a GPU with weights and per-request cache until it will not take another user. 320 KiB per token adds up faster than anyone expects.

glossary/ 2 terms

High bandwidth memory
The stacked DRAM on the GPU package that holds the weights and the KV cache.
Memory bandwidth
Bytes per second the GPU can move between its HBM and its compute units.

intermediate/ 1 entry

The mental models the rest of the field takes for granted.

interactiveInteractive H100 architecture glossary
Rotate the die and hover the parts. Memory bandwidth stops being a number you quote and starts being a place on the chip.
Kyle Jeong, with definitions from Modal's GPU Glossary31 Aug 2026#gpu#hardware#hbm

advanced/ 1 entry

Current work, at the depth practitioners actually argue about.

articleInside the megakernel serving engine for North Mini Code
A decode step on one H100 moves 6.6 GB of weights and 0.5 GB of cache, so its speed of light is about 470 tokens a second; vLLM gets 185 of them and the gaps between kernels are where the rest go. One persistent kernel with counter barriers, weight prefetch across tasks and idle SMs backfilled from the next layer gets 292, 1.58x at batch 1 and 1.32x at batch 8, with the code published. The clearest account yet of what a memory-bound step loses to launch boundaries.
Xiaochun Tong, Conway Zhu and Donglu Wang8 Sep 2026#kernels#decode#gpu#hbm#moe

Related topics