glossary/memory/hbm
High bandwidth memory
also HBM, HBM3, HBM3e
The stacked DRAM on the GPU package that holds the weights and the KV cache. Its capacity caps how many tokens can be in flight; its bandwidth caps how fast a decode step can run, because every step reads all of the weights once. Generations differ mostly here: capacity and bandwidth per card have both grown several-fold, and that, more than arithmetic, is what makes a newer card serve faster.
1.5 TB/s
On the A100 40GB used throughout this site; the H100 SXM has 3.35 TB/s and 80 GB, and the site's arithmetic scales with it.
See it happen
Related
- glossary/
- Memory bandwidth
Bytes per second the GPU can move between its HBM and its compute units.
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- Memory-bound
Limited by how fast bytes can be read, not by how fast they can be operated on.
- KV cache offloading
Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.
- sources/
- topics/