glossary/scale/interconnect-bandwidth
Interconnect bandwidth
Bytes per second between GPUs, whether over NVLink inside a node, InfiniBand or Ethernet between nodes, or PCIe to the host. Every form of parallelism pays it: all-reduces for tensor parallelism, activations for pipeline, all-to-all for experts, and whole KV caches for disaggregation and offload. The hierarchy of bandwidths, from HBM down to the network, is the map of what can be split where.
1.5 TB/s to 12 GB/s
From an A100's HBM to a realised host link, the range the prefix cache explainer's tiers are priced across.
Related
- glossary/
- NVLink
NVIDIA's GPU-to-GPU interconnect, an order of magnitude faster than PCIe, and the switch fabric that joins every GPU in a node or rack to every other at full rate.
- KV cache offloading
Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.
- Disaggregated serving
Running prefill and decode on separate pools of GPUs and shipping the KV cache between them.
- High bandwidth memory
The stacked DRAM on the GPU package that holds the weights and the KV cache.
- sources/
- topics/