feed/tags/moe

moe

One reading ordered as a path, each with one line on why it earns your time.

advanced/ 1 entry

Current work, at the depth practitioners actually argue about.

articleInside the megakernel serving engine for North Mini Code
A decode step on one H100 moves 6.6 GB of weights and 0.5 GB of cache, so its speed of light is about 470 tokens a second; vLLM gets 185 of them and the gaps between kernels are where the rest go. One persistent kernel with counter barriers, weight prefetch across tasks and idle SMs backfilled from the next layer gets 292, 1.58x at batch 1 and 1.32x at batch 8, with the code published. The clearest account yet of what a memory-bound step loses to launch boundaries.
Xiaochun Tong, Conway Zhu and Donglu Wang8 Sep 2026#kernels#decode#gpu#hbm#moe

Related topics