moe
2 readings ordered as a path, each with one line on why it earns your time.
Current work, at the depth practitioners actually argue about.
- toolDeepInfer: DeepSeek's inference framework
The engine behind DeepSeek's own production service, open-sourced for DeepSeek-V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950. It is one place to see FlashMLA, DeepGEMM and DeepEP wired together with attention data parallelism, expert parallelism with load balancing, and a scheduler that overlaps batch preparation with the GPU. Two details are worth reading closely. Speculative decoding verifies drafts on a confidence schedule instead of a fixed length. Batch-invariant kernels with deterministic sampling make output independent of what else is in the batch, on NVIDIA only. This release runs offline only. The KV cache system, prefill and decode disaggregation and load balancing across replicas are left out, so it shows the inside of one replica, not the whole service.
#kernels#moe#speculative-decoding#determinism#scheduling - articleInside the megakernel serving engine for North Mini Code
A decode step on one H100 moves 6.6 GB of weights and 0.5 GB of cache, so its speed of light is about 470 tokens a second; vLLM gets 185 of them and the gaps between kernels are where the rest go. One persistent kernel with counter barriers, weight prefetch across tasks and idle SMs backfilled from the next layer gets 292, 1.58x at batch 1 and 1.32x at batch 8, with the code published. The clearest account yet of what a memory-bound step loses to launch boundaries.
Xiaochun Tong, Conway Zhu and Donglu Wang#kernels#decode#gpu#hbm#moe