glossary/tooling/flashinfer
FlashInfer
A kernel library for the operations serving engines share: attention over paged KV cache, in prefill and decode shapes, plus sampling, GEMM and MoE routines. It exists so that vLLM, SGLang and others do not each maintain their own attention kernels, and it is where techniques like block-sparse attention and cascade attention over shared prefixes are implemented once. Its kernels are JIT-compiled per shape and GPU.
one attention kernel
Shared by the major open engines, instead of one per engine.
Related
- glossary/
- FlashAttention
An attention kernel that never writes the full attention matrix to HBM.
- PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.
- vLLM
The open-source serving engine from UC Berkeley that introduced PagedAttention and made continuous batching with paged KV cache the default architecture.
- SGLang
A serving engine and frontend language built around RadixAttention, its radix-tree prefix cache, and a compressed finite-state machine for fast constrained decoding.
- sources/
- FlashInfer on GitHub
- FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving (Ye et al., 2025)
- topics/