glossary/tooling/vllm
vLLM
The open-source serving engine from UC Berkeley that introduced PagedAttention and made continuous batching with paged KV cache the default architecture. It serves an OpenAI-compatible API, supports most open models and quantization formats, and its scheduler, block manager and prefix cache are the reference implementations most explanations of serving describe. Its V1 engine rewrote the core around a simpler scheduler and CUDA graph capture.
2 to 4x
The throughput gain over the systems of the time reported in the 2023 paper, from memory management alone.
See it happen
Related
- glossary/
- PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.
- Continuous batching
Scheduling at the granularity of a single decode step instead of a whole batch.
- SGLang
A serving engine and frontend language built around RadixAttention, its radix-tree prefix cache, and a compressed finite-state machine for fast constrained decoding.
- FlashInfer
A kernel library for the operations serving engines share: attention over paged KV cache, in prefill and decode shapes, plus sampling, GEMM and MoE routines.
- sources/
- topics/