glossary/tooling/sglang
SGLang
A serving engine and frontend language built around RadixAttention, its radix-tree prefix cache, and a compressed finite-state machine for fast constrained decoding. It began as the runtime for programs that call a model many times with shared prefixes, agents and few-shot pipelines, and grew into a general engine that competes with vLLM on throughput and leads on prefix reuse and structured output.
up to 6.4x
Higher throughput than the baselines of the time on workloads with heavy prefix sharing, per the paper.
See it happen
Related
- glossary/
- RadixAttention
SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder.
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- vLLM
The open-source serving engine from UC Berkeley that introduced PagedAttention and made continuous batching with paged KV cache the default architecture.
- FlashInfer
A kernel library for the operations serving engines share: attention over paged KV cache, in prefill and decode shapes, plus sampling, GEMM and MoE routines.
- sources/
- SGLang on GitHub
- SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023)
- topics/