feed/tags/sglang
SGLang
One explainer you can run, 2 glossary terms, and 2 readings ordered as a path, each with one line on why it earns your time.
- RadixAttention
SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder.
- SGLang
A serving engine and frontend language built around RadixAttention, its radix-tree prefix cache, and a compressed finite-state machine for fast constrained decoding.
Current work, at the depth practitioners actually argue about.
- paperContext Language Models
An agent's context becomes a file it can edit, not just a history its harness periodically summarizes. Existing models learn to keep notes, remove irrelevant tool output and update state in place. On BrowseComp-Plus with Qwen3.6-27B and a 32K context budget, that beats Codex-style summarization by 11.4% relative accuracy with 21.5% fewer prefix-reuse FLOPs. The serving consequence is the part to read here: editing the middle breaks prefix reuse and forces the suffix through prefill again. Suffix Cache Reuse keeps surviving tokens' old states instead, cutting compute by another 35% in the paper's SGLang comparison at matched task accuracy. Those states still encode the previous prefix, so this is approximate reuse, not an exact cache hit, and FLOPs are not a latency or billing measurement.
Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin and others#agents#long-context#prefix-cache#kv-cache#prefill#sglang - paperSGLang: Efficient Execution of Structured Language Model Programs
Prefix caching as a first-class scheduling primitive. If your agent workload is 80% cache hits, this is the paper explaining your bill.
#prefix-cache#sglang