feed/tags/routing
routing
2 explainers you can run, 2 glossary terms, and one reading ordered as a path, each with one line on why it earns your time.
- advancedCross-Model KV Cache Transfer in LLM Families
Escalate a request from a 14B model to its 32B sibling and price the handoff both ways: re-prefill the context, or map the small model's cache across with a per-head linear fit.
- intermediateThe load balancer decides the hit rate
The same sessions against the same fleet under round-robin, sticky sessions and a cache-aware router. One of them throws the prefix cache away.
- KV-cache-aware routing
A load balancer that knows which replica's memory holds which prefixes, and routes a request to the one with the longest match instead of the next one in line.
- Cross-model KV cache transfer
Reusing the KV cache a small model built for a prompt as the starting cache of a larger model in the same family, so a cascade or router can escalate a request without the large model prefilling the context again.