cost

A bet that a structured decision should not be generated one token at a time. Jev samples its whole output in a single parallel query rather than autoregressively, and trains for calibrated confidence so the score can be thresholded by software instead of read by a person. The 70 ms to 500 ms end-to-end latency, the 40x to 200x speedup and the free output tokens are the vendor's own figures on an early-access model and nobody outside has audited them. The architectural claim is the part worth arguing with: that most of what an agent pays for is decoding text which was only ever going to be parsed.

Which provider fits your workload? 12,795 calls across 14 serving routes show how cost, first-token latency, generation speed and reliability change with prompt size and caching. Repeated prompts can change the cost ranking, including the comparison between automatic routing and pinned hosts. Explore the interactive charts, download the measurements, and run your own experiments with Compound.

Latency, cost, quality: you get two. The clearest statement of the trade every other decision on this site is downstream of.

Every model swap in a cascade or a router throws the prefill away, because the bigger sibling cannot read the smaller one's cache. This paper fits a per-head ridge regression from the small model's keys and values to the large one's, with RoPE stripped so the map is position-free, on 500 calibration sequences. Four of six pairs keep 73 to 98% of standalone accuracy and the map runs 2.7 to 25x faster than re-prefill; two pairs fall apart, which is the honest part. Prefix caching across models, not just across requests.