feed/tags/benchmark

benchmark

2 glossary terms, and 2 readings ordered as a path, each with one line on why it earns your time.

glossary/ 2 terms

Time per output token
The average time each output token took, computed as total generation time divided by tokens generated, excluding the first.
Goodput
Throughput counted only for requests that met their latency targets.

intermediate/ 1 entry

The mental models the rest of the field takes for granted.

interactiveDeepSeek V4 Flash across 14 providers: cost, speed and caching
Which provider fits your workload? 12,795 calls across 14 serving routes show how cost, first-token latency, generation speed and reliability change with prompt size and caching. Repeated prompts can change the cost ranking, including the comparison between automatic routing and pinned hosts. Explore the interactive charts, download the measurements, and run your own experiments with Compound.
Batuhan Aktas7 Sep 2026#benchmark#prefix-cache#cost#latency

advanced/ 1 entry

Current work, at the depth practitioners actually argue about.

articleAgentX and InferenceX v3: does the CUDA moat hold up in agentic inference?
An agent turn is 88K tokens in and 413 out, so the benchmark is really a KV cache hit-rate benchmark: 91% on B300 at 384 sessions, 73% on B200 with the rest spilling to DRAM. Nvidia, AMD, five open models, 70 upstream PRs. Partly paywalled.
Cam Quilici, Bryan Shan, Alec Ibarra and others24 Aug 2026#agents#prefix-cache#kv-cache#benchmark#hardware

Related topics