Inference & the KV cache

Serving has two phases with wildly different economics. Understanding them explains half of LLM pricing — including ours.

Prefill vs decode

• Prefill — your whole prompt is processed in one parallel sweep. Fast per token (GPUs love parallelism), and its cost scales with prompt length.
• Decode — tokens come out one by one, each needing a full forward pass over everything so far. Slow, sequential, and the reason long answers take visibly longer than long prompts.

The KV cache — never recompute what you already read

During prefill, attention computes keys and values for every prompt token. Decoding token N+1 needs those same keys and values — so the server keeps them in GPU memory instead of recomputing. That stored state is the KV cache.

Two consequences run the whole industry:

• Memory grows with context. A 100K-token session parks gigabytes of KV state on the GPU. Long contexts are a memory problem before they are a quality problem.
• Reused prefixes skip prefill. If this turn's prompt starts exactly like a cached one, the server reuses the stored keys/values — no recompute. That skip is what providers sell as prompt caching, typically ~90% off input price.

turn 1:  [system + history] → prefill 60K tokens (full price)
turn 2:  [system + history] (identical bytes → cache HIT)
         + new message      → prefill only the new tail ( ~90% off )

Why session shape matters

The cache matches on exact prefix bytes. Timestamps, reordered tool schemas, or reshuffled history in the middle invalidate everything after them — the server re-prefills from the first changed byte. Stable system prompt first, append-only history after: that discipline is worth more than most model upgrades.