Prompt caching in practice

~7 min read

Previous lesson: reused exact prefixes skip prefill at ~90% off. This one is the field manual — how to actually get those hits, and how to verify them on the Bhaskara API.

The three rules

Stable system prompt, always first. Same bytes every turn. Per-session toggles (terse mode, doc packs) are fine — they change once and stay. Per-request fiddling is cache poison.
Append-only history. New turns go at the end, old ones are never edited, reordered, or regenerated mid-stream. Compaction (ours fires past 200K tokens) is the deliberate exception: one budgeted re-warm, then stable again.
Volatile content goes last. Timestamps, request IDs, progress counters — anything that changes every turn belongs at the tail, never in the middle of settled context.

Reading the proof on our API

Every response's usage carries prompt_tokens_details.cached_tokens — the tokens served from cache this turn. Our theta display rates make the value visceral:

theta display rates (per 1M tokens):
  input      $0.20
  cache hit  $0.04     ← 5× cheaper
  output     $0.40

A healthy agentic session shows 70–90% of prompt tokens arriving as cache hits. If yours doesn't, the cause is almost always one of the three rules above — usually volatile content smuggled into the middle of history (timestamps in tool output are the classic offender; our context engine masks them before forwarding).

Cache hits are a property of your session, not our mood

Same bytes → hit, deterministically. If hit rate collapses on a session, diff what changed at the turn it dropped: a reordered tool schema, a regenerated message, a new system toggle. Fix the bytes, the hits return.

Check your understanding

Progress saves on this device only

1.Where should a per-request timestamp live in your context?
2.Your session's cached_tokens suddenly drops to ~0. First suspect?
3.Why do per-session (sticky) settings preserve cache while per-request ones destroy it?