Prompt caching is the productized form of the KV cache. Every major provider now offers it: Anthropic, OpenAI, and Google all let you mark or automatically detect a stable prefix, and on subsequent requests that share it the model skips prefill for that portion entirely. The savings are large and boringly reliable. Cached input typically bills at somewhere around a tenth of normal input cost, and time-to-first-token on a long prompt can drop by an order of magnitude.
The mechanism imposes one rule that governs every design decision around it: matching is prefix-based and exact. The cache is keyed on the token sequence from position zero, so a change at token 40 invalidates everything from token 40 onward. Nothing about semantic similarity applies here. Two prompts that mean the same thing but differ by one word share nothing. This gives you a straightforward discipline, which is to order your prompt from most stable to most volatile: system instructions, then tool definitions, then long reference documents, then conversation history, then the current turn. It also gives you a list of things that silently break caching, and they are all things that look harmless in a template. A rendered timestamp. A session ID in a header block. A tool list assembled from a set and therefore ordered differently on each process. A model version bump, which invalidates every cache entry you have.
Caches also expire, and the TTL shapes your architecture more than people expect. The common default is around five minutes of inactivity, with longer windows available at a higher write price. That works beautifully for a user mid-conversation and works poorly for a batch job that touches each tenant once an hour. Providers also charge a premium on the write, so a prefix that is written once and never reused costs more than not caching at all. Caching is worth it when a prefix is long and reused often, which is why it pays best on system prompts, tool schemas, and reference documents, and pays worst on anything user-specific.
The strategic read matters more than the tactics. Prompt caching changed the economics of stuffing large static context into a prompt, and a lot of teams concluded from that they no longer need retrieval or memory. That does not follow. Caching reduces the price of carrying context; it does nothing about relevance, and relevance is the constraint that actually degrades quality. A 100,000-token cached prompt still dilutes attention, still pushes the important sentence into the middle where the model attends to it least, and still cannot contain the thing your user told you three weeks ago unless something decided to put it there. Cheap context is not the same as the right context.
Why It Matters
Prompt caching is the single highest-leverage cost optimization available to most AI products, and it requires no model change and no infrastructure. It is also easy to leave switched off by accident: a prompt template that interpolates anything dynamic near the top will report a cache hit rate near zero while looking perfectly reasonable in code review. For teams running long system prompts or large tool schemas at scale, cache hit rate belongs on the same dashboard as latency and error rate.
Example
A coding agent sends 30,000 tokens of tool definitions and repository conventions with every request. Uncached, at typical frontier-model input pricing, a team running 50,000 requests a month spends heavily on tokens that are byte-identical every time. Marking that block as cacheable and keeping it strictly ahead of the per-request payload turns nearly all of it into cache reads. Input spend on that block falls by roughly 90 percent and median time-to-first-token drops from about 3 seconds to under half a second, with no change to what the model actually receives.