A transformer generates one token at a time, and every new token attends to every token before it. Done naively, producing token 1,001 means recomputing attention over the first 1,000 tokens from scratch, then doing it again for token 1,002. The KV cache is the fix, and it is almost embarrassingly simple: when the model processes a token, it computes a key vector and a value vector for that token at every layer, and those vectors never change afterward. So you keep them. Each subsequent token computes only its own query and reads the stored keys and values for everything that came before.
This splits inference into two phases with completely different performance characteristics, and most confusion about LLM latency dissolves once you separate them. Prefill processes your whole prompt in parallel and fills the cache. It is compute-bound, it scales with prompt length, and it is what you are waiting on during time-to-first-token. Decode then emits one token at a time, reading the entire cache at every step. It is memory-bandwidth-bound, and its speed barely depends on how long the prompt was. That is why a 50,000-token prompt takes noticeably longer to start answering but streams at roughly the same tokens per second as a short one.
The cache is not free, and its size is where serving economics actually live. You are storing two vectors per token, per layer, per attention head. For a 70B-class model in fp16 that lands in the neighborhood of 2.5 GB for a single 10,000-token conversation, before you have served a second user. Model weights are a fixed cost you pay once; KV cache is a per-request cost that scales with context length times concurrency, which means the number of simultaneous long-context sessions a GPU can hold is usually bounded by cache memory rather than by the model. This is the reason grouped-query and multi-query attention exist, why vLLM built PagedAttention to stop fragmenting cache memory, and why quantizing the cache itself has become a standard lever.
The practical consequence for anyone building on an API is prompt ordering. Because keys and values depend only on a token and everything before it, two requests that share a prefix produce byte-identical cache entries for that prefix, which is exactly what makes prompt caching possible. Put stable content first, volatile content last. A single dynamic token near the top of your system prompt, a timestamp or a user name, invalidates every cached entry after it and quietly turns a cheap request into a full prefill.
There is one more thing worth being precise about, because the vocabulary invites the mistake. A KV cache is not memory. It is a compute optimization scoped to a single request, holding no semantics, discarded when the session ends, and incapable of telling you anything about what the user said last Tuesday. It makes carrying context cheaper within a conversation. Deciding what deserves to be in that context across conversations is a different layer of the stack entirely.
Why It Matters
KV cache is where inference cost and latency are actually determined, and it is invisible in most application code. Teams optimizing token counts while ignoring cache behavior routinely leave an order of magnitude on the table, because the same prompt content in a different order can be the difference between a 4-second first token and a 400-millisecond one. It also sets the hard ceiling on concurrency for long-context products: if your assistant carries 100k tokens of context per user, your serving capacity is a cache-memory problem, not a model problem.
Example
A support assistant ships with a 12,000-token system prompt: policies, tone guide, and product documentation. The template opens with a line rendering the customer name and current timestamp. Cache hit rate is effectively zero, every request pays a full 12,000-token prefill, and time-to-first-token sits around 4.2 seconds. Moving those two dynamic fields to the end of the prompt changes nothing about what the model sees, but the 12,000-token prefix is now identical across requests. Hit rate climbs above 90 percent, first token arrives in roughly 0.6 seconds, and input cost on cached requests drops by about 90 percent.