How to Reduce LLM Token Costs in Long Conversations: What Caching Saves, and Where It Stops

Published 26 September 2026 · Updated 31 October 2026 · Prices are Anthropic's list rates as of 31 October 2026. The method behind every cost figure is at the end of the article.

A long conversation costs far more than its number of turns suggests, because a language model holds nothing between calls: every request carries the system prompt and the entire history again, so total input grows with the square of the number of turns. The fixes, in the order to apply them, are prompt caching, which cuts a 40-turn chat on Claude Sonnet 5.5 from $1.09 to $0.24 in our worked example, then shrinking what each turn adds (clearing old tool output in large batches, not every turn), then stopping an agent from paying the model to re-make decisions it already made, then trimming or compacting old turns, and finally bounding the context with a memory layer once conversations span sessions. Caching loses its grip when users return after the cache has expired, and in our arithmetic on Sonnet 5.5 a bounded context becomes cheaper than cached full history from about 130 turns (65,000 tokens of history) when the cache goes cold between sessions, but not until about 475 turns against a cache that never goes cold, memory layer included.

Three things have moved since the first version ran on 26 September. Anthropic now prices cache reads on Sonnet 5.5 at 0.05 times the input rate, half the Sonnet 5 rate, which makes cached history cheaper and pushes the point where a memory layer pays for itself later. Anthropic's caching documentation now lists a change to the effort setting as something that invalidates the message cache, which turns one of the most common cost tips (lower the effort on easy turns) into a way to pay for the whole history again. And two October papers measured levers that most cost guides do not mention: dropping old tool output from the KV cache beat summarising it on both cost and score, although trimming the same output from the text every turn cost 1.85 times as much as leaving it alone, and reusing decisions an agent harness had already settled cut tokens per task sharply on one harness.

Why a long conversation costs more than its turns

An LLM API is stateless. Anthropic's context window documentation describes each turn's input as "all previous conversation history plus the current user message", with previous turns "preserved completely". If a turn adds S tokens and the system prompt is P tokens, the request at turn k carries roughly P + (k − 1)·S tokens, and a conversation of N turns sends N·P + S·N(N − 1)/2 input tokens in total. The second term is quadratic, and past a handful of turns it is most of the bill.

A concrete conversation makes it tangible: a 2,000-token system prompt, 150-token user messages and 350-token replies. Turn 1 sends 2,150 tokens. Turn 40 sends 21,650. The 40-turn conversation consumes 476,000 input tokens to produce 14,000 output tokens, and on Claude Sonnet 5.5 at $2 input and $10 output per million tokens it costs $1.09, of which 87% is input. Going from 40 to 100 turns multiplies the turns by 2.5 and the bill by 5.2, to $5.73. Sonnet 5 has the same base prices, so these uncached figures hold for both.

Two things make the curve steeper on current models. Thinking tokens are billed as output, and on Claude Opus 4.5 and later, Sonnet 4.6 and later, Haiku 5.5 and the Fable and Mythos models, earlier thinking blocks stay in the conversation by default and count as input on every later turn. In agent loops, tool results do the same: a search result or file read added at turn 5 is paid for again on every turn after it. One more adjustment matters when you estimate from text rather than from logged usage: Anthropic's pricing page notes that Claude 4.7 and later models use a tokenizer that produces approximately 30% more tokens for the same text, so a budget built from word counts on an older model will come in low.

Lever 1: prompt caching, and what it saves

Prompt caching stores a prompt prefix so that later requests read it instead of paying full price for it. On Claude, a 5-minute cache write costs 1.25 times the base input rate and a cache read costs 0.1 times on most models, 0.05 times on Claude Opus 5.5 and Sonnet 5.5, and 0.025 times on Claude Fable 5.1 and Mythos 5.1. Automatic caching (a single top-level cache_control field) places the breakpoint on the last cacheable block and moves it forward as the conversation grows, so each turn writes only the new exchange and reads everything before it.

For the 40-turn conversation above, that takes the Sonnet 5.5 cost from $1.09 to $0.24 and the input share of the bill from 87% to 42%. On Sonnet 5, whose cache reads cost twice as much, the same conversation comes to $0.29. Caching needs no change to what the model sees, which is why it comes first, and cache hits are not deducted against your rate limit.

The 5-minute cache suits rapid turns and agent loops. The 1-hour cache costs 2 times the input rate to write and pays back after the second read, so it suits chat where users pause for more than five minutes but return within the hour.

Prompt caching is not response caching. A response cache (exact-match or semantic) stores whole answers and returns one when a new request looks like an old one, which works for standalone questions such as an FAQ bot's first turn. Inside a conversation every request carries a different history, so a response cache rarely matches past the first turn; a prompt cache is the one that pays off there, because the shared part is the prefix, not the answer.

Where caching stops working

Caching discounts the history; it does not shrink it. Five things limit how far it goes.

• Users leave and come back. When a user returns after the cache has expired, the first request of the new session writes the entire history again at 1.25 times the input rate. For an assistant used in short sessions over days or weeks, that rewrite happens every session and grows with the history.
• Changes invalidate it. The cache follows the order tools, then system, then messages, and a change at one level invalidates that level and everything after it. Changing a tool definition invalidates everything. Toggling web search, citations or fast mode invalidates the system and message caches. Adding an image, changing tool_choice, changing the thinking configuration or changing the effort setting invalidates the message cache, which is the expensive part of a long conversation.
• Misses are silent. A cache that misses returns no error. Watch cache_read_input_tokens in the usage object, and use cache diagnostics: pass the previous response's id as diagnostics.previous_message_id and the API reports where the new request's prefix diverged (model, system prompt, tools or message history). It no longer needs a beta header.
• The window still fills. Current Claude Opus, Sonnet, Haiku and Fable models take 1 million tokens, and a cached history counts toward the window at full size.
• Quality does not improve with volume. Anthropic's own compaction documentation justifies the feature by noting that "response quality degrades as a conversation grows". A cheaper token is still a token the model has to read past.

Two newer features keep the prefix stable where you previously had to break it. On Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5 and 5.1 and Mythos 5 and 5.1, you can append a {"role": "system"} message to messages instead of editing the top-level system prompt, and the system and message caches survive (Sonnet 5 does not support this). For how coding agents keep their prefixes byte-identical across hundreds of requests, see prompt caching in coding agent harnesses.

Lever 2: shrink what each turn adds

Before touching the conversation itself, cut what each turn contributes.

Drop old tool output at the cache level if you can; on a hosted API, drop it in large batches. Tool results are usually the bulkiest thing in an agent's history and the least read. In Cache Engineering for Agents (Fu et al., arXiv 2610.28577, 20 October 2026), a policy that kept the KV cache of the most recent 12 tool responses and dropped older ones from attention cut average cost by 27.7% and raised average benchmark score by 1.7 points against summarising the history once it passed a threshold (96,000 tokens for two of the models, 192,000 for the third). The comparison covered three open models (MiniMax-M2.7, GPT-OSS-120B and MiroThinker-1.7-Mini) and four agent benchmarks: BrowseComp-Plus, xbench-DeepSearch-2510, the 103-task text-only validation subset of GAIA, and SWE-bench Verified. GLM-5.2 judged the answers on the three question-answering benchmarks, SWE-bench Verified was scored by its official test scripts with no model judge, and costs were priced at Groq, MiniMax and Qwen list rates. The paper does not state how many BrowseComp-Plus, xbench-DeepSearch-2510 or SWE-bench Verified tasks went into that headline comparison (it runs its ablations on each benchmark's first 100 tasks), so read the 27.7% as an average across a task mix the paper only partly specifies. The authors also found that older tokens receive less attention than recent ones, and that tool responses receive less attention than assistant messages at comparable positions, which is why dropping them costs so little accuracy.

The mechanism decides whether any of this transfers to you. The paper drops entries inside its own serving engine (AKV), so the retained history keeps its cached state, and you cannot do that through the Claude API. Doing the same thing in text is a different result, and the paper measured it: on the first 100 BrowseComp-Plus tasks with GPT-OSS-120B, trimming the text to the latest 12 tool responses cost 1.85 times as much as keeping the full history, because every trim changed the prefix and the retained context had to be processed again, while summarising at 48,000 tokens saved about 7% and cost 5 points of accuracy. The lesson for a hosted API is not to drop old output every turn. A rolling trim is the expensive way to drop; clearing pays only when each clearing removes a lot at once.

Clear old tool results on a hosted API, in large batches. Claude's context editing (beta header context-management-2025-06-27) clears tool results once the input passes a threshold (100,000 tokens by default) and keeps the three most recent tool uses, replacing each older result with a short note that it was removed. It can also clear old thinking blocks. Unlike the KV-level drop in the paper, clearing here changes the prompt and breaks the cache from the point it clears, so set clear_at_least high enough that each clearing removes more tokens than the fresh cache write costs you.

Cap output and thinking, but set effort once. Output costs five times input on current Claude models, and thinking tokens bill as output. A max_tokens limit reduces both what you pay now and what later turns carry. Effort is the trap: changing output_config.effort from one turn to the next invalidates the message cache, so on a 40-turn conversation a lower effort on an easy turn can cost more in cache rewrites than it saves in thinking. Pick the effort for the conversation and keep it, or, on models that support per-message effort, carry the change in a role: "system" message inside messages, which leaves the cached prefix intact.

Route side tasks to a cheaper model. A summarisation or classification step inside a conversation rarely needs the model that holds the conversation, and sending it as a separate request leaves the main conversation's cache untouched. Claude Haiku 5.5 costs $0.10 input and $0.50 output per million tokens for prompts up to 100,000 tokens (a twentieth of Sonnet 5.5) and $0.50 and $2.50 above that, and anything that can wait belongs on the Batch API at half price.

Compress before content enters, not after. Prompt compression tools shorten text, and that works when applied once, to a document or tool result before it joins the history. Rewriting text that is already in a cached prefix changes the prefix and forces a fresh write, so the order matters more than the compression ratio.

Lever 3: stop paying the model to re-decide

Agent loops carry a second cost that has nothing to do with history: the harness calls the model again after every tool result to decide a next step that an earlier call often settled already. HarnessMM (Jiang, Xu, Gao and Zhao, arXiv 2610.32813, 23 October 2026) stores those settled decisions in what it calls an Executable Decision Graph (the exact action, its literal arguments, its dependencies and a guard) and reuses a decision when its guard holds, sending anything new back to the model. On the same 98 HumanEval test tasks, with each harness pair on the same pinned backbone (DeepSeek-flash in the paper's headline figure) and scored by the benchmark's own tests, it cut tokens per task on DeepSeek Harness from 96,280 to 11,620 while success rose from 71.43% to 85.71%, and on Codex from 66,720 to 47,020 (29.54% fewer) with success unchanged at 90.82%. Paid model calls on DeepSeek Harness fell from 954 to 415.

Read the headline number with the paper's own caveat: the authors say the integrated harness results also include context management and finalization changes, so the 87.93% saving is not decision reuse alone. Their isolated test, on 257 MBPP tasks, measured persistent decision reuse at 13,170 tokens per task against 18,030 for single-use decision graphs, 26.91% less. These are coding benchmarks with short tasks, not chat. The transferable idea is the guard: a cached decision is only safe while the condition that justified it still holds, which is the same invalidation problem a memory layer has with stale facts. The cheap first step needs no paper: log each model call's chosen action and count how often it repeats one the agent already chose under the same conditions.

Lever 4: trim or compact the history

A sliding window keeps the last K exchanges and drops the rest. It is free and caps per-turn input, and it forgets everything outside the window, including the user's name if they gave it at turn 2. Keep a pinned block for facts that must survive (identity, standing constraints, open tasks), or the window will drop them.

Compaction replaces older turns with a summary. Claude now does this server-side, either on demand (beta header compact-2026-09-04), with an option to keep the most recent turns word for word, or automatically when input tokens reach a threshold you set; both are in beta and neither needs summarisation code of your own. A summary costs a model call to produce and pays for itself quickly once the history is long, because every later turn reads the summary instead of the transcript. For a full comparison of when summarising a session is cheap and when exact lines win, see context compaction vs retrieval.

What compaction risks is detail. A summary that turns "takes 20mg of Atorvastatin daily" into "takes a cholesterol medication" has saved tokens and lost the fact that mattered. Whatever compacts your history should tell you what it kept. Maximem Synap's compaction, which we build, returns a validation score and a count of preserved facts on every pass, runs when a conversation passes 3,000 tokens, 10 messages or 5 minutes idle, targets 1,500 tokens, and keeps the last three exchanges verbatim.

Lever 5: bound the context with a memory layer

A memory layer changes the unit that is resent. Instead of replaying the transcript, it extracts what is worth keeping (facts, preferences, decisions, open tasks) as the conversation happens, stores them outside the prompt, and returns a ranked block of the relevant ones within a token budget on each turn. Per-turn input stops growing with the history: system prompt, a memory block (2,000 tokens by default in Maximem Synap), the last few exchanges verbatim and the new message.

That is not free, and the comparison is only honest if the memory layer's own cost is counted. The table below does that. It uses the same conversation as above, split into sessions of 10 turns with the user returning after the cache has expired, on Sonnet 5.5. The bounded column includes Maximem Synap's published usage cost for storing each exchange and fetching context every turn (3 credits a turn at $0.00175 a credit).

The table cuts both ways. Below about 130 turns, caching alone is cheaper, and a memory layer is not a cost decision there; if you adopt one at that length, adopt it for what it remembers across sessions, not for the bill. From about 130 turns, when the history passes 65,000 tokens, the bounded context overtakes cached history whose cache goes cold between sessions, and at 200 turns it costs 77% of the cold-cache figure. Against a cache that never goes cold, Sonnet 5.5's cheap cache reads hold out until about 475 turns, and at 500 turns the bounded context is only 4% cheaper. The gap keeps widening past that point, because one column grows with the square of the turns and the other grows linearly, but on Sonnet 5.5 the honest summary is that the bill alone justifies a memory layer for returning users with long histories, not for long single sittings.

The crossover moves with the model. On Sonnet 5, whose cache reads cost $0.20, the bounded context beats the cold-cache case from about 110 turns and a cache that never expires from about 240. On Claude Opus 5.5 ($4 input, $0.20 cache read) it beats the cold-cache case from about 100 turns ($5.63 against $8.67 at 200 turns) and a never-expiring cache from about 370 ($14.09 against $17.40 at 500). On Claude Haiku 5.5 the memory layer never wins on cost: at 100 turns the cold-cache conversation costs $0.08 and the bounded one $0.58, almost all of it the memory layer's own fee, because the model is cheaper per turn than the memory call. For every provider's cached-input rate in one table, see the LLM API pricing comparison.

An independent cost study, arXiv 2603.04814, reached a consistent shape on different assumptions: at a 100,000-token context a fact-based memory system became cheaper than long-context prompting after about ten turns, with the break-even arriving sooner as context grows. The same study, with GPT-5-mini answering and a three-vote GPT-5-mini judge, found long-context prompting recalled more facts on two of its three benchmarks (LongMemEval and LoCoMo), which is the quality trade to test on your own conversations before you move a workload.

Which technique for which workload

How to measure it

Measure per conversation, not per request. For every turn, log input_tokens, cache_creation_input_tokens, cache_read_input_tokens and output_tokens from the response, keyed by conversation and session. From that you get the numbers that matter: cost per conversation by length bucket, cache read share per turn (a sudden drop means something is invalidating the cache, and cache diagnostics will tell you what), and the input tokens carried at the median and the 95th percentile turn. For agents, add model calls per task, because that is where repeated decisions show up. The distribution matters more than the average, because the longest conversations are where the quadratic term lives. Use the token counting endpoint with the model ID you will run, since the tokenizer differs between model generations.

Set a context budget explicitly, meaning the number of tokens a turn may carry beyond the system prompt. Without one, the budget is whatever the history happens to be, and that number only grows.

What this means for pricing your own product

If your product charges a flat fee per user, your heaviest users cost you the most, and on unmanaged context they cost you disproportionately more, since a user with five times the conversation length can cost up to roughly twenty-five times as much to serve. Either bound the context so cost per turn stays flat, or price in a way that acknowledges length, through usage tiers or credits. Doing neither means your best-retained customers are your least profitable ones. If you need US-only processing, budget for it: on Claude 4.6 and later models, inference_geo: "us" multiplies every token category, cache reads and writes included, by 1.1. For every Claude rate in one place, see Claude API pricing in 2026.

Frequently asked questions

How do I reduce LLM token costs in long conversations?

Turn on prompt caching first; on Claude Sonnet 5.5 it cuts a 40-turn conversation from $1.09 to $0.24 in our example. Then cap output, keep the effort setting fixed for the conversation, clear old tool results in large batches, compact or trim old turns once a single session runs long, and bound the context with a memory layer for assistants that users return to across sessions, where caching stops helping once the cache expires between visits.

Why do chatbot conversations get more expensive as they go on?

Because the API is stateless, each request resends the system prompt and every previous turn, so input per turn grows with the conversation and total input grows with the square of the number of turns. A 40-turn chat with a 2,000-token system prompt and 500 tokens per exchange sends 476,000 input tokens.

Does prompt caching solve the cost of long conversations?

Within a sitting, mostly. It bills the repeated history at a tenth of the input rate on most Claude models and a twentieth on Opus 5.5 and Sonnet 5.5. It does not shrink the history, it has to rewrite the whole history when a user returns after the cache expires or when a setting such as effort changes, and it does nothing for the context window limit or for answer quality on very long inputs.

Is it cheaper to drop old tool results or to summarise them?

It depends on where the dropping happens. In arXiv 2610.28577, dropping all but the latest 12 tool responses from the KV cache, inside the authors' own serving engine, cut average cost by 27.7% and raised average score by 1.7 points against threshold-triggered summarisation, across three open models and four agent benchmarks. The same paper found that trimming those tool responses from the text every turn cost 1.85 times as much as keeping the full history, because each trim forced the retained context to be processed again. On the Claude API only the text-level version exists, through context editing, so clear in large batches with clear_at_least rather than every turn.

Is summarising the conversation cheaper than sending the full history?

Once the history is long, yes, because every later turn reads the summary instead of the transcript. The risk is losing specific details, so check what each summary kept, and clear raw tool output in large batches instead of summarising it.

When is a memory layer cheaper than prompt caching?

From about 130 turns spread over sessions, in our arithmetic on Claude Sonnet 5.5, when the history passes roughly 65,000 tokens and the cache goes cold between visits, including the memory layer's own cost. Against a cache that never expires, not until about 475 turns. On Claude Haiku 5.5 the memory layer costs more than the model and does not pay for itself on cost alone.

Does reducing context make responses faster?

Every input token has to be processed before the first output token, so a smaller prompt starts answering sooner. Caching reduces cost more than it reduces the amount the model reads, while trimming, compaction and a bounded context reduce both.

Method

Every cost figure comes from one synthetic conversation: a 2,000-token system prompt, 150-token user messages and 350-token replies, priced at Anthropic's list rates on 31 October 2026 (Claude Sonnet 5.5: $2 input, $10 output, $2.50 for a 5-minute cache write and $0.10 for a cache read per million tokens; Claude Sonnet 5: $2, $10, $2.50 and $0.20; Claude Opus 5.5: $4, $20, $5 and $0.20; Claude Haiku 5.5, prompts up to 100,000 tokens: $0.10, $0.50, $0.125 and $0.01). "Cache never expires" assumes automatic caching where each turn writes the new exchange and reads the prefix. "Cache cold at each session" assumes sessions of 10 turns with the cache expired at the start of each, so the first request of a session writes the full prefix at the write rate. The bounded context sends the system prompt (cached within a session), a 2,000-token memory block, the last three exchanges and the new message at the full input rate, plus Maximem Synap's published usage cost of 3 credits a turn (2 credits to store a 500-token exchange and 1 for a fast context fetch) at the list rate of $0.00175 a credit; plan credits cost less than that. Crossover points are the first turn count at which the bounded total falls below the cached total. Output cost is included everywhere. Real conversations vary in length and shape, and whether a 2,000-token memory block holds enough for a given product is a quality question this arithmetic does not answer. Figures from the two October papers are the authors' own, on their benchmarks and models, and were not reproduced by us.

---

Sources, retrieved 31 October 2026: Anthropic pricing and models overview; Claude Platform documentation on context windows, prompt caching, cache diagnostics, compaction, context editing and token counting; Fu et al., Cache Engineering for Agents, October 2026; Jiang, Xu, Gao and Zhao, HarnessMM, October 2026; Pollertlam and Kornsuwannawit, Beyond the Context Window, March 2026. Maximem Synap figures come from its published documentation and pricing.