Agent Skills Do Not Save Tokens
The strongest head-to-head comparison published this month has the skill arm scoring 4.8 percentage points higher than workflow memory and spending 95,300 more tokens per task to get there. Both figures come from the 83 tasks that every arm attempted, and the token counts are averages per task rather than totals.
That is the trade in its cleanest form. Demystifying agent skills ran three arms over the same benchmark set with token accounting attached, and the arm that scored highest is not the arm that cost least. That same table cuts the other way too: against the raw baseline with no skill at all, the skill arm scored 5.5 percentage points higher and spent 34,200 fewer tokens per task, on the same 83 tasks, again as per-task averages. So a skill is not uniformly more expensive than no skill. What it is not is free. The progressive disclosure argument, which says a skill pays for itself because the agent only ever loads the part it needs, does not survive a matched comparison against a cheaper technique.
Running overhead shows up again in a different lab with a different method. The SkillCorpus authors report 158K tokens per task with skills on against 122K with them off, roughly 30% more, on the 87-task Opus 4.7 frontier check only. That is one run on one harness, and the figure is estimated from released eval-cache counters rather than billed, so read the shape and not the value: turning skills on made each task cost about a third more to run. Their own counter-example sits a page earlier. One task that the no-skill run failed was rescued by a skill and finished in less real time and fewer tokens than the failing run had already burned. Both things are true. Skills add overhead per task on average, and occasionally remove a failure loop that was costing far more than the overhead.
The extra cost is extra work, not a longer file
Most arguments about skill cost treat it as a context-length problem. That is why most of them reach the wrong answer.
Agent Skills Can Be Harmful looked at paired runs where the skill run and the reference run both passed the automated check, but the skill run cost substantially more. Across those runs, 62.6% of the cases came from the agent doing more procedure than the task needed. Skills turn optional validation checklists and construction recipes into mandatory work: the agent runs the verification pass the skill describes, then the construction pipeline it lays out, on a task that needed neither. Excessive verification, meaning checking work nobody asked to have checked, accounts for 36.8% of the same set and sits nested inside that 62.6%, so the two figures never add together.
Where the cost does come from context, the split is lopsided and useful. Of the 46 context-overhead cases, 43 came from the skill body itself and 3 from supplementary files the agent pulls in only when needed. The body always loads, so it is the expensive part, and lazy-loaded references are close to free. That is an instruction for whoever writes the skill rather than whoever runs it, and it is the cheapest fix in this entire literature: shorten the part that always loads.
Filtering shaped that dataset, and the shape matters. Candidate skills were kept only above a cosine similarity threshold to the task, which measures how closely the skill text matches the task text, and the agent was free to decline any skill it judged irrelevant. These are regressions caused by skills that were topically right for the job.
None of the figures above price the library itself. Building a curated skill set means judge calls, deduplication passes and revision cycles, and maintaining one means repeating a slice of that work every time a tool or an application programming interface (API) changes underneath it. The Demystifying authors state plainly that they did not measure the cost of maintaining or updating a skill library over time, and nobody else in this corpus did either. Every published cost comparison is therefore a running-cost comparison with the build cost set to zero. The nearest thing to a price anyone prints sits in the SkillCorpus accounting: one curation pass over a 96,401-skill corpus runs to roughly 269,000 judge calls, and that line item never makes it into a per-task cost comparison.
Your token counter can rank two options in the wrong order
A measurement warning, rather than a finding about skills, is the most useful result in the cost literature.
Skill Blocks compared four ways of getting the same skill content into context while accounting for the prompt cache, where content the model has already seen is served back at a lower price than fresh input. In multi-turn runs, meaning runs where the agent goes back and forth over many exchanges, cache reads come to 74% to 94% of raw input. The cache attribution model behind that range is unverified, and the authors concede it, which matters because every headline number in the paper rests on it. On single-turn benchmarks, where the agent gets one exchange, raw input remains the primary metric and none of this applies.
What the cache does to a comparison is not a rounding error. In one multi-turn arm the raw counts read 70,528 tokens against 65,380, and yet the arm with more raw input had effective input 14.8% lower, effective input being what you actually pay for once cache reads are priced at their cheaper rate. A team that ranks its loading strategies on raw token counters, which is what almost every dashboard shows by default, will pick the wrong one and have a number to justify it with. The author's instruction is to record raw, cache-read, new and output tokens at turn level before making any cost claim at all, and it is the correct instruction.
No loading strategy wins on every benchmark
Across five benchmarks the paper finds no configuration that wins everywhere, and the map of which setup suits which situation is the deliverable. On a small single-turn skill, a hybrid arrangement of short stubs with an optional fetch, meaning a one-line summary sits in context and the full text arrives only if the agent asks, came in 27.4% below full inlining. Pure on-demand tool loading of the same skill ran 48.4% above inlining, costing more than pasting the skill in, because the round trip and the tool schema outweigh the text avoided; that arm used a deliberately aggressive loading prompt the authors call an overhead diagnostic rather than a recommended configuration. On a large compressible skill in a multi-turn setting, conditional block loading reached a 62.47% reduction in cache-adjusted effective tokens, at 62 of 84 tasks succeeding against 57 of 84 for full inlining, with no outcome comparison reaching statistical significance. Between those poles the effects shrink toward parity, and several confidence intervals, the ranges the true effect probably sits in, include zero.
Then there is the arm that beat everything on cost. On the large single-turn benchmark, hybrid loading came in 39.8% below full inlining while static deletion of unused sections came in 55.5% below it. Static deletion permanently strips content out of the skill and registers no loader at all. It was cheaper than hybrid on 236 of 276 cases, and the authors attach two labels to it: non-content-parity, and exploratory. Both labels change what the result can support. Non-content-parity means it deleted coverage rather than deferring it, so it can say nothing about how the agent behaves once the task mix shifts toward the deleted material. Exploratory is the authors' own label for the entire primary outcome family, because the final cases were reused during development, and no outcome comparison in that family reached statistical significance. A non-significant outcome test is not evidence of equivalence.
Read at its true strength, the result is still the most interesting thing in the paper. Deleting content beat routing to content, on the benchmark where routing had the most to gain, which is exactly what you would expect if the overhead of conditional loading is comparable to the mass it defers.
How to measure this on your own workload
One instrumentation change and an afternoon will settle it for a given stack.
Pick fifty real tasks and run them twice, once with the skill available and once without, holding the model, the harness and the repository state fixed. Record raw input, cache reads, new input and output at turn level rather than at run level, because the run-level counter is the one that inverts. Then divide by completed tasks rather than attempted ones. That single step decides the answer: a skill that adds overhead to every task and removes a retry loop often enough has paid for itself, while one that adds the same overhead and changes nothing else has not. Look separately at the runs where the skill arm cost more and the automated check still passed, because that is where excessive procedure hides, and it will show up as verification and exploration steps the task never required.
What actually saves tokens
Loading fewer things saves tokens. Loading the same things more cleverly mostly does not.
One very large saving in this corpus comes from not putting a catalogue in context at all. Holding the whole catalogue costs 46,915 tokens per task against roughly 560 for fetching one skill on demand, priced over an 875-entry catalogue, and the authors never tested whether the fetched skill was enough to finish the job, so that comparison counts catalogue rows rather than completed tasks. A retrieval step that fails to surface the needed skill costs a failed task, and nothing in that arithmetic prices a failed task. Their own summary of where the saving comes from is the sentence to keep, because it generalises past their setup: it comes "from loading less rather than from loading smarter". The retrieval side of that trade is where the risk sits, and it is measured badly almost everywhere.
So the honest position on cost is that skills move it rather than remove it. The prompt gets shorter and the procedure gets longer. The catalogue leaves context and a retrieval failure mode arrives. The per-task running bill rises by about a third in one frontier check, and falls on the individual tasks where a skill turns a failure loop into a completion. None of that adds up to a savings argument, and the unit that reflects both halves is effective input per completed task rather than tokens per run.
Saving tokens was never the good reason to use a skill anyway. What skills are demonstrably good for is anchoring a procedure the model already knows how to execute, and that case stands on its own without a token argument propping it up.
So the cheap-alternative argument does not survive a matched comparison. A skill is not free, and against a store built from the agent's own past runs it is the more expensive way to buy the same points. What that means if you are choosing between the two is the skills question.
Every figure here is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, where the full claim register, including the eight claims that did not survive verification, is published in the open. We build Synap, which is a memory layer for agents, so weigh the argument accordingly and check the register.