Getting Retrieval Right Does Not Save You
The skill that breaks a task is usually the right skill for that task.
Agent Skills Can Be Harmful, one of the thirty-five papers behind The Agent Skills Evidence Base, audited that claim by pairing every suspect run against a reference run on the identical task, verifier, repository state, agent harness and model, varying only which skill was loaded. After filtering candidates to skills semantically matched to the task at cosine 0.7 or above, and letting the agent decline the ones it judged irrelevant, 68.8% of the failures that remained, 86 of 125 cases, were on-topic skills corrupting the implementation. Off-topic skills applied where they did not belong are not what this is about.
That result is stranger than it first reads, and it is why this sits apart from the rest of that corpus. The working model most teams carry is that a skill misfires because the wrong one got picked up, which makes the fix a matcher problem: a tighter description, a better embedding, a reranker in front of the library, a rewritten frontmatter. Here the wrong-skill case was suppressed by construction, and the failures did not go away. They changed shape. Whether retrieval finds the right skill in the first place is a real problem with its own evidence, and it is a separate piece. Everything below begins after retrieval has already done its job.
What the number can and cannot carry
68.8% is a share of confirmed skill-induced failures, not a share of tasks. The failures were harvested from a deliberately expanded comparison space assembled by scraping public skill hubs and pairing plausible matches against each task, so nothing in the paper says how often a typical user's skill damages a typical task, and the authors do not claim otherwise. The reference run is described as a pseudo-oracle rather than ground truth, which means it establishes that the task was solvable under identical conditions and not that the skill was the sole cause of the divergence. One harness, one frontier model, no repeated runs per configuration.
There is a sharper version of the same limit. Both of the mechanisms that produce a genuinely off-topic match in the wild, a matcher that surfaces something unrelated and an agent that accepts something it should have refused, were suppressed by the pipeline before the failures were counted, so the study has nothing to say about how often that case occurs and should not be read as saying it is rare. The conditional form is the whole claim: given a skill that is on topic and accepted, this is what goes wrong.
What survives that is the composition of the failures rather than their frequency, and composition is the part that decides where the budget goes. A library of failures dominated by mismatch tells you to invest in selection. A library of failures dominated by on-topic skills mis-implementing the task tells you that selection is already spending money in the wrong place.
Reusable guidance read as a hard requirement
The dominant mechanism is over-trust. An agent handed a topically matched skill treats that skill's reusable defaults, worked examples, templates and optional workflows as task-specific obligations, and then satisfies the skill instead of the task. Incorrect required-element fill accounts for 46 cases and required-element omission for 36, both counted against all 125 functional failures rather than against the 86, and together they are most of what the category contains.
One of the paper's own examples is a spreadsheet task asking for net exports as a percentage of GDP. The audited run computed exports minus imports over GDP, exactly the expression the skill supplied, and never converted the ratio into a percentage; the reference run did. The skill was correct as a piece of general guidance. It was wrong as an instruction for that task, and the agent had no way to tell which of the two it was holding, because a SKILL.md file does not distinguish between the parts that must be obeyed and the parts that are illustration.
Authoring is where the authors put the remedy: "Skill authors should separate mandatory task requirements from examples, defaults, reusable templates, and optional workflows, so that agents do not mistake reusable guidance for task-specific obligations." That is a real fix and a partial one, since it depends on every author in a shared library holding the same discipline, which is a heavy assumption anywhere the library accepts contributions from more than a handful of people.
A failure mode that the no-skill agent cannot have
Loading a skill opens a surface that did not previously exist. In the contrastive taxonomy built by Demystifying Agent Skills, skill guidance misapplied or ignored accounts for 10.0% of skill-arm cases against 0.8% in the raw arm, and the honest note is that this category is definitionally unavailable to an agent with no skill loaded, so the comparison is not a like-for-like regression.
Nobody should read that delta as the finding. What matters is that the category has any mass at all. As the authors put it, "A skill is not self-executing: the agent must decide whether it applies, which parts to follow, how to adapt it, and when to abandon it." Four judgement calls sit between a correctly retrieved skill and a correct outcome, and every one of them is made by the same model that would otherwise have been reasoning from the task alone. Adding a skill does not remove decisions from the agent. It replaces one set of decisions with another set that the evaluation harness usually does not measure.
The same behaviour, priced
Skills that do not break a task can still make it substantially more expensive, and the cost does not come from where most people assume. Excessive procedure accounts for 62.6% of efficiency regressions, with excessive verification nested inside it at 36.8%, so the two never add: the agent turns an optional validation checklist into mandatory work and runs it every time. Prompt length is the smaller story, and the one most teams instrument first because it is the one their dashboards already show.
Where cost does come from context, it is almost entirely the always-loaded body of the skill file, with 43 of the 46 context-overhead cases attributable to the mandatory body against 3 for lazy-loaded supplementary material. Progressive disclosure is doing its job. The part nobody discloses progressively, because it is the part the agent reads on every relevant turn, is where the bill accumulates.
Libraries that evolve without a gate
None of this is stable over time, which is the part that turns a nuisance into an operational risk. When the SkillCommit authors benchmarked four published skill-evolution methods, 31 of 72 task-level results fell below the no-skill baseline and another six merely matched it. Their instance-level decomposition shows why: these methods repair genuine failures and simultaneously overturn predictions that were already correct, because a lesson extracted from one trajectory gets generalised to a scope no evidence supports, which is precisely the move that makes a library feel as though it is learning while the aggregate number sits still. That accounting belongs to SkillCommit, whose authors propose the alternative and whose figures these are rather than each competing method's own reporting, so weigh it accordingly.
Their alternative treats library construction as a sequence of behaviourally validated scope expansions. A candidate skill widens its claimed applicability only if the frozen executor can replay every instance already inside that scope; where replay succeeds on part of the scope the skill is narrowed and revalidated, and where it fails the candidate is rejected and kept as an instance-specific note. Versions are preserved for traceability rather than overwritten.
Replay-gated expansion repairs 24 and 21 cases against 0 and 2 regressions on 100 and 45 test cases respectively, on one model and one benchmark family, and a method engineered around a regression gate producing almost no regressions is partly a design guarantee rather than a discovery. The result to take is not the margin. It is that a gate exists at all, that it is cheap to state, and that four published methods shipped without one and regressed on more than a third of the task-level results they were measured on.
Running the audit on your own library
The method here transfers more cheaply than the finding does, and it needs no new tooling. Pick twenty tasks your agent already passes reliably. Run each one twice under identical conditions, once with the relevant skill loaded and once with nothing loaded, holding the verifier, the repository state and the model constant so the skill is the only variable. Count two things: tasks that pass without the skill and fail with it, and tasks that pass both ways while the skill arm costs materially more in tokens or wall-clock time.
Open the first category by hand. The question that decides what you do next is whether the skill was on topic, and it almost always will be. When it was, read the trajectory for the specific move where a default, a path convention or an example value from the skill was treated as a requirement of the task, because that is the repair, and it is an authoring repair rather than a retrieval one. When the second category is large, check whether the cost sits in the mandatory body of the file or in material the agent chose to load, since only one of those is fixable by moving text into a reference.
The discipline this actually needs
A skill library is a shared code library whose consumers cannot read the type signatures. Every entry claims a scope, no entry is tested against the scope it claims, an agent has no way to distinguish a default from a requirement, and every widening of scope is applied to every future task in the organisation. Handled the same way, a Python package would be considered unmaintained.
An operational version of that is short, and none of it is exotic. Every skill declares its applicability condition and its exclusion condition rather than only its topic. Every widening of that condition is validated by replay against the cases the skill already covers, not by an embedding similarity or a model's judgement that two things look related. Mandatory requirements are marked as mandatory and separated from the examples, and the examples are lazy-loaded so they are not paid for on every turn. Nothing enters the library on the strength of one successful trajectory, and nothing stays in it without evidence that it is still contributing, which is a harder problem than it sounds and one the governance literature is still arguing about.
What none of this is, is a retrieval problem. The failures documented above happened with the correct skill in context, offered under the normal loading interface, on a task it genuinely matched. Better matching gets you to the starting line faster, and a library where every entry is precisely described will still produce these failures at close to the same rate, because the mechanism that produces them is triggered by the skill being relevant rather than by its being found. Matching has nothing to say about what happens next, and what happens next is where the tasks break. The case for using skills at all remains strong and rests on a different mechanism entirely, which is covered in What Skills Are Actually Good For.
Every figure here is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, with the full claim register published in the open. We build Synap, a memory layer for agents, so read the argument with that in mind and check the register.