The Agent Picked the Right Skill and Still Broke the Task
The skill that breaks a task is usually the right skill for that task.
Agent Skills Can Be Harmful is one of the thirty-five papers behind The Agent Skills Evidence Base, and its authors tested that claim by running every task twice. One run had a skill loaded; the other was identical in every other respect, with the same task, the same checker, the same repository state, the same agent software and the same model. Before counting anything they threw out every skill that did not closely match the task, keeping only those scoring 0.7 or above on cosine similarity, a standard measure of how close two pieces of text are in meaning. They also let the agent refuse any skill it judged irrelevant. Of the failures that survived both filters, 68.8%, or 86 of 125, came from a skill that was on topic and corrupted the implementation anyway.
That result is stranger than it first reads, and it is why this sits apart from the rest of the corpus. Most teams carry a working model where a skill misfires because the wrong one got picked up, which makes the fix a matching problem: a tighter description, a better embedding of what the file means, a reranker that reorders candidates, or a rewritten metadata block at the top of the file. Here the wrong-skill case was removed by construction, and the failures did not go away. They changed shape. Whether retrieval finds the right skill in the first place is a real problem with its own evidence, and it is a separate piece. Everything below begins after retrieval has already done its job.
What this number does and does not tell you
68.8% is a share of confirmed skill-induced failures, not a share of tasks. The failures came out of a deliberately widened comparison space, built by scraping public skill hubs and pairing plausible matches against each task. Nothing in the paper says how often a typical user's skill damages a typical task, and the authors do not claim otherwise. They call the second run a pseudo-oracle rather than ground truth, which means it proves the task was solvable under identical conditions and not that the skill was the only cause of the difference. One agent harness, meaning one piece of software running the agent, one frontier model, and no repeated runs per configuration.
There is a sharper version of the same limit. Two things produce a genuinely off-topic match in the wild: a matcher that surfaces something unrelated, and an agent that accepts something it should have refused. The pipeline suppressed both before any failure was counted. So the study has nothing to say about how often that case occurs, and it should not be read as saying the case is rare. The conditional form is the whole claim. Given a skill that is on topic and accepted, this is what goes wrong.
What survives is the composition of the failures rather than their frequency, and composition is what decides where the budget goes. A library of failures dominated by mismatch tells you to invest in selection. A library of failures dominated by on-topic skills mis-implementing the task tells you that selection is already spending money in the wrong place.
The agent reads an example as a rule
Over-trust is the dominant mechanism. An agent handed a topically matched skill treats that skill's reusable defaults, worked examples, templates and optional workflows as obligations for the task in front of it, and then satisfies the skill instead of the task. Two categories carry most of it. Filling a required element in incorrectly accounts for 46 of the 125 functional failures, and leaving a required element out accounts for 36 of the 125. Both counts run against all 125 functional failures rather than against the 86.
One of the paper's own examples is a spreadsheet task asking for net exports as a percentage of gross domestic product. The audited run computed exports minus imports divided by GDP, exactly the expression the skill supplied, and never converted that ratio into a percentage; the reference run did. The skill was correct as a piece of general guidance. It was wrong as an instruction for that task, and the agent had no way to tell which of the two it was holding, because a SKILL.md file, the markdown file that tells an agent how to do something, does not mark which parts must be obeyed and which parts are only illustration.
Authoring is where the authors put the remedy: "Skill authors should separate mandatory task requirements from examples, defaults, reusable templates, and optional workflows, so that agents do not mistake reusable guidance for task-specific obligations." That is a real fix and a partial one. It depends on every author in a shared library holding the same discipline, which is a heavy assumption anywhere the library accepts contributions from more than a handful of people.
A failure that an agent without a skill cannot have
Loading a skill opens a surface that did not previously exist. Demystifying Agent Skills built a side-by-side taxonomy of what went wrong with a skill loaded and without one. Skill guidance misapplied or ignored accounts for 10.0% of the labels in the runs with a skill, against 0.8% in the runs without. The honest note is that this category is definitionally unavailable to an agent with no skill loaded, so the comparison is not a like-for-like regression.
Nobody should read that gap as the finding. What matters is that the category has any mass at all. As the authors put it, "A skill is not self-executing: the agent must decide whether it applies, which parts to follow, how to adapt it, and when to abandon it." Four judgement calls sit between a correctly retrieved skill and a correct outcome, and the same model that would otherwise have reasoned from the task alone makes every one of them. Adding a skill does not remove decisions from the agent. It swaps one set of decisions for another set that the evaluation harness usually does not measure.
Skills that do not break a task can still make it cost more
Skills that leave a task passing can still make it substantially more expensive, and the cost does not come from where most people assume. Excessive procedure accounts for 62.6% of the efficiency regressions, meaning the runs where a skill made a task more expensive. Excessive verification sits nested inside that 62.6% at 36.8%, so the two never add together: the agent turns an optional validation checklist into mandatory work and runs it every time. Prompt length is the smaller story, and the one most teams instrument first because their dashboards already show it.
Where cost does come from context, almost all of it is the always-loaded body of the skill file. Of the 46 context-overhead cases, 43 trace to that mandatory body, against 3 for supplementary material the agent loads only when it decides it needs it. Progressive disclosure is doing its job. The part nobody discloses progressively, because the agent reads it on every relevant turn, is where the bill accumulates.
What happens when a library updates itself
None of this is stable over time, which is the part that turns a nuisance into an operational risk. The SkillCommit authors benchmarked four published methods for growing a skill library automatically. Across the four, 31 of 72 task-level results fell below the baseline of running with no skill at all, and another six merely matched it. Their per-instance breakdown shows why. These methods repair genuine failures and at the same time overturn predictions that were already correct, because a lesson extracted from one run gets generalised to a scope no evidence supports. That is precisely the move that makes a library feel as though it is learning while the aggregate number sits still. Those figures come from the SkillCommit authors, who propose the alternative, rather than from each competing method's own reporting, so weigh them accordingly.
Their alternative treats library construction as a sequence of scope expansions that each have to be validated by behaviour. A candidate skill widens its claimed applicability only if the agent, held unchanged, can successfully replay every case already inside that scope. Where replay succeeds on part of the scope, the skill is narrowed and revalidated. Where it fails, the candidate is rejected and kept as a note attached to that single case. Versions are preserved for traceability rather than overwritten.
Replay-gated expansion repairs 24 and 21 cases against 0 and 2 regressions, on test sets of 100 and 45 cases, using one model and one benchmark family. A method engineered around a regression gate producing almost no regressions is partly a design guarantee rather than a discovery. The result to take is not the margin. It is that a gate exists at all, that it is cheap to state, and that four published methods shipped without one and regressed on more than a third of the task-level results they were measured on.
How to run this test on your own library
Running the same audit on your own library needs no new tooling, and the method transfers more cheaply than the finding does. Pick twenty tasks your agent already passes reliably. Run each one twice under identical conditions, once with the relevant skill loaded and once with nothing loaded, holding the checker, the repository state and the model constant so the skill is the only variable. Count two things: tasks that pass without the skill and fail with it, and tasks that pass both ways while the skill run costs materially more in tokens or elapsed time.
Open the first category by hand. The question that decides what you do next is whether the skill was on topic, and it almost always will be. When it was, read the run log for the specific move where a default, a path convention or an example value from the skill was treated as a requirement of the task. That is the repair, and it is an authoring repair rather than a retrieval one. When the second category is large, check whether the cost sits in the mandatory body of the file or in material the agent chose to load, since only one of those is fixable by moving text into a separate reference file.
The rules a skill library needs
A skill library is a shared code library whose consumers cannot read the type signatures. Every entry claims a scope, and no entry is tested against the scope it claims. An agent has no way to distinguish a default from a requirement, and every widening of scope is applied to every future task in the organisation. Handled the same way, a Python package would be considered unmaintained.
An operational version of that is short, and none of it is exotic. Every skill declares the condition under which it applies and the condition under which it does not, rather than only its topic. Every widening of that condition is validated by replaying the cases the skill already covers, not by an embedding similarity score or a model's judgement that two things look related. Mandatory requirements are marked as mandatory and kept separate from the examples, and the examples load only on demand so they are not paid for on every turn. Nothing enters the library on the strength of one successful run, and nothing stays in it without evidence that it is still contributing, which is a harder problem than it sounds and one the governance literature is still arguing about.
What none of this is, is a retrieval problem. The failures documented above happened with the correct skill in context, offered under the normal loading interface, on a task it genuinely matched. Better matching gets you to the starting line faster. A library where every entry is precisely described will still produce these failures at close to the same rate, because the mechanism that produces them is triggered by the skill being relevant rather than by its being found. Matching has nothing to say about what happens next, and what happens next is where the tasks break. The case for using skills at all remains strong and rests on a different mechanism entirely, which is covered in What Skills Are Actually Good For.
None of this is an argument against skills. It is an argument that a skill is an instruction, and that instructions stay brittle in a way stored state does not. Where we think that line falls is the skills question.
Every figure here is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, with the full claim register published in the open. We build Synap, a memory layer for agents, so read the argument with that in mind and check the register.