89.3% of Public Skill Files Break a Specification Rule. The Most Common Break Is Also the Costly One.

A study read 138,133 public SKILL.md files without running any of them, checked each file against the official Agent Skills specification, and found that 89.3% of them break at least one rule the specification states. That is a conformance rate. It is not a verdict on whether any of those files work, and the distinction belongs to the authors rather than to me. They recommend "reading defect counts as static-analysis findings rather than as direct evidence of task-time failure". They also lead with the specification-conformance number instead of their looser all-category number, expressly "so that the headline is not read as '92% of skills are broken.'" The looser number puts a duplicated heading in the same total as a live application programming interface (API) key committed to a public repository, which is why they refused to lead with it.

What the study establishes is narrower than the headline. Most public skill files ignore the packaging rules the format defines. One of those rules has a measurable consequence, and that consequence is about whether an agent ever loads your skill, not about whether it does the right thing once loaded. Every figure below sits with its caveat in The Agent Skills Evidence Base.

How they counted

Their pipeline starts with public SKILL.md files deduplicated by content hash, so an identical copy of a file counts once. They read the specification clause by clause and turned each clause into a structural check or a text-pattern check. Then they split the checks into two tiers: rules the specification states, and recommendations that are best practice rather than requirement. The 89.3% figure counts only the first tier.

They hand-coded a stratified sample to build the taxonomy of violations. Each detector was validated against paired fixtures, meaning one file deliberately carrying the defect and one with it fixed. They cross-checked the categories against GitHub issues filed on platform repositories, then swept the detector thresholds from lenient to strict. Across lenient, baseline and strict settings the headline runs 88.8% to 94.6% on a 10,000-file sample, with only the five numeric detectors varied. That range bounds how much the thresholds are doing and says nothing about how the detectors were designed. Routing, body and resource rules stay the top three categories at every setting.

No agent runs anywhere in that pipeline, apart from two small side experiments. That is the shape of the evidence, and it limits what the number can carry.

The one violation with a cost attached

52.3% of the 138,133 files carry no trigger guidance at all, which is to say they never state when the agent should use them. That makes it the most common single violation, and the one tied to a measured routing consequence. Routing as a category reaches 67.0% of files, ahead of name-as-heading duplication at 44.3%, which the authors call cosmetic themselves, and too many inline examples at 32.1%.

A description is not documentation in this format. It is the string the model matches a task against when it decides whether to load the skill at all. A description that says what the skill does without saying when to use it hands the router a topic and withholds the condition.

That measurement is deliberately conservative. The authors indexed the description line from each file's frontmatter, the small metadata block at the top, and ran BM25 over those descriptions, which is a retriever that matches on shared words and has no access to meaning. Skills with clean routing metadata, meaning the name and description the retriever matches against, came back at 88.5% Hit@1, which is to say the right skill was the top result 88.5% of the time, against 82.6% for skills with defective metadata. The authors frame that gap as a lower bound rather than as a routing measurement. They then argue against their own result: production harnesses select with a language model reading the full set of descriptions, the absolute retrieval numbers should not be read as production routing performance, and the gap may narrow under semantic selection. So the honest reading is directional. A missing trigger condition costs you retrieval under a weak router, probably costs you less under a strong one, and nobody has measured how much less.

Two of the findings partly measure their own definitions

Violation density rises with file size. Spearman's rho, a rank correlation where 0 means no relationship and 1 means a perfect one, comes in at 0.508. Files over 500 lines carry 4.76 detected defects on average, against 1.48 for files under 50 lines. That relationship is partly definitional, since several detectors fire on size by construction. A body longer than the recommended length is itself a violation, and so is a large block of inline examples that should have been a reference file. Count those checks, correlate the total against length, and part of what you measured is the taxonomy correlating with itself. What survives independently is a cost claim rather than a conformance claim, and it carries its own measurement.

A second finding has the same problem in a sharper form. Skills whose descriptions follow the "[Verb] [what]. Use when [trigger]" shape carry 1.83 detected defects on average, against 3.00 for descriptions written without that shape. Cliff's delta, an effect size measuring how far the two groups sit apart, comes in at −0.40. The gap is partly definitional, since the most common single violation in the collection is the absence of that very pattern. A skill written in the specified shape cannot trigger the most prevalent check, so part of the distance between 1.83 and 3.00 is the measurement finding its own definition. The remainder is still interesting, because routing rules are not the only category that separates the two groups, but read the effect as an upper bound on what following the pattern buys you.

What the authors say against their own number

"Defect-free status is not the goal." That sentence is theirs, and they follow it with a severity ordering in which routing and safety violations matter, body bloat matters less, and a repeated heading wastes tokens without breaking anything.

They also concede that legitimate deviations get labelled as violations. A skill written as narrative rather than as directives registers as nonconforming, and so does a body longer than the recommended ceiling that stays actionable throughout, while both may work well in practice. Two of their safety and persona detectors misfire, one on skills that audit credentials and one on skills that deliberately teach an agent a role. Those per-category rates are therefore upper bounds, and the manual precision audit that would settle it was not run. Most of the categories have no end-to-end task benchmark behind them at all.

Their own sharpest objection follows: "a syntactically valid skill can still be generic, stale, unsafe, or tied to a local environment". Conformance is a floor. The exemplar analysis in the same paper makes the point from the other side. When they pulled the zero-violation skills out of heavily starred repositories, the one non-tautological trait those skills shared was that they encoded project-specific local procedure rather than tutorial content a model could already generate. That is a claim about content, and no automated checker measures it. What skills are actually good for is a separate argument with separate evidence.

One more limit is worth keeping in view. This is a snapshot of a collection that spreads by folder-copying rather than through a registry, so a single widely reproduced template can push a violation rate around, and nobody has measured how concentrated that copying is.

How to write one that follows the rules

Four things carry most of the weight, and each traces to something specific rather than to taste.

Say what the skill does and when to use it

Write the description in the third person, because the model reads it as part of its own context and a first or second person voice can degrade matching. State what the skill does and the conditions under which it should be used, and put the concrete terms a real request would contain into the text. The authoring note that sets this out uses a pair of examples that make the point better than a rule does. "Handles releases" gives the model nothing to match against. A description saying the skill drafts release notes from the pull requests merged between two version tags, and then naming cutting a release, updating a changelog and summarising what changed as the triggers, gives the router operations and conditions it can actually match.

That note carries no experiments whatsoever. It is a conceptual paper drawing on vendor documentation, so treat it as a well-organised statement of the specification rather than as evidence. The study above is what supplies the evidence that the advice is not being followed.

Anything that must happen every time belongs in a hook

Anything that must happen every time is not a skill. The decision rule from the same note: "If a step depends on judgement, varies with context, or encodes domain procedure, put it in a skill. If a step must happen every time the triggering event occurs, put it in a hook."

Hooks have the one property skill prose does not. An instruction sitting in a skill body, however firmly worded, is something the model may read, defer, or skip; the same instruction expressed as a hook runs unconditionally. Loading a specification file at session start, running a validator before a commit, formatting after a file write, refusing a destructive command: all of those are lifecycle events with deterministic answers, and writing them as emphatic paragraphs inside a SKILL.md converts a guarantee into a suggestion. Capitalising the instruction does not help. Repeating it does not help either.

Keep the always-loaded body short and put the rest one link away

Body length is a real cost, and the evidence for that is separate from the size-triggered checks. In a manual failure analysis of skill-induced regressions, one team traced context overhead back to its origin and found that 43 of 46 context-overhead cases came from the always-loaded mandatory body, against 3 from supplementary material that only loads when something references it. Material sitting behind a reference file is close to free until something reads it. Material in the body is paid for on every activation, on every task, whether or not it was relevant.

Procedure is the other cost, and in that same analysis it is the larger of the two: 62.6% of efficiency regressions were excessive procedure, with excessive verification at 36.8% nested inside that 62.6% rather than added to it. What that describes is a skill turning an optional checklist into mandatory work, so the agent verifies things nobody asked it to verify, on every run, forever. Trim by asking which lines change what the agent does on the common path, and move everything else into a reference file, one level deep, without chaining onward to further files.

Write the tests before you write the skill

Run the model on representative tasks with no skill loaded and record where it fails. Turn those failures into a small set of cases with expected behaviours, measure the no-skill baseline, then write the minimum instructions needed to pass, and iterate against that. Two instances make the loop honest, one to draft and a fresh one that has only the skill loaded to test against, because the instance that helped you write it already knows what you meant.

Skipping this step is how a skill ends up encoding a gap that was never there.

What revising against a score is worth

Revising against a measured signal beats revising blind, and the size of the difference is worth knowing. SkillEval scores a SKILL.md document on learned quality directions without executing it, then uses those per-metric scores to steer a rewrite. Mean downstream pass rate went from 18.6% for the original skills to 48.1% under metric-guided revision, against 38.3% when the same rewrite ran with no metrics to guide it. So roughly two thirds of the improvement is what an ordinary language-model rewrite would have given you anyway, and the remaining third comes from knowing which part of the document was weak.

Behind that comparison sits a single benchmark, one executor configuration and a handful of runs per condition. The same paper shows that one skill's usefulness swings enormously from task to task, which is its own argument against trusting any single number here. Take the ordering rather than the magnitudes.

What I would want tested

Routing is the result that matters most here, and it has only been measured with word matching. Someone should rerun the stress test with a language-model selector reading full descriptions, on the same corpus, and publish the gap. If a strong selector recovers most of what a missing trigger condition costs, then the trigger rule is a hygiene preference and the practical advice above collapses to two items. If the gap survives semantic selection, then 52.3% of the public collection is carrying a routing weakness that no amount of body quality can compensate for, because a skill that never loads has no body.

Write them well, for the job they are demonstrably good at. Then keep the things that change somewhere with a write path, for the reasons set out in the skills question.

We build Synap, which is a memory layer for agents, and the register behind every figure above is published in the open so the arithmetic can be checked rather than trusted.