# Nine in Ten Public Skills Break the Spec They Were Written Against A static-analysis census of the format found that 89.3% of 138,133 files violate at least one rule in the official Agent Skills specification. That is a conformance rate and not a verdict on whether any of those files work, and the distinction belongs to the authors rather than to me. They recommend "reading defect counts as static-analysis findings rather than as direct evidence of task-time failure", and they foreground the specification-conformance number instead of their looser all-category number expressly "so that the headline is not read as '92% of skills are broken.'" The looser number counts a duplicated heading in the same total as a live API key committed to a public repository, which is precisely why they refused to lead with it. Quoting that caution costs nothing and buys the rest of the piece its credibility, because the number is real and the inference most people will draw from it is not. What [the study](https://arxiv.org/abs/2608.08453) actually establishes is that the public corpus mostly ignores the packaging contract the format defines, that one part of that contract has a measurable consequence, and that the consequence is about whether an agent ever loads your skill rather than about whether it does the right thing once loaded. Every figure below sits with its caveat in [The Agent Skills Evidence Base](https://www.maximem.ai/research/agent-skills). ## What the census actually did Working over public SKILL.md files deduplicated by content hash, the authors read the specification clause by clause, turned each clause into a structural or regular-expression check, and split the checks into two tiers: rules the specification states, and recommendations that are best practice rather than requirement. The 89.3% figure counts only the first tier. They coded a stratified sample by hand to build the taxonomy, validated each detector against paired mutation fixtures, cross-checked the categories against GitHub issues filed on platform repositories, and swept the detector thresholds from lenient to strict. Across lenient, baseline and strict settings the headline runs 88.8% to 94.6% on a 10,000-file sample, with only the five numeric detectors varied, so the range bounds how much the thresholds are doing and says nothing about how the detectors were designed. Routing, body and resource rules stay the top three categories throughout. No agent runs anywhere in that pipeline except in two small side experiments; that is the shape of the evidence, and it constrains what the number can carry. ## The violation with a consequence attached 52.3% carry no trigger guidance at all, which makes it the most common single violation and the one tied to a measured routing consequence. Routing as a category reaches 67.0% prevalence, ahead of name-as-heading duplication at 44.3%, which the authors call cosmetic themselves, and too many inline examples at 32.1%. A description is not documentation in this format. It is the string the model matches a task against when deciding whether to load the skill at all, and a description that says what the skill does without saying when to use it hands the router a topic and withholds the condition. That measurement is deliberately conservative. The authors indexed frontmatter descriptions and ran a lexical BM25 retriever over them, and skills with clean routing metadata came back at 88.5% against 82.6% Hit@1 for defective ones, which they frame as a lower bound rather than as a routing measurement. They then argue against their own result: production harnesses select with a language model over the full description set, the absolute retrieval numbers should not be read as production routing performance, and the gap may narrow under semantic selection. So the honest reading is directional. A missing trigger condition costs you retrieval under a weak router and probably costs you less under a strong one, and nobody has measured how much less. ## Two of the findings are partly circular, and saying so is cheaper than being corrected Violation density rises with file size at Spearman ρ = 0.508, with mean detected defects running 4.76 for files over 500 lines against 1.48 for files under 50, and that relationship is partly definitional, since several detectors are size-triggered by construction. A body longer than the recommended length is itself a violation, and so is a large block of inline examples that should have been a reference file. Count those checks, correlate the total against length, and part of what you measured is the taxonomy correlating with itself. The part that survives independently is a cost claim rather than a conformance claim, and it carries its own measurement. The spec-aware description finding has the same problem in a sharper form. Skills whose descriptions follow the "\[Verb\] \[what\]. Use when \[trigger\]" shape carry 1.83 mean detected defects against 3.00 for spec-unaware ones, at Cliff's δ = −0.40, and the gap is partly definitional, since the most common single violation in the corpus is the absence of that very pattern. A skill written in the specified shape therefore cannot trigger the most prevalent check, so a portion of the distance between 1.83 and 3.00 is the measurement finding its own definition. The remainder is still interesting, since routing rules are not the only category that separates the two groups, but the effect size should be read as an upper bound on what following the pattern buys you. ## What the authors say against their own number "Defect-free status is not the goal." That sentence is theirs, and they follow it with a severity ordering in which routing and safety violations matter, body bloat matters less, and a repeated heading wastes tokens without breaking anything. They also concede that legitimate deviations get labelled as violations. A skill written as narrative rather than as directives, or a body longer than the recommended ceiling that stays actionable throughout, both register as nonconforming while possibly working well in practice. Two of their safety and persona detectors misfire on skills that audit credentials and on skills that deliberately teach an agent a role, so those per-category rates are upper bounds and the manual precision audit that would settle it was not run. Most of the categories have no end-to-end task benchmark behind them at all. Their own sharpest objection follows: "a syntactically valid skill can still be generic, stale, unsafe, or tied to a local environment". Conformance is a floor. The exemplar analysis in the same paper makes the point from the other side, because when they pulled the zero-violation skills out of heavily starred repositories, the one non-tautological trait those skills shared was that they encoded project-specific local procedure rather than tutorial content a model could already generate. That is a claim about content, and no linter measures it. What skills are actually good for is [a separate argument with separate evidence](https://www.maximem.ai/research/agent-skills/what-agent-skills-are-good-for). One more limit worth keeping in view: this is a snapshot of a corpus that propagates by folder-copying rather than through a registry, so a single widely reproduced template can push a violation rate around, and [nobody has measured how concentrated that copying is](https://www.maximem.ai/research/agent-skills/agent-skills-ecosystem-data). ## Writing one that conforms Four things carry most of the weight, and each traces to something specific rather than to taste. ### Write the description as an interface Third person, because the model reads it as part of its own context and a first or second person voice can degrade matching. State what the skill does and the conditions under which it should be used, and put the concrete terms a real request would contain into the text. [The authoring note](https://arxiv.org/abs/2607.25032) that sets this out uses a pair of examples that make the point better than a rule does: "Handles releases" gives the model nothing to match against, while a description that says it drafts release notes from the pull requests merged between two version tags and then names cutting a release, updating a changelog and summarising what changed as the triggers, gives the router operations and conditions it can actually match. That note carries no experiments whatsoever. It is a conceptual paper drawing on vendor documentation, so treat it as a well-organised statement of the specification rather than as evidence. The census is what supplies the evidence that the advice is not being followed. ### Put hard guarantees in hooks Anything that must happen every time is not a skill. The decision rule from the same note: "If a step depends on judgement, varies with context, or encodes domain procedure, put it in a skill. If a step must happen every time the triggering event occurs, put it in a hook." The reason is the one property skill prose does not have. "A standing instruction in a skill body, however firmly worded, is something the model may read, defer, or skip. The same instruction expressed as a hook runs unconditionally." Loading a specification file at session start, running a validator before a commit, formatting after a file write, refusing a destructive command: all of those are lifecycle events with deterministic answers, and writing them as emphatic paragraphs inside a SKILL.md converts a guarantee into a suggestion. Capitalising the instruction does not help. Repeating it does not help. ### Keep the mandatory body short and put the rest one link away Body length is a real cost and the evidence for that is separate from the size-triggered checks. In a manual failure analysis of skill-induced regressions, [one team](https://arxiv.org/abs/2608.11888) traced context overhead to its origin and found 43 of 46 context-overhead cases came from the always-loaded mandatory body against 3 from lazy-loaded supplementary material. The material sitting behind a reference is close to free until something reads it; the material in the body is paid for on every activation, on every task, whether or not it was relevant. Procedure is the other cost, and it is larger than context in that same analysis: 62.6% of efficiency regressions were excessive procedure, with excessive verification at 36.8% nested inside it rather than added to it. What that describes is a skill turning an optional checklist into mandatory work, so the agent verifies things nobody asked it to verify, on every run, forever. Trim by asking which lines change what the agent does on the common path, and move everything else into a reference file, one level deep, without chaining onward to further files. ### Write the evaluations before the skill Run the model on representative tasks with no skill loaded and record where it fails. Turn those failures into a small set of cases with expected behaviours, measure the no-skill baseline, then write the minimum instructions needed to pass, and iterate against that. Two instances make the loop honest, one to draft and a fresh one that has only the skill loaded to test against, because the instance that helped you write it already knows what you meant. Skipping this step is how a skill ends up encoding a gap that was never there. ## Revision, measured Revising against a measured signal beats revising blind, and the size of the difference is worth knowing. [SkillEval](https://arxiv.org/abs/2608.06891) scores a SKILL.md document on learned quality directions without executing it, then uses those per-metric scores to steer a rewrite. Mean downstream pass rate went from 18.6% to 48.1% under metric-guided revision, against 38.3% for unguided revision, which puts roughly two thirds of the improvement inside what an ordinary language-model rewrite would have given you anyway and the remaining third attributable to knowing which part of the document was weak. Behind that comparison sits a single benchmark, one executor configuration and a handful of runs per condition, and the same paper shows that a single skill's usefulness swings enormously from task to task, which is its own argument against trusting any one number here. Take the ordering rather than the magnitudes. ## What I would want tested The routing result is the load-bearing one, and it has only been measured lexically. Someone should rerun the stress test with a language-model selector over full descriptions, on the same corpus, and publish the gap. If a strong selector recovers most of what a missing trigger condition costs, then the trigger rule is a hygiene preference and the practical advice above collapses to two items. If the gap survives semantic selection, then 52.3% of the public corpus is carrying a routing weakness that no amount of body quality can compensate for, because a skill that never loads has no body. We build [Synap](https://www.maximem.ai/), which is a memory layer for agents, and the register behind every figure above is published in the open so the arithmetic can be checked rather than trusted. --- Source: [https://www.maximem.ai/research/agent-skills/how-to-write-a-skill-md](https://www.maximem.ai/research/agent-skills/how-to-write-a-skill-md)