# The Agent Skills Evidence Base

## How this was built

Roughly fifty papers on the SKILL.md artifact reached arXiv between the 25th of July and the 18th of August 2026. We read thirty-five end to end, pulled 299 claims into a register, then re-derived the 195 load-bearing ones from the source PDFs across five adversarial passes asking what correction a hostile reader would force. Ninety-four came back publishable, ninety-three publishable only with a caveat welded on, and eight did not survive. Twenty-nine of the thirty-five were read in the first pass. The other six had been excluded by a selection rule that preferred empirical and negative-result studies over method papers; we went back for them on the 20th of August, and two of the six falsified claims that were already written into a finished draft. Every figure below carries its caveat in the same sentence, because most of these results are single-run, small-sample or measured on one harness.

## The standing claim

Skills are a procedural technology. They stabilise how an agent acts rather than supplying what it did not know: [Demystifying Agent Skills](https://arxiv.org/abs/2608.14036) labels 65.7% of skill mechanisms procedural anchoring against 4.5% knowledge injection, over 528 paired triples. Capability gain is smaller than the framing suggests, since skill against raw is +2.84 points, 95% CI \[−2.27, +7.95\], crossing zero, and only skill against workflow memory excludes zero, at +6.06, CI \[+0.76, +11.36\]. Skills cannot carry episodic state: an agent-curated store scores 37.5% against 78.1% on PersonaMem 32k, a 32-question single run whose authors call small-tier gaps suggestive rather than settled. And the binding constraint on a mature library is retrieval rather than authoring: execution-time actual-use precision falls from 29.6% to 3.3% as pools grow from 5 to 100 skills, while downstream success does not follow, moving only 36.4% to 39.3%. Governance is where the evidence moved most recently, and it moved against the reassuring reading. [SkillTV-Bench](https://arxiv.org/abs/2608.05573) measures seven judges at false-pass rates of 0.455 to 0.795 on 156 truly-failed real agent trajectories, every one of them at or past the 0.45 threshold that disables skill retirement, with nothing injected, on an evaluation set made deliberately hard by screening out tasks the agent always passed or always failed, at a single run per judge and no error bars.

## The nine questions

### [Can skills replace agent memory?](https://www.maximem.ai/research/agent-skills/skills-vs-agent-memory)

No, and the failure does not present as a retrieval failure. In [Filesystem-Based Memory](https://arxiv.org/abs/2607.26637), all thirteen questions that the verbatim dump answered and curation missed had the gold-critical fact present in the curated store, at grounding of 0.94 to 0.98 and attribution of 0.97 to 1.0, so the material was found, cited, and answered wrongly anyway.

### [Why do agent skills fail?](https://www.maximem.ai/research/agent-skills/why-agent-skills-fail)

Because a correctly chosen skill can still corrupt the implementation. Even after filtering to skills semantically matched at cosine 0.7 or above and letting the agent decline the ones it judged irrelevant, [Agent Skills Can Be Harmful](https://arxiv.org/abs/2608.11888) found 68.8% of surviving failures, 86 of 125, were on-topic skills damaging the work.

### [How many skills can an agent have?](https://www.maximem.ai/research/agent-skills/skill-retrieval-at-scale)

Confusability sets the ceiling rather than count. Top-1 precision at 100 skills runs 53.4% on a semantically similar pool against 84.1% random and 93.2% dissimilar, an offline embedding diagnostic never passed through to execution.

### [Does a self-evolving skill library keep improving?](https://www.maximem.ai/research/agent-skills/self-evolving-skill-libraries)

Only while its judge stays inside a reliability cliff that [Ratchet](https://arxiv.org/abs/2605.22148) puts at (1−τ)/2, which is 0.45 at an eviction margin of 0.10, past which nothing is retirable at any sample size, because more trials shrink the estimator's radius and not the bias. The one real judge Ratchet audited came in at roughly 0.01, clearing the cliff by a factor of 45; seven judges measured on 156 truly-failed real agent trajectories run 0.455 to 0.795 at a single run per judge with no error bars, so the distance between one audited grader and seven measured ones is the open question rather than a settled verdict on either.

### [What are skills actually good for?](https://www.maximem.ai/research/agent-skills/what-agent-skills-are-good-for)

Environment and format discipline. Environment and infrastructure failures drop 5.3% to 0.2% from the raw arm to the skill arm and format and schema mismatches 7.4% to 3.2%, while algorithmic logic errors move 8.3% to 7.4%, essentially unchanged.

### [How many agent skills exist?](https://www.maximem.ai/research/agent-skills/agent-skills-ecosystem-data)

3.8 million SKILL.md files (3,797,117) matched by filename across 282,200 public repositories in July 2026, nine months after the format appeared, a count [GitSkills](https://arxiv.org/abs/2608.10906) deliberately calls a lower bound, having retained files that merely contain the term. Of the files collected, 50.5% are verbatim copies, byte-identical only and therefore a floor.

### [How should a SKILL.md be written?](https://www.maximem.ai/research/agent-skills/how-to-write-a-skill-md)

Against the published specification, which 89.3% of 138,133 public files violate on at least one rule. Of those, 52.3% carry no trigger guidance at all, the most common violation and the one tied to a measured routing consequence.

### [Are agent skills safe?](https://www.maximem.ai/research/agent-skills/agent-skill-security)

Static analysis catches part of it; the composition surface stays open. One representative framework in [SkillsMetric](https://arxiv.org/abs/2608.08468) reaches 62.2% recall at its best operating point, at 89.4% precision and AUC 0.93; a cross-skill composition attack reaches 96.0% across six representative scanners given iterative query access to the scanners being scored, against 36.7% for naive splitting alone.

### [Do skills save tokens?](https://www.maximem.ai/research/agent-skills/agent-skill-token-cost)

Not on their own. Loading a whole catalogue costs 46,915 tokens per task against roughly 560 for on-demand retrieval, priced over an 875-entry catalogue whose sufficiency the authors never tested, so that comparison counts rows and not completed tasks; the skill arm buys 4.8 points over workflow memory for 95,300 more tokens per task on an 83-task matched intersection.

## The papers

Thirty-five papers, grouped by theme. Dates are 2026 arXiv stamps; a version marker gives the revision we read. Rows 30 to 35 joined the corpus on the 20th of August, after the first pass had closed.

| # | Paper | arXiv | Date | Theme | What it establishes |
| --- | --- | --- | --- | --- | --- |
| 1 | Demystifying Agent Skills: Why They Work Until They Do Not | [2608.14036](https://arxiv.org/abs/2608.14036) | 14 Aug | Mechanism and efficacy | What a skill changes inside a run, and where that stops |
| 2 | Skill Use or Skill Theater? | [2607.27484](https://arxiv.org/abs/2607.27484) | 29 Jul | Mechanism and efficacy | Traces cannot tell you whether a skill was causally used |
| 3 | From Context to Skills (Ctx2Skill) | [2604.27660](https://arxiv.org/abs/2604.27660) | v4 27 Jul | Mechanism and efficacy | Skills as a distillation layer sitting on top of context, never replacing it |
| 4 | ContinualSkillBench: Can LLM Agents Truly Evolve? | [2608.03874](https://arxiv.org/abs/2608.03874) | 4 Aug | Mechanism and efficacy | Skill maintenance against plain in-context learning, head to head |
| 5 | When and How Context Rot Appears in Coding Agents | [2607.17937](https://arxiv.org/abs/2607.17937) | v2 1 Aug | Context and loading | A fixed skill degrading as the context around it grows |
| 6 | Skill Blocks: How Should an Agent Load Its Skill? | [2608.14943](https://arxiv.org/abs/2608.14943) | 14 Aug | Context and loading | Cache-correct accounting inverts the ranking raw token counts give |
| 7 | @skills: Attention Is All You Have | [2608.12610](https://arxiv.org/abs/2608.12610) | 12 Aug | Context and loading | The attention-budget argument, and the procedural against declarative line |
| 8 | Filesystem-Based Memory for LLM Agents | [2607.26637](https://arxiv.org/abs/2607.26637) | 29 Jul | Context and loading | Puts skills and declarative memory in one store and measures what breaks |
| 9 | Comparative Approaches to Agent Retrieval over Large Skill Libraries | [2608.06196](https://arxiv.org/abs/2608.06196) | 6 Aug | Retrieval at scale | A real library, and the share of queries a good ranker still leaves unserved |
| 10 | SkillSight: Calibrating Generic Content Bias for Skill Retrieval | [2607.18785](https://arxiv.org/abs/2607.18785) | v3 14 Aug | Retrieval at scale | Why skill documents are harder to retrieve than ordinary text |
| 11 | Do Not Offer What Cannot Be Done (executability gating) | [2608.01050](https://arxiv.org/abs/2608.01050) | 2 Aug | Retrieval at scale | Relevant is not runnable, measured on live production traffic |
| 12 | GitSkills: A Dataset of Agent Skills on GitHub | [2608.10906](https://arxiv.org/abs/2608.10906) | 11 Aug | Ecosystem and authoring | The census of what has actually been published, and how much of it repeats |
| 13 | SkillCorpus | [2607.15557](https://arxiv.org/abs/2607.15557) | v5 6 Aug | Ecosystem and authoring | The strongest positive result on skills, and its boundary conditions |
| 14 | What Keeps Agent Skills from Being Reusable? (138K SKILL.md) | [2608.08453](https://arxiv.org/abs/2608.08453) | 9 Aug | Ecosystem and authoring | Conformance against the official specification, rule by rule |
| 15 | Authoring Agent Skills: A Software-Engineering Approach | [2607.25032](https://arxiv.org/abs/2607.25032) | 27 Jul | Ecosystem and authoring | The skill against hook decision rule, argued rather than measured |
| 16 | Library Drift | [2605.19576](https://arxiv.org/abs/2605.19576) | v3 29 Jul | Updating and governance | Names and measures the silent degradation mode |
| 17 | Ratchet: How Reliable Must an LLM Judge Be to Retire a Skill? | [2605.22148](https://arxiv.org/abs/2605.22148) | v3 7 Aug | Updating and governance | The judge-reliability cliff, derived and then audited against a real judge |
| 18 | The Blind Curator | [2607.07436](https://arxiv.org/abs/2607.07436) | v2 18 Aug | Updating and governance | A biased judge switches genuine retirement off while the counts look busy |
| 19 | Rethinking Self-Evolving Agent Skills: Feedback Dynamics | [2608.02636](https://arxiv.org/abs/2608.02636) | 31 Jul | Updating and governance | Evolution is sparse filtered search, not steady improvement |
| 20 | On the Fragility of Self-Improving Agents | [2608.18066](https://arxiv.org/abs/2608.18066) | 18 Aug | Updating and governance | Gains vanish under reshuffled task order, in textual memory banks |
| 21 | SkillCommit: Behaviorally Validated Scope Expansion | [2608.15165](https://arxiv.org/abs/2608.15165) | 15 Aug | Updating and governance | Replay-gated commits, the strongest fix currently on offer |
| 22 | Agent Skills Can Be Harmful | [2608.11888](https://arxiv.org/abs/2608.11888) | 12 Aug | Harm from ordinary skills | A taxonomy of confirmed skill-induced failures with no attacker present |
| 23 | Do Personalized Skills Help Coding Agents? | [2608.10319](https://arxiv.org/abs/2608.10319) | v2 15 Aug | Harm from ordinary skills | Personalisation distilled from history mostly loses to a pooled generic skill |
| 24 | SkillEval: Decomposing Agent Skill Quality | [2608.06891](https://arxiv.org/abs/2608.06891) | 7 Aug | Harm from ordinary skills | Execution-free skill scoring that predicts downstream uplift |
| 25 | Practice Makes Unsafe: Skill Misevolution | [2608.12851](https://arxiv.org/abs/2608.12851) | 13 Aug | Security and supply chain | Unsafe success becomes reusable, persistent policy |
| 26 | When Experience Becomes Instruction | [2608.05563](https://arxiv.org/abs/2608.05563) | v2 7 Aug | Security and supply chain | A minority of poisoned records writes attacker behaviour into a shared skill |
| 27 | SkillsMetric: Detection Boundary of Static Analysis | [2608.08468](https://arxiv.org/abs/2608.08468) | 9 Aug | Security and supply chain | Maps where scanning fails, deliberately and by category |
| 28 | ColluSkill: Cross-Skill Composition Evading Scanners | [2608.09732](https://arxiv.org/abs/2608.09732) | 10 Aug | Security and supply chain | Splitting a payload across skills defeats six representative scanners |
| 29 | Towards a Risk Assessment of Malicious Skill Files | [2608.05223](https://arxiv.org/abs/2608.05223) | 5 Aug | Security and supply chain | The demand side: how often an agent declares compliance, and how rarely it refuses |
| 30 | SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution | [2608.05573](https://arxiv.org/abs/2608.05573) | 6 Aug | Updating and governance | What a judge misses when it grades a whole agent trajectory instead of a final answer |
| 31 | Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills | [2607.12790](https://arxiv.org/abs/2607.12790) | v2 30 Jul | Updating and governance | An evaluator evolved alongside the skills, and what it turns into once its guards come off |
| 32 | Rethinking Self-Evolution: Mitigating Skill Overfitting (SkillBoost) | [2607.26643](https://arxiv.org/abs/2607.26643) | v2 19 Aug | Updating and governance | Skill evolution fitting its own trajectories, measured against held-out splits |
| 33 | VCE-Skill: Skill Self-Evolution with Version-Change Experience | [2608.16544](https://arxiv.org/abs/2608.16544) | 17 Aug | Updating and governance | Public version histories as evidence, and the first read on how published skills get revised |
| 34 | When Self-Evolution Backfires: Pre-Commit Gating against Skill Contamination | [2608.05810](https://arxiv.org/abs/2608.05810) | 6 Aug | Updating and governance | A defective skill teaching the skills distilled after it, and why deleting it does not undo that |
| 35 | From Relevance to Execution Utility: Reward-Aware Dynamic Execution Gating | [2608.09168](https://arxiv.org/abs/2608.09168) | 10 Aug | Retrieval at scale | Retrieval relevance tested as a predictor of whether running the bundle is worth the rollout |

## The open claims register

Every claim we extracted is published with its paper, page and verdict, including the ones we cannot use. Of the 299 entries, the 195 load-bearing enough to re-derive from source carry one: 94 publishable, 93 publishable only with the caveat attached, 8 not publishable at all. [Download the register](https://www.maximem.ai/research/agent-skills/claims-register.md).

## What we got wrong

Eight load-bearing claims failed verification, and every one had been written down as fact in our notes before anybody reopened the source PDF.

Our summary carried a ratio saying that picking a skill unsuited to the task accounts for a negligible share of skill-induced failures. That pipeline kept only candidates at cosine 0.7 or above to the task's curated skill, explicitly to filter out off-topic matches, and let the agent decline what it judged irrelevant, so both mechanisms that produce applicability mismatch in the wild were suppressed before the ratio was computed. It measures the filter rather than the world.

Progressive disclosure sat in our notes as the worst loading option, which the source table contradicts: it beats the no-skill baseline in all three model rows. Weakest of the skill-selection methods and most expensive in latency and tokens is the narrower claim we should have written.

A quotation about skill runners granting third-party instructions the same authority as direct user commands turned out to be two sentences spliced together, appearing in that form nowhere in the paper. We had also attributed a Cohen's κ to human validation of the 528 paired triples, when it measures one human against an LLM mapping 238 labels onto 12 modes. Neither would have survived a determined reader, which is why they are here.

A second round of failures came out of the six papers our own selection rule had excluded. That rule preferred empirical and negative-result studies over method papers, which holds up until a method paper measures something no empirical paper in the set had measured, and four claims in a finished draft on self-evolving libraries did not survive contact with two of them.

The draft said that no judge had been observed past the reliability cliff, and that every cliff number in this literature came from a bit-flip channel injected onto a grader that was already correct. Both are now wrong. SkillTV-Bench measures seven judges at false-pass rates of 0.455 to 0.795 on 156 truly-failed real agent trajectories with nothing injected anywhere, on a deliberately hard evaluation set, at a single run per judge and no error bars; [Who Grades the Grader](https://arxiv.org/abs/2607.12790) shows an evolving metric collapsing into an always-pass state on three of three seeds once two guards are removed, which the search arrives at on its own, in a paper its authors call a mechanism study rather than a scaling result. That same paper closes a third gap. Our draft called the coupled case, where a library gradually learns the phrasing that makes its own judge lenient, out of scope and unmeasured, and the coupled case sits in that paper, where evolved skills lifted the rubric score by 0.26 partly by writing evidence tags without values, about 30% of tags at peak rounds, with an independent outer judge catching it and one added detector repairing it. The fourth correction is a counting error of ours. We described the governance cluster as three papers sharing seven authors, and it is five, since Who Grades the Grader comes from the same team, adopts the same loop unchanged, and cites a fifth paper by the same authors.

None of the four reached publication. What the original exclusion cost was six unread papers and one draft rewritten before anybody outside the project saw it, which is cheaper than the alternative and still more expensive than reading them in the first pass.

## Changelog

We sweep arXiv monthly and rewrite the standing claim whenever the evidence moves. Each change lands here with a date.

| Date | Change |
| --- | --- |
| 20 August 2026 | First publication. Twenty-nine papers read in full, 299-claim register opened, nine spokes mapped. |
| 20 August 2026 | Corpus extended to thirty-five. Six papers the selection rule had excluded as method work were read in full; two of them falsified four claims in a finished draft on self-evolving libraries, and the judge-reliability line of the standing claim was rewritten around seven judges measured past the cliff. |

---

Source: [https://www.maximem.ai/research/agent-skills](https://www.maximem.ai/research/agent-skills)