What 35 Research Papers Say About Agent Skills

An Agent Skill is a markdown file that tells an AI agent how to do something. The agent decides on its own whether the file is relevant, loads it, and follows it. The format is called SKILL.md, it arrived in late 2025, and millions of these files now sit in public code repositories.

Between the 25th of July and the 18th of August 2026, about fifty research papers on that format appeared on arXiv, the open research archive. We read thirty-five of them and re-derived the numbers the argument rests on from the original papers. Five things came out of it, and most of them cut against how skills are usually sold.

The short version

A skill changes how an agent works, not what it knows. Researchers labelled the mechanism at work in 528 matched sets of runs, using a language model to assign the labels. Two thirds of the time, 65.7%, the skill was steadying the agent's process: which setup steps to run, which order to do things in, what to check. Only 4.5% of the time was it supplying a fact the model did not already have.

The gain in raw capability is smaller than the pitch suggests. Against an agent given no skill at all, adding one moved the success rate by 2.84 percentage points. The error range on that measurement runs from minus 2.27 to plus 7.95, and because that range crosses zero, the honest reading is that the true effect might be nothing. The one comparison that does hold up is a skill beating a store built from the agent's own past runs, by 6.06 points, with a range of plus 0.76 to plus 11.36 that stays above zero.

A skill cannot carry what a user told you last month. In one study, a store of tidy notes written by an agent answered 37.5% of questions correctly where a raw, untouched transcript of the same conversations answered 78.1%. That is 32 questions in a single run, and the authors call gaps of this size suggestive rather than settled. The interesting part is why it failed. On all thirteen questions the transcript got right and the notes got wrong, the fact needed was already sitting in the notes. What the tidying destroyed was the chronology, so the agent could no longer tell which of two statements about the same thing had replaced the other.

Once a library gets large, picking the right skill is harder than writing one. Grow the pool from 5 skills to 100 and how often the skill the agent actually used was the right one falls from 29.6% to 3.3%. Task success barely moves over the same range, from 36.4% to 39.3%, so this is a warning about your dashboards rather than proof that the work suffers.

If your library evolves on its own, it depends on a grader that is probably broken. A self-improving library adds and retires skills based on whether they seem to be working, which means a grader decides what stays. Seven graders were scored against 156 agent runs that genuinely failed, with nothing corrupted, on a test set built to be hard, at one run each and no error bars. Every one of them waved through between 45.5% and 79.5% of those failures as passes. Somewhere around 45%, depending on how aggressive your retirement rule is, the retirement logic stops removing anything at all, no matter how much data you feed it.

Some key conclusions

Nine questions, one answer each, every one linking to the page that shows the working. We also state our own position on skills and memory on a separate page, so this one can stay a summary of what the papers found.

What are skills actually good for?

Keeping an agent tidy about its environment and its output format. Environment and infrastructure failures fell from 5.3% to 0.2%, and failures where it produced the wrong output shape fell from 7.4% to 3.2%. Failures where it simply reasoned badly barely moved, from 8.3% to 7.4%.

Can skills replace agent memory?

No, and it will not look like a search problem when it breaks. In the study above, thirteen questions were answered by the raw transcript and missed by the tidy notes. In all thirteen, the needed fact was sitting in the tidy notes. The agent found it, cited it correctly, and still answered wrongly, because the notes no longer said which statement had replaced which.

Does a self-improving skill library keep improving?

Only while its grader still works. There is a threshold, and where it sits depends on how aggressive your retirement rule is: at a typical setting it lands near 45%. Once a grader is passing that share of genuine failures, the logic that retires bad skills fires on nothing, at any sample size, because more test runs shrink random error but do nothing to a bias. One carefully audited grader came in at about 1%, well clear. Seven graders scored against 156 real failed runs came in at 45.5% to 79.5%, on one run each with no error bars, so the gap between those two pictures is the open question.

How many skills can an agent have?

What matters is how similar your skills are to each other, not how many you have. With 100 skills to choose from, the agent picks correctly 53.4% of the time when they resemble each other, 84.1% when the pool is random, and 93.2% when they are clearly different. This was measured offline, on the matching step alone, and never carried through to a live run.

Do skills save tokens?

Not by themselves. Loading an entire catalogue of 875 entries costs 46,915 tokens per task against roughly 560 for fetching one skill on demand, but that figure counts catalogue entries rather than finished tasks, and nobody tested whether one fetched skill is enough, so it is not a like-for-like saving. Where a skill does earn its keep, it costs: against a store built from past runs, 4.8 percentage points of extra success for 95,300 extra tokens per task, measured across 83 tasks.

Why do agent skills fail?

Because the right skill can still ruin the job. Researchers filtered down to skills that genuinely matched the task, and let the agent refuse any it thought irrelevant. Of the failures that survived that filtering, 68.8%, or 86 out of 125, were on-topic skills that damaged the work anyway.

Are agent skills safe to install?

Scanning catches the obvious cases and misses the interesting ones. One scanner caught 62.2% of malicious skills at its best setting, while being right 89.4% of the time when it did flag something. An attack that splits a payload across several innocuous-looking skills got past six representative scanners 96.0% of the time, though only when the attacker could query those scanners repeatedly and adjust. Splitting alone, with no feedback, worked 36.7% of the time.

How many agent skills exist?

A July 2026 sweep of public repositories found 3,797,117 files named SKILL.md across 282,200 projects, nine months after the format appeared. The authors call that a floor rather than a count, since they kept anything with a matching filename. Half of what they collected, 50.5%, was byte-for-byte identical to something else, and that share is a floor too, since only exact matches were counted. There is no registry and no package manager, so skills spread by copying folders.

How should a SKILL.md be written?

Against the published specification, which 89.3% of 138,133 public files break in at least one place. The most common break, in 52.3% of files, is that the file never says when the agent should use it, which is the violation with a measured effect on whether the agent finds it.

The 35 papers

Grouped by theme. Dates are 2026 arXiv stamps, and a version marker shows which revision we read. Rows 30 to 35 joined later, after the first pass had closed.

Questions people ask

Is this saying skills are a bad idea?

No. Skills do a specific job well, and the evidence for that job is solid. They keep an agent consistent about setup, sequence and output format. The argument here is that three other jobs get handed to them, and they are not built for those.

Should I stop using skills for memory?

If the thing you are storing never changes, a file is fine. If it can be updated, corrected or reversed, the research says a summarised file will eventually answer with the old version and sound confident doing it, because summarising throws away the order in which things were said. Keep a record of what happened, in order, and let the agent read it.

How many skills is too many?

Nobody has run the experiment that would answer this properly, and we say so. What the evidence does show is that similarity hurts more than volume. Skills that overlap each other are the problem, and the count is not.

Are these papers peer reviewed?

Mostly not. This is a three-week-old literature, largely preprints, and a lot of it is single-run with small samples. Every figure on this page carries its caveat in the same sentence for exactly that reason, and we would rather you discount a number than take it on trust.

You sell a memory product. Why should I believe this?

You should not take our word for it. Every number here comes from someone else's paper, and every claim is published with its source and page number so you can check it. We also list the eight claims we got wrong and pulled, below.

Every claim, with its source

We publish all 299 claims we pulled, with the paper and page each came from, including the ones we could not use. Of those, 195 mattered enough to re-derive from the original documents: 94 held up, 93 held up only with a specific caveat attached, and 8 did not survive. Download the register.

What we got wrong

Eight claims failed checking. Every one of them had been written down as fact in our notes before anyone reopened the source paper.

Our summary said that picking a skill unsuited to the task causes almost none of the failures. The study behind it had already thrown out every skill that did not closely match the task, and let the agent refuse the rest. Both of the things that cause a wrong-skill failure in real life had been removed before the ratio was calculated. It measures the filter, not the world.

We had written down that loading skills on demand was the worst option available. The source table says the opposite: it beats using no skills at all in all three model rows tested. The narrower claim, which is what we should have written, is that it is the weakest of the skill-selection methods and the most expensive in time and tokens.

A quotation about skill files being given the same authority as a direct user instruction turned out to be two separate sentences spliced together. It appears in that form nowhere in the paper. We had also credited a reliability score to human checking of 528 runs, when it actually measures one person against a language model sorting 238 labels into 12 buckets. Neither would have survived a determined reader.

A second round of failures came from six papers our own selection rule had thrown out. That rule preferred studies reporting real-world results over papers proposing a new method, which works until a method paper measures something nobody else measured. Four claims in a finished draft did not survive contact with two of them.

The draft said no grader had ever been caught passing failures at a dangerous rate, and that every number of that kind came from researchers deliberately corrupting a working grader. Both are wrong. Seven graders were measured passing between 45.5% and 79.5% of genuine failures, with nothing corrupted, on a deliberately hard test set at one run each. A separate paper watched a grader collapse into passing everything, on three runs out of three, once two safety checks were removed, and the collapse happened on its own rather than being induced. The same paper closes a third gap: we had called it unmeasured when a library gradually learns the phrasing that makes its own grader lenient, and that paper measures exactly that, with evolved skills lifting a rubric score partly by writing evidence tags with no values behind them, about 30% of tags at the peak, caught by a separate outside judge and repaired by one added detector. Its authors call it a mechanism study rather than a scaling result. The fourth correction is a counting error of ours. We described a group of papers as three sharing seven authors, and it is five.

None of the four reached publication. The cost of the original exclusion was six unread papers and one draft rewritten before anyone outside the project saw it. Cheaper than the alternative, and still more expensive than reading them the first time.

How we did this

We searched arXiv for papers on the SKILL.md format posted between the 25th of July and the 18th of August 2026, which returned about fifty. We read thirty-five end to end and pulled 299 claims into a register. The 195 that carried real weight were then re-derived from the original documents across five separate checking passes, each one asking what correction a hostile reader would force. Ninety-four came back clean, ninety-three needed a caveat attached, and eight were dropped.

Most of these results are single-run, small-sample, or measured on one setup. Every figure above carries that context in the same sentence, because a number from this literature quoted bare will almost always overstate what was found.

Changelog

We sweep arXiv monthly and update this page when the evidence moves. Each change lands here with a date.