Skills Fix How an Agent Works, Not What It Knows

A skill steadies how an agent works. It almost never tells the agent something it did not already know. Researchers labelled the mechanism at work in each of 528 matched sets of runs, using a large language model to assign the labels. In 65.7% of those sets the skill was anchoring the procedure: which steps to run, in what order, what to check before moving on. In only 4.5% was it supplying a fact the model was missing. That split is the most useful thing published about skills this year, because it tells you which problems to point a skill at and, by subtraction, which ones a skill will leave exactly where they were.

Anchoring the procedure means the file decides which setup steps run, which sequence of tool calls gets followed, what check happens before the agent moves on, and which recurring pitfall it avoids. Supplying a fact means the file tells the agent something new about the domain. Almost nobody writes skills with the first purpose in mind; almost everybody writes them as documentation, which is the second. Demystifying Agent Skills, the paper that produced the 65.7% against 4.5% split, is one of thirty-five in the evidence base, and it is the one that spends its effort on what mechanism is operating rather than on whether the aggregate score moved.

Consequences for what you write are direct. A skill that explains what a Kubernetes namespace is lands in the 4.5%, competing against training data that covers the topic better than your file will. A skill that says to confirm the namespace exists before applying the manifest, and to read the rollout status rather than trusting the exit code of the apply command, is doing the work the other 65.7% describes. Facts about the domain are mostly redundant. Ordering, preconditions and the one check that catches the specific way this pipeline breaks are mostly not.

Which failures a skill fixes, and which it leaves alone

Three conditions ran across those same 528 matched sets. One agent had no skill at all. One had a raw pool of its own past run transcripts, which the paper calls workflow memory. One had a distilled skill file. Each cell below is that failure type's share of the labels assigned in that condition, and the middle column is what the skill is really being compared against.

Environment and infrastructure failures fall from 5.3% with no skill to 0.2% with a skill. Failures where the output came back in the wrong shape fall from 7.4% to 3.2%. In both, the correct behaviour is knowable in advance and the agent's only problem is remembering to do it. The other two rows barely move. Algorithmic logic errors go from 8.3% to 7.4%, and runs where the agent checked its work by reading the code instead of running it go from 12.5% to 11.7%. A skill does not repair a wrong algorithm, and it does not make an agent execute the thing it just wrote.

Read the middle column before crediting the skill file with the first two rows. An undistilled pool of past transcripts already takes environment failures to 1.7% and wrong-shape output to 3.8%, which is most of the distance. Where distillation earns its place is the third row: the raw pool sits at 11.0% against 8.3% with no skill at all, so it makes logic errors worse than doing nothing, while the skill brings them to 7.4%. Undistilled experience helps with setup and hurts with reasoning; distilling it stops the second thing.

A fifth failure category is the cost of admission, and it sits outside that table because only one condition can produce it. Skill guidance misapplied or ignored runs at 10.0% in the skill condition against 0.8% with no skill, and a condition with no skill has nothing to misapply, so this is a new failure surface rather than a fair comparison. An agent handed a procedure has to decide whether it applies, which parts to follow, how to adapt it and when to abandon it. Roughly one time in ten it gets that decision wrong in a way that shows up in the trace.

Counter-evidence belongs in the same breath as the headline. Against an agent given no skill at all, the skill condition scored 2.84 percentage points higher, with a 95% confidence interval, the range the true value probably sits in, running from minus 2.27 to plus 7.95. That range crosses zero, so this sample does not establish that a skill beats nothing. The comparison that does stay clear of zero is the skill against the raw pool of past transcripts, at 6.06 points, with a range of plus 0.76 to plus 11.36. Distillation is what the evidence supports; the raw pool is what it beats.

A second experiment in the same paper answers the objection that a skill is only a shorter prompt. Every condition ran 26 tasks five times over, 130 trials each. No skill scored 50.0%. A short plan compressed out of the task instructions scored 47.7%. A test-first template derived from past runs scored 59.2%, the raw pool of past runs 62.3%, and the skill 79.2%. Look at the second number. Compressing the instructions into a plan and handing that to the agent performs no better than handing it nothing, and slightly worse, so whatever the skill is doing at 79.2% is not brevity; the short plan is the shortest artifact in the set.

Use a skill when the output has to match a contract

Two findings from a different research group point the same way, and the second is the sharper one.

ContinualSkillBench ran agents through a hundred ordered subtasks per domain and let them write and revise their own skill files as they went. Averaged over three of the five domains on GPT-5.3-Codex, at one run per cell, the skill-maintaining condition scored 0.602 and plain in-context learning, where past examples simply sit in the prompt and nothing gets written to a file, scored 0.605. An independent baseline that resets everything scored 0.466. Do not read the gap between the first two numbers as a result. Read the authors' own conclusion, which is that explicit skill maintenance provides no consistent aggregate advantage over in-context learning.

Underneath that null sits a division of labour worth more than the aggregate. Explicit skills won on tasks with rigid output requirements, the ones where an exact-match check or a program decides the score. In-context learning won on the open-ended work graded against a rubric, in all three of those domains. The aggregate is a wash because two effects of opposite sign are being averaged. The practical instruction is to reach for a skill file when the task has a contract, and to leave the model alone when the task has a judgement call in it.

The gain depends on whether the procedure matches the task

Skills pay when the procedure they carry matches the structure of the task, not when they merely share its topic. SkillCorpus sorts tasks by how well its retriever matched a skill to them, and reports mean gains of 2.2, 6.2 and 25.1 points across bins of 26, 40 and 17 tasks. That sorting variable is the paper's own ranking model scoring its own confidence, which makes the top bin suggestive rather than settled. Pooled across the whole grid the corpus is worth 7.5 points plus or minus 2.3, over 72 main-grid runs, which puts the pooled gain 3.2 standard errors above zero. On the two benchmarks where the baseline was already high, the authors say per-task judge noise exceeds the effect size, so the pooled figure is the defensible one and the per-benchmark ones are not.

Matching the structure is not the same as matching the topic, and the distinction has teeth. The authors document a case where on-topic skills about building slide decks all failed on a task that put an Excel table into a slide, because the procedures they carried did not extend to that task's structure, and following one displaced an improvisation that would have succeeded. Topic overlap is what a retriever measures. Procedure overlap is what decides whether the skill helps, and nothing published scores it directly.

Frontier models have not made this redundant. Claude Opus 4.7 goes from 39.1% to 47.1% with the corpus attached, on a single run, one harness and 87 tasks, which is a real number with no repeats behind it. Taken with the bins above, that suggests the supply of well-matched procedures is the binding constraint rather than the capability of the model reading them. Where the match is thin the gain floors near zero instead of turning negative, which is the mild good news buried in the same table.

What actually changes in how the agent behaves

Behavioural evidence comes from the study that set out to test something else and failed to find it. Do Personalized Skills Help Coding Agents? mined skills from individual developers' own coding-agent histories and found almost nothing on task score. What it did find is a shift in behaviour. Runs that reported a successful validation step rose from 43.1% to 58.9% under personalized skills, while agent tokens rose from 442,096 to 597,120 across the same comparison.

Both halves of that sentence matter. The agent started checking its work far more often, and it spent roughly a third more tokens doing so, with tool calls, wall-clock time and patch churn all moving the same way. Nothing here is free. The same trade shows up in the mechanism study, where the skill buys 4.8 points over the pool-of-past-runs condition for 95,300 more tokens per task on an 83-task matched intersection, with token figures reported as per-task averages rather than totals.

Discipline is the product. A skill made the agent run the validation step, follow the schema, set up the environment the way it is supposed to be set up, and do all of it on the run where it would otherwise have been in a hurry. That is a behavioural guarantee, and it is worth paying tokens for in any workflow where the cost of an unvalidated output lands on someone downstream. It is not an intelligence gain, and the papers that looked hardest for one did not find it.

That sets the boundary cleanly. Skills make an agent do the thing the same way every time. They do not make it better at deciding what the thing should be. They do not carry what the agent has learned about a particular user or a particular history, which is a separate store with an opposite write policy, covered in Skills Versus Agent Memory. And they only pay when the procedure inside the file matches the structure of the work. Getting that file right is its own discipline, and most published ones do not manage it, which is the subject of how to write a SKILL.md.

Consistency has a second-order value that none of these papers measure and every operator will recognise. An agent that behaves the same way on every run is an agent whose failures are reproducible, and a reproducible failure is one you can fix once. Variance is what makes an agent expensive to own, and anchoring the procedure attacks variance directly even where it leaves the average score alone.

Same way every time is a smaller claim than the marketing around this format suggests, and a much better one than it sounds. Most production failures are not failures of reasoning. They are failures of an agent that knew perfectly well what to do and did something else on the fourth run.

The split worth carrying away is behaviour against knowledge. A skill fixes how the agent works, and nothing measured here says it holds what the agent knows about a person. That second job is where we think skills get misused, and we say so in the skills question.

---

Every figure here is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, with the full claim register published alongside them. We build Synap, which is a memory layer for agents, so read the argument above with that in mind and check the register.