What Skills Are Actually Good For

Skills work by stabilising action rather than by supplying missing facts, and the split is not close: 65.7% procedural anchoring against 4.5% knowledge injection, from LLM-assigned mechanism labels over 528 paired triples. That single ratio is the most useful thing published about skills this year, because it tells you which problems to point them at and, by subtraction, which ones they will leave exactly where they were.

Procedural anchoring means the skill file decides which setup steps run, which tool sequence gets followed, what intermediate check happens before the agent moves on, and which recurring pitfall it avoids. Knowledge injection means the file tells the agent something it did not know. Almost nobody writes skills with the first purpose in mind; almost everybody writes them as documentation, which is the second. Demystifying Agent Skills, which produced that ratio, is one of thirty-five papers in the evidence base, and it is the one that spends its effort on the question of what mechanism is operating rather than on whether the aggregate score moved.

Consequences for what you write are direct. A skill that explains what a Kubernetes namespace is lands in the 4.5%, and it is competing against training data that covers the topic better than the file will. A skill that says to confirm the namespace exists before applying the manifest, and to read the rollout status rather than trusting the exit code of the apply command, is doing the work the larger share describes. Facts about the domain are mostly redundant; ordering, preconditions, and the one check that catches the specific way this pipeline breaks are mostly not.

The delta table

Sorting failures by mode across matched runs shows a clean separation between what skills repair and what they do not touch, on shares of arm labels across the same 528 paired triples, where the middle column is what the skill arm is actually being compared against.

Environment and infrastructure failures fall from 5.3% to 0.2% between the raw arm and the skill arm, and output format and schema mismatches from 7.4% to 3.2%, which is a large reduction in the two categories where the correct behaviour is knowable in advance and the agent's only problem is remembering to do it. Algorithmic logic errors move 8.3% to 7.4% and static verification without runtime moves 12.5% to 11.7%, both essentially unchanged. A skill does not repair a wrong algorithm, and it does not make an agent run the thing it just wrote instead of reasoning about whether the code looks correct.

Read the middle column before crediting the skill file with the first two rows, because an undistilled pool of prior trajectories already takes environment failures to 1.7% and format mismatches to 3.8%, which is most of the distance. Where distillation earns its place is the third row, where the pool sits at 11.0% against the raw arm's 8.3% and makes logic errors worse than doing nothing, while the skill arm brings them to 7.4%. Undistilled experience helps with setup and hurts with reasoning; the distilled version stops doing the second thing.

A fifth category is the cost of admission, and it sits outside that table because only one arm can produce it. Skill guidance misapplied or ignored runs at 10.0% in the skill arm against 0.8% in the raw arm, and that category is definitionally unavailable to an arm with no skill to misapply, so it is a new failure surface rather than a fair comparison. An agent handed a procedure has to decide whether it applies, which parts to follow, how to adapt it, and when to abandon it, and one time in ten it gets that decision wrong in a way that shows up in the trace.

Counter-evidence belongs in the same breath as the headline. The skill arm beat the no-experience baseline by +2.84 points with a 95% CI of [−2.27, +7.95], and that interval crosses zero, so the paper does not establish a skill-over-nothing gain in this sample. The comparison that does clear zero is skill against an undistilled pool of prior trajectories, at +6.06 with a CI of [+0.76, +11.36]. Distillation is what the evidence supports; the raw pool is what it beats.

An ablation in the same paper answers the objection that a skill is only a shorter prompt. Over 26 tasks at five trials each, 130 trials per arm, the raw arm scores 50.0%, an instruction-derived short plan 47.7%, a workflow-derived test-first template 59.2%, workflow memory 62.3% and the skill 79.2%. Look at the second number. Compressing the task instructions into a plan and handing that to the agent performs no better than handing it nothing, and marginally worse, so whatever the skill is doing at 79.2% is not brevity, since the short plan is the shortest artifact in the set.

The task-type split

Two findings from a different research group point the same way, and the second is the sharper one.

ContinualSkillBench ran agents through a hundred ordered subtasks per domain, letting them write and revise their own skill files as they went. Averaged over three domains on GPT-5.3-Codex, the skill-maintaining condition scored 0.602, plain in-context learning scored 0.605, and an independent baseline that resets everything scored 0.466, on one run per cell. Do not read the gap between the first two numbers as a result; read the authors' own conclusion, which is that explicit skill maintenance provides no consistent aggregate advantage over in-context learning.

Underneath that null sits a division of labour that is worth more than the aggregate. Explicit skills won on tasks with rigid output requirements, where an exact-match or programmatic verifier decides the score. In-context learning won on the open-ended, rubric-scored work, in all three domains they tested. So the aggregate is a wash because two effects of opposite sign are being averaged, and the practical instruction is to reach for a skill file when the task has a contract and to leave the model alone when the task has a judgement call in it.

Coverage decides the size of the gain

Skills pay when the procedure they carry matches the structure of the task, and not when they merely share its topic. SkillCorpus bins tasks by how well its retriever matched a skill and reports mean gains of +2.2, +6.2 and +25.1 points across bins of n = 26, 40 and 17 tasks, where "coverage" is the paper's own reranker scoring its own confidence, which makes the top bin suggestive rather than settled. Pooled across the whole grid the corpus is worth +7.5 ± 2.3 points, z = 3.2, over 72 main-grid runs, and on the two high-baseline benchmarks the authors say per-task judge noise exceeds the effect size, so the pooled figure is the defensible one and the per-benchmark ones are not.

Coverage is a structural match rather than a topical one, and the distinction has teeth. The authors document a case where on-topic skills about building slide decks all failed on a task that put an Excel table into a slide, because the procedures they carried did not extend to that task's structure, and following one displaced an improvisation that would have succeeded. Topic overlap is what a retriever measures. Procedure overlap is what decides whether the injection helps, and nothing published scores it directly.

Frontier models have not made this redundant. Claude Opus 4.7 goes from 39.1% to 47.1% with the corpus attached, on a single run, one harness and 87 tasks, which is a real number with no repeats behind it. What that suggests, taken with the coverage bins, is that the supply of well-matched procedures is the binding constraint rather than the capability of the model reading them. Where coverage is thin the gain floors near zero instead of turning negative, which is the mild good news buried in the same table.

What actually changes in the agent's behaviour

Behavioural evidence comes from the study that set out to test something else and failed to find it. Do Personalized Skills Help Coding Agents? mined skills from individual developers' own coding-agent histories and found almost nothing on task score. What it did find is a behavioural shift: runs reporting successful validation rose from 43.1% to 58.9% under personalized skills, against agent tokens rising from 442,096 to 597,120 over the same comparison.

Both halves of that sentence matter. The agent started checking its work far more often, and it spent roughly a third more tokens doing so, with tool calls, wall-clock time and patch churn all moving in the same direction. Nothing here is free, and the same trade shows up in the mechanism study, where the skill arm buys 4.8 points over the trajectory-pool arm for 95,300 more tokens per task on an 83-task matched intersection, with token figures reported as per-task averages rather than totals.

Discipline is the product. A skill made the agent run the validation step, follow the schema, set up the environment the way it is supposed to be set up, and do all of it on the run where it would otherwise have been in a hurry. That is a behavioural guarantee, and it is worth paying tokens for in any workflow where the cost of an unvalidated output lands on someone downstream. It is not an intelligence gain, and the papers that looked hardest for one did not find it.

That sets the boundary cleanly. Skills make an agent do the thing the same way every time. They do not make it better at deciding what the thing should be, they do not carry what the agent has learned about a particular user or a particular history, which is a separate substrate with an opposite write policy and covered in Skills Versus Agent Memory, and they only pay at all when the procedure inside the file matches the structure of the work. Getting that file right is its own discipline, and most published ones do not, which is the subject of how to write a SKILL.md.

Consistency has a second-order value that none of these papers measure and every operator will recognise. An agent that behaves the same way on every run is an agent whose failures are reproducible, and a reproducible failure is one you can fix once. Variance is what makes an agent expensive to own, and procedural anchoring is a direct attack on variance even where it leaves the average score alone.

Same way every time is a smaller claim than the marketing around this format suggests, and it is a much better one than it sounds. Most production failures are not failures of reasoning. They are failures of an agent that knew perfectly well what to do and did something else on the fourth run.

---

Every figure here is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, with the full claim register published alongside them. We build Synap, which is a memory layer for agents, so read the argument above with that in mind and check the register.