THE SKILLS QUESTION
We do not need a memory layer. We have skills.
Three versions of the same instinct. Write what you learn about the user into a SKILL.md, because the agent already reads those files and every change shows in a diff. Let the agent write and retire its own skills, so the system improves without you. Or skip the memory layer entirely, because a folder of files costs nothing to run. All three are right that agents should take instruction from files, and right that procedure belongs in version control. All three are asking a file that holds instructions to hold state instead. Here is where each one breaks.
About fifty research papers on the SKILL.md format went up on arXiv between the 25th of July and the 18th of August 2026. We read thirty-five of them and published every claim with its source, including the eight we got wrong. The clearest result in the set is that a skill changes how an agent works and almost never changes what it knows. Researchers labelled the mechanism at work in 528 matched sets of runs, using a language model to assign the labels: 65.7% of the time the skill was steadying the procedure, and 4.5% of the time it was supplying a fact the model was missing. That is a real job, done well. It is not the memory job.
THE WRITE-IT-DOWN QUESTION
Skills hold facts that stay true. They fail on facts that get replaced.
One study built the same memory three ways and put the same questions to each: an untouched transcript, whole sessions moved into folders, and a store of tidy notes written by an agent. On the benchmark constructed around preferences that change, where the right answer is whatever the user most recently said, the tidy notes scored 37.5% and the untouched transcript scored 78.1%. That comparison rests on 32 questions in a single run, and the authors call gaps of a few questions on their small tiers suggestive rather than settled. On the benchmark that asks mostly about facts which were true when said and stayed true, the tidy notes tie for best at 86.1%. Four benchmarks were tested and the collapse happens on exactly one of them. Something uniformly worse would be easy to dismiss and easy to fix. Something that ties for best on one benchmark and loses forty points on another is telling you what it cannot represent.
The reason it fails is the part that should worry you. On all thirteen questions the transcript answered and the notes missed, the fact the answer depended on was present in the notes. Across those store designs, grounding scored 0.94 to 0.98 and attribution 0.97 to 1.0, both out of a maximum of 1.0, which is to say the search agent found relevant material and cited material that genuinely supported what it said. The answer was wrong anyway. The paper traces three ways this happens. The one doing the work on this benchmark is that summarising threw away the order things were said in, so nothing in the file could say which statement had replaced which. This does not look like a retrieval problem in your logs. It looks like your agent being confidently out of date.
The strongest objection comes from the same paper, and we would rather you heard it here. Rebuild the store with a more capable management agent and the score recovers from 37.5% to 56.3%, which the authors read as possibly a limitation of the backbone model rather than of the curated form itself. A sign test on that recovery returns p = 0.073, so it is directional rather than decisive, and 56.3% still sits well below 78.1%. A better curator narrows the gap. Nothing published so far closes it. The full comparison.
THE SELF-EVOLVING QUESTION
If the library maintains itself, something decides which skills are working. That something is a grader, and the honest position is that you cannot tell how good yours is without measuring it.
There is a threshold, derived rather than observed. Once a grader passes some share of genuine failures as passes, the logic that retires bad skills fires on nothing, at any sample size, because more test runs shrink random error and do nothing to a bias. Where the threshold sits depends on how aggressive your retirement rule is. At the one margin the paper works through, it lands at 45%. No published default has ever been calibrated against a measured grader, so treat 45% as an example rather than as your number. Staying under the threshold is necessary rather than sufficient. A grader can sit comfortably inside the safe region and the loop can still retire nothing.
Two measurements exist and they point in opposite directions. One team audited a single real grader inside its own loop and put its false-pass rate at roughly 1%, clear of the threshold by a wide margin. A separate team scored seven graders against 156 agent runs that had genuinely failed, and every one waved through between 45.5% and 79.5% of those failures as passes, on a test set built to be hard, at one run each with no error bars. Nothing about your model choice, your rubric, or the accuracy number on your evaluation set tells you which of those populations yours belongs to.
One disclosure, because it cuts against the case we are making. The threshold, the 1% audit and the corruption experiment all come from one group of seven authors publishing across several papers. Only the seven-grader measurement is independent of them. Read this section as one team's framework plus one outside measurement, not as four teams agreeing.
What makes the bad case expensive is that it is invisible. In a third experiment, where researchers deliberately corrupted a working grader to see what happened, genuine retirements fell to zero while the raw count of skills removed per run declined but stayed positive, because skills were still being pushed off the shelf to make room for new ones. The clean baseline was only 1.3 genuine retirements per run to begin with, give or take 1.2, so this is a small quantity going to nothing rather than a collapse you would feel. Your dashboard keeps showing removals. The removals that were doing useful work have stopped. The threshold, and how to audit your own grader in a day.
THE SELECTION QUESTION
Anything you put in a skill file joins a pool the agent has to choose from, and pools behave badly as they grow.
Take a pool from 5 skills to 100 and how often the skill the agent actually used was the right one falls from 29.6% to 3.3%. Task success barely moves over the same range, from 36.4% to 39.3%, so this is not evidence that the work suffers. It is evidence that your selection metrics stop tracking anything. The authors put the mechanism this way: "pool size contributes to the difficulty, but semantic confusability is the more important stressor for identifying the correct procedural artifact." Semantic confusability means how easily a retriever mistakes one description for another. At a fixed pool of 100, an offline ranking test puts the right skill first 53.4% of the time when the skills resemble each other, 84.1% when the pool is drawn at random, and 93.2% when they are clearly different. That test was never carried through to a live run.
All of that was measured on pools of procedures. Nobody has run it on a library of facts about users, and we are not going to pretend otherwise. Our expectation is that facts are the harder case, because two procedures differ in kind while two user records differ only in their details. Treat that as our inference from the result, not as a finding. Why similarity beats count.
THE COST QUESTION
Skills are not the free option, and the cheapest thing in the comparison is usually not the skill.
The strongest head-to-head published this month ran three conditions over the same tasks with token accounting attached. Against a store built from the agent's own past runs, the skill condition scored 4.8 percentage points higher and spent 95,300 more tokens per task to get there, over the 83 tasks every condition attempted, as per-task averages rather than totals. A separate lab measured 158K tokens per task with skills switched on against 122K with them off, roughly 30% more, on an 87-task check at one run, with figures estimated from released cache counters rather than billed.
The honest other half: on those same 83 tasks, against no skill at all, the skill condition scored 5.5 points higher on 34,200 fewer tokens per task. Skills are not uniformly expensive. Read that alongside the paper's wider analysis of the same contrast, which is a different and larger set of runs: there, a skill against no skill comes out at 2.84 points with a range running from minus 2.27 to plus 7.95. That range crosses zero, so the paper does not establish that a skill beats nothing at all. The 83-task figure is one slice, not a settled result. What they are not is free, and the argument that a skill pays for itself because the agent only ever loads the part it needs does not survive a matched comparison against a cheaper technique. The full token accounting.
How skills actually travel
There is no registry and no package manager for skills, so a skill spreads by someone copying a folder. A July 2026 sweep matched 3,797,117 files by that filename across 282,200 public repositories, nine months after the format appeared. The authors kept every hit, including files whose names merely contain the term, and call the total a lower bound rather than a count of skills. Of what it collected, 50.5% is byte-for-byte identical to something else in the set, and because only exact matches were counted, that share is a floor. The sweep is a census of filenames. It runs no agent and scores no task, so it says nothing about whether any of those files are any good.
What follows is our reading of what that implies, not something the census measured. For a procedure, copying is the distribution mechanism working exactly as designed, and a pinned copy that never moves under you has an argument in its favour. For anything that changes, a copy with no link back to what it came from is the problem in its purest form.
Questions people ask before deciding
No. Use them for the job the evidence supports, which is procedure: which setup steps to run, in what order, what shape the output takes. With a skill in place, failures where the agent broke its own environment fall from 5.3% to 0.2%, and failures where the output came back in the wrong shape fall from 7.4% to 3.2%. Read the middle column of that table before crediting the skill with all of it: an undistilled pool of the agent's past runs already reaches 1.7% and 3.8%, so the skill is buying the last stretch. Where distilling clearly earns its place is the row nobody looks at: on reasoning errors that undistilled pool sits at 11.0% against 8.3% with no skill at all, so raw experience makes reasoning worse than doing nothing, and distilling it stops that. A skill does not repair a wrong algorithm. It stops your own transcripts from damaging one.
Then a file is fine and you do not need us for it. The question worth asking is how many of your facts are genuinely permanent. A plan, a preference, a shipping address, an entitlement, a decision someone reversed last week: each of those is a fact with a history, and a summarised file keeps only whatever was written to it last.
That is the right instinct, and it is the first step of building a write path rather than a file. The next steps are contradiction handling, entity resolution across channels, and deciding what decays. That is a different conversation and we have written it up at build vs buy.
The write path. Corrections get recorded instead of overwriting what they correct. Supersession is kept as a relationship between two statements rather than resolved into one. Chronology survives, because chronology is the cheapest possible encoding of which fact replaced which. How agent memory works.
Read the evidence base, then try Synap.