Skills Are Fine for Facts. They Break on Anything That Changes.

Plenty of teams are routing agent memory through skill files right now. The research published this month says that works for facts and fails for anything that changes.

Skills made the easy thing easy, which is why this keeps coming up. A skill file is a markdown document sitting in a folder, named SKILL.md. There is no schema to design, no store to run, and no eviction policy to argue about, and the agent picks the file up on its own when it looks relevant. Roughly 3.8 million files matching that filename sat in public GitHub repositories in July 2026, nine months after the format appeared, and I keep meeting teams who have noticed that a file the agent reads on every relevant turn looks a great deal like a place to keep what it knows about a user. It is a reasonable move. It works well enough at first that the failure, when it arrives, arrives late and quietly.

Between the 25th of July and the 18th of August 2026, arXiv, the open research archive, took in roughly fifty papers on the SKILL.md format. I read thirty-five of them end to end. Then I had every number the argument rests on re-derived from the source PDFs rather than from my own notes, which is how I found out that eight of the claims I had written down were not safe to print. What survived is narrower than what I started with and a good deal more interesting. The clearest single result in the set is not framed as being about memory at all.

What happens when you put skills and memory in one store

Filesystem-Based Memory for LLM Agents does something almost nobody else in this literature does. It puts procedural skills and declarative memory, meaning how-to instructions and facts about the user, into one store on purpose, and then measures both. The store is a directory tree of markdown files that a management agent writes and reorganises, which is what deployed agents actually do.

Three roles sit around that store. A management agent takes in each incoming chunk of conversation and reorganises what is already there. A read-only search agent answers questions and cites where each answer came from. Grading is done by the same model family at high effort, and a second judging pass over one benchmark put the noise floor at roughly one and a half points, which is how far a score can move with nothing having actually changed.

Three store designs were compared, running from doing nothing to doing a great deal. A verbatim dump keeps the raw chronological record and organises none of it. Foldered sessions move whole sessions into folders without editing a byte of their contents. The agent-curated store is the interesting one, and it is the one most production systems resemble: the management agent reads the incoming material, decides what it means, and writes structured notes about the user rather than a record of what the user said.

Each cell below is the share of that benchmark's questions answered correctly, as a percentage, under each of the three designs.

Everyone will quote the 37.5% against 78.1% on PersonaMem 32k. What matters more is the whole column, because the pattern across all four benchmarks is the argument.

Agent curation ties for the best score on LoCoMo, at 86.1%. It sits within a few points of the verbatim dump on PersonaMem 128k and on REALTALK. Then it collapses on exactly one benchmark of the four, losing forty points to a store that does no organising whatsoever and simply keeps the conversation in the order it happened. Something uniformly worse would be easy to dismiss and easy to fix. Something excellent three times and catastrophic once is telling you what it cannot represent, and the useful work is figuring out what.

The fact was in the store, and the answer was still wrong

Coverage is the obvious explanation. Curation compresses, compression drops information, and the dropped information is what the question needed. That explanation is wrong, and the authors checked.

For all thirteen questions that the verbatim dump answered correctly and the curated store missed, the fact the correct answer depended on was present in the curated store. Thirteen out of thirteen. Grounding scores across those conditions run from 0.94 to 0.98 and attribution from 0.97 to 1.0, both marked out of a maximum of 1.0, which is to say the search agent found relevant material and cited material that genuinely supported what it said. Retrieval worked. The fact was there. The answer was wrong anyway.

That single observation is the reason this piece exists, because it rules out the entire class of fixes teams reach for first. Better embeddings do not help. Pulling back more chunks per query does not help. Reranking does not help. The information is already in the context window when the model gets it wrong. Whatever failed, failed at the level of how the material was written down, not at the level of whether it was found.

Why it fails on that benchmark and not the others

PersonaMem 32k is not a general memory benchmark. Its questions ask for a persona's current state on a preference that changed at some point, and the wrong answer options are drawn from the superseded earlier state or from the static profile. It is built, deliberately, to punish a store that knows both what someone used to want and what they want now but cannot tell you which is which.

LoCoMo, where curation ties for best at 86.1%, asks mostly for facts that were true when they were said and stayed true. REALTALK is drawn from three weeks of real messaging and sits near parity at 75.3% against the dump's 77.6%. The one benchmark that collapses is the one constructed around supersession.

That matters for how this question gets argued. The benchmarks people reach for when they want to show that a summarised store works are mostly LoCoMo-shaped: they ask whether a fact from a long conversation can be recovered, and a well-organised store recovers facts beautifully. It is genuinely better than a raw dump at that job, which is what 86.1% against 84.2% on LoCoMo says. So a team that evaluates its memory layer on factual recall gets a clean result and ships, and does not find out what it broke until a user changes their mind about something and the agent keeps acting on the old answer months later.

Distillation does not fail by losing information. It fails because it resolves information, and resolution is destructive when the thing you needed was the unresolved history. A curated store answers "what does this person prefer" by writing down a preference. A chronological store answers it by keeping every time the person said something about it, in order, and letting the reader work out which one is current. The second is worse at almost everything and better at the one thing that turns out to matter for memory.

The three ways a summarised store goes wrong

Tracing those failures gave the authors three mechanisms, and each one is recognisable to anyone who has looked at a summarised memory store in production.

A superseded preference sits in the store as a present-tense fact about the user, right beside its own update. One section says the user prefers morning meetings, stated as a trait. Three sections later, another says they moved to afternoons after a schedule change. Both sentences are true statements about the record. Neither is marked as replacing the other, and a reader with no chronology has no principled way to choose.

First-person feeling gets rewritten into a neutral feature list. What was "I hated the onboarding, it took me four days and I nearly gave up" becomes an entry under difficulties encountered. The fact survives. The stance does not, and stance is frequently the thing a downstream agent needs in order to behave correctly.

Worth sitting with what that costs in a product. An agent that knows a user found onboarding painful will offer help before being asked, will avoid recommending the same path to a colleague at the same company, and will read a terse reply as friction rather than as disinterest. An agent holding the neutral version knows a fact about a completed event and behaves like it is reading a changelog.

Mechanism three is quieter and probably the most damaging. One narrative arc gets scattered across sections and files, filed by subject rather than by episode, so the fact that does the updating ends up somewhere the reader never sees it beside the thing it updates. Good filing separates related things whenever the filing scheme cuts across the narrative, which it usually does.

What weakens this result

Thirty-two questions in a single run carry the headline PersonaMem 32k comparison. The authors are explicit about the resolution: gaps of a few questions on the small tiers are, in their words, suggestive rather than settled. Thirteen questions moved. That is a real effect on a small sample, not a settled magnitude.

There is a stronger objection, and it comes from the same paper. When the store was rebuilt with a more capable management agent, the score recovered from 37.5% to 56.3%, and the authors read that as possibly a limitation of the backbone model rather than of the curated representation itself. A sign test on that recovery returns p = 0.073, which is the chance of a swing that large if the two curators were really equally good, so it is directional rather than decisive. And 56.3% still sits well below the dump's 78.1%. But the honest reading is that a better curator narrows the gap substantially. The other half of the same hypothesis, whether a better reader over the same curated store closes the rest of it, was not tested.

What survives all of that is the direction and the mechanism, not the size of the number. The thirteen-of-thirteen result does not depend on the magnitude at all; the facts were present regardless of how many questions moved. And the three failure modes are structural properties of resolving a chronology into a description, which is what any distillation step does by definition.

Four other papers point the same way

On its own the finding would be thin. It is not on its own.

Ctx2Skill distils long, unseen context into a generated SKILL.md file and prepends it to the prompt. The gains are real, taking GPT-4.1 from 11.1% to 16.5% task success on a 1,899-task benchmark. The detail that matters here sits in the formalism rather than the results: the raw context is supplied alongside the generated skill in every single condition. The skill is an addition to context and never a replacement for it, and the paper runs no version of the experiment that removes the context, because removing it was never the design. Even with both, the best configuration leaves roughly three quarters of tasks unsolved.

ContinualSkillBench ran the comparison that most closely resembles the choice a team is actually making. Agents worked through a hundred ordered subtasks per domain and could create and modify their own skill files as they went. Scores below are on a zero-to-one scale, averaged over three of the five domains on GPT-5.3-Codex. An independent baseline that resets everything scores 0.466. Plain in-context learning, which keeps the history and the feedback in the prompt but cannot write skill files, scores 0.605. The skill-maintaining condition scores 0.602. Do not read that gap as a result, because with one run per cell it is a coin flip. Read the authors' own conclusion instead, which is that explicit skill maintenance provides no consistent aggregate advantage over in-context learning. Where the skills did win was on rigid output formats, a real and useful capability I have written about separately in what skills are actually good for. Where in-context learning won was on the open-ended, rubric-scored work, in all three of those domains.

Do Personalized Skills Help Coding Agents? mined skills from individual developers' own coding-agent histories, then tested them on those developers' later tasks. Against a generic skill pooled across everyone, a developer's own personalised skill scored −8.00, −3.81, −7.20 and only +5.67 points, across four bins defined by how many task-relevant prior sessions that developer had. Personalisation only wins in the top bin, and those four bins hold three, sixteen, eleven and twelve tasks on a single seed, so treat the shape rather than the values. The cleaner statistic sits in the appendix: six of thirteen developers improved with their own skill, while the generic skill helped eleven of thirteen. Roughly twelve sessions of history is not enough evidence to distil a person into a procedure. A memory layer needs no such threshold, because it is not trying to generalise; it is trying to remember.

One last corroboration comes from the people building the delivery infrastructure rather than from anyone studying memory. @skills, a protocol paper accompanied by a crawl of 56,804 published skills, scopes a skill to procedural knowledge throughout and draws the boundary explicitly in its related work: retrieval over facts is implicit and similarity-based, whereas skill references are explicit and deterministic, which is what reliability requires for instructions as opposed to facts. Their own framing separates the two cases, and the paper says nothing about user state, history or personalisation because those were never in scope. It is a design argument rather than a measured one, and the author discloses that he founded the company operating the hub the paper recommends, so weigh it accordingly. I include it because it is the clearest statement of the boundary I found, and it comes from someone with every incentive to draw the circle around skills as widely as possible.

One store is fine. One write policy is not.

A single substrate is fine. The filesystem-memory authors argue for exactly that, describing declarative memory and skills as one filesystem memory carrying different content, and nothing in this corpus refutes it. Files are files.

One write policy is not fine, and that is where the distinction actually lives.

A skill store wants distillation. Its whole value is that a procedure gets generalised out of specific episodes, so the agent does the thing the same way every time. It wants versioned commits, a replay gate before any widening of scope is accepted, and retirement based on measured contribution, all of which the rest of this literature is converging on independently.

A memory store wants the opposite discipline at write time. Corrections recorded rather than overwritten. Supersession preserved as a relationship between two statements rather than resolved into one. Chronology kept, because chronology is the cheapest possible encoding of which fact replaced which. Stance kept alongside content, because a downstream agent that knows a user disliked something behaves differently from one that knows the thing happened.

Concretely, the same event gets written down twice and differently. The skill store learns that when a user reports an onboarding failure, the agent should check the integration logs before suggesting a reinstall, and that procedure is written once, versioned, and applies to everyone. The memory store records that this particular user hit that failure on a Tuesday in March, said they nearly gave up, and later said the second attempt went fine. Neither entry substitutes for the other. Ask the skill store what this user thinks and it has nothing; ask the memory store what to do about it and it has nothing.

Both of those write policies have a literature behind them now, and the skill side is further along than most teams realise; the governance machinery for versioning, replay-gating and retiring skills is covered in self-evolving skill libraries.

Those two policies are in direct opposition. Distillation is what makes a skill useful and it is exactly what makes a memory wrong. Running one store under one policy means choosing which of the two jobs to do badly, and the evidence above says the memory job is the one that breaks first and least visibly, because the facts are all still sitting there and every diagnostic you have will tell you retrieval is fine.

That is the practical warning. A team routing memory through skills will not see a retrieval failure, will not see missing data, and will see an agent confidently answering with a preference the user abandoned two months ago.

How to test this on your own stack

None of the above requires taking my word for it, and the experiment is small enough to run in an afternoon.

Take fifty exchanges from a real user's history where something changed: a preference reversed, a project got cancelled, a role changed, a decision got revisited. Write down by hand what the correct current answer is for each. Then ask your system two kinds of question about the same fifty exchanges. The first kind is factual and stable, of the form what did this user tell us about X. The second kind is stateful, of the form what does this user currently want regarding X, where the honest answer changed at some point in the record.

Score both. If the two scores are close, your write path is preserving supersession and you can stop reading. If the stateful score is materially worse, open the failures and check the one thing that decides what to do next: was the superseding fact present in the retrieved context. When it is present and the answer is still wrong, no amount of retrieval work will help you, and the fix is in how the material is written down rather than in how it is found.

Teams skip that second half, and it is the half that tells you which budget to spend.

What would change my mind

One test the authors did not run, on the search side. Their own hypothesis has two halves, and only one has been examined: a stronger writer recovered the score from 37.5% to 56.3%. Nobody has yet put a stronger reader over the same curated store. If a better search agent closes the remaining gap to the verbatim dump's 78.1%, then the failure is a retrieval problem after all, the whole argument above weakens considerably, and the right answer is to spend on retrieval rather than on representation.

I do not think it will, because the thirteen-of-thirteen result already shows a competent reader finding the right material and still getting the answer wrong. But that is an inference from one experiment on thirty-two questions, and it is the experiment I would want run before anyone rebuilds anything on the strength of this piece.

Retrieval was never the problem here. A summarised file cannot say which fact replaced which, and holding that relationship is the whole job a memory layer exists to do. We argue that case directly in the skills question.

---

Every figure in this piece is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, where the full claim register, including the eight claims that did not survive verification, is published in the open. We build Synap, which is a memory layer for agents, so read the argument above with that in mind and check the register.