I Read 35 Papers on Agent Skills Published in the Last Three Weeks. Here Is What They Say About Memory.

A lot of teams are currently routing agent memory through skills, and the research published this month says that works for facts and fails for state.

Skills made the easy thing easy, which is why this comes up constantly right now. A SKILL.md file is a markdown document in a folder; there is no schema to design, no store to run, no eviction policy to argue about, and the agent picks it up on its own. Roughly 3.8 million of these files sit in public GitHub repositories as of July 2026, nine months after the format appeared, and I keep meeting teams who have noticed that a file the agent reads on every relevant turn looks a great deal like a place to keep what it knows about a user. It is a reasonable move. It works well enough at first that the failure, when it arrives, arrives late and quietly.

Between the 25th of July and the 18th of August 2026, arXiv absorbed roughly fifty papers on the SKILL.md artifact. I read thirty-five of them end to end, then had every load-bearing number re-derived from the source PDFs rather than from my own notes, which is how I found out that eight of the claims I had written down were not safe to print. What survived is narrower than what I started with and considerably more interesting. The single clearest result in the set is not framed as being about memory at all.

The table, not the cell

Filesystem-Based Memory for LLM Agents does something almost nobody else in this literature does: it puts procedural skills and declarative memory in one store on purpose, then measures both. The store is a directory tree of markdown files that a management agent writes and reorganises, which is what deployed agents actually do.

Three roles sit around that store. A management agent integrates each incoming chunk of conversation and reorganises what is already there; a read-only search agent answers questions with citations; and the judging is done by the same model family at high effort, with a re-judging pass on one benchmark establishing a noise floor of roughly a point and a half. The store designs run from doing nothing to doing a great deal. A verbatim dump keeps the raw chronological record and organises nothing. Foldered sessions move whole sessions into folders without editing a byte of their contents. The agent-curated store is the interesting one and the one most production systems resemble: the management agent reads the incoming material, decides what it means, and writes structured notes about the user rather than a record of what the user said.

Everyone will quote 37.5 against 78.1. What matters is the whole column, because the pattern is the argument.

Agent curation ties for the best score on LoCoMo. It sits within a few points of parity on PersonaMem 128k and on REALTALK. Then it falls off a cliff on exactly one benchmark, losing forty points to a store that does no organising whatsoever and simply keeps the conversation in the order it happened. A technology that were uniformly worse would be easy to dismiss and easy to fix. A technology that is excellent three times and catastrophic once is telling you something specific about what it cannot represent, and the useful work is figuring out what.

Nothing was lost, and the answer was still wrong

Coverage is the obvious explanation. Curation compresses, compression drops information, and the dropped information is what the question needed. That explanation is wrong, and the authors checked.

For all thirteen questions that the verbatim dump answered correctly and the curated store missed, the gold-critical fact was present in the curated store. Grounding scores across those conditions run from 0.94 to 0.98 and attribution from 0.97 to 1.0, which is to say the search agent found relevant material and cited material that genuinely supported what it said. The retrieval worked. The fact was there. The answer was wrong anyway.

That single observation is the reason this piece exists, because it rules out the entire class of fixes that teams reach for first. Better embeddings do not help. A larger top-k does not help. Reranking does not help. The information is already in the context window when the model gets it wrong. Whatever failed, failed at the level of how the material was written down, not at the level of whether it was found.

Why it fails on that benchmark and not the others

PersonaMem 32k is not a general memory benchmark. Its questions ask for the persona's latest state of a preference that changed, and the distractors are drawn from the superseded earlier state or from the static profile. It is built, deliberately, to punish a store that knows both what someone used to want and what they want now but cannot tell you which is which.

LoCoMo, where curation ties for best, asks mostly for facts that were true when they were said and stayed true. REALTALK is drawn from three weeks of real messaging and sits near parity. The benchmark that collapses is the one constructed around supersession.

This matters for how the question gets argued, because the benchmarks people reach for when they want to show that a summarised store works are mostly LoCoMo-shaped. They ask whether a fact from a long conversation can be recovered, and a well-organised store recovers facts beautifully; it is genuinely better than a raw dump at that job, which is what the 86.1 against 84.2 says. A team that evaluates its memory layer on factual recall will therefore get a clean result and ship, and will not find out what it broke until a user changes their mind about something and the agent keeps acting on the old answer months later.

So the failure is not that distillation loses information. It is that distillation resolves information, and resolution is destructive when the thing you needed was the unresolved history. A curated store answers "what does this person prefer" by writing down a preference. A chronological store answers it by keeping every time the person said something about it, in order, and letting the reader work out which one is current. The second is worse at almost everything and better at the one thing that turns out to matter for memory.

The three ways it goes wrong

The authors traced the failures and named three mechanisms, and each one is recognisable to anyone who has looked at a summarised memory store in production.

Superseded preferences are left standing as present-tense traits beside their updates. The store says the user prefers morning meetings, in one section, as a fact about them; three sections later it says they moved to afternoons after a schedule change. Both sentences are true statements about the record. Neither is marked as replacing the other, and a reader with no chronology has no principled way to choose.

First-person affect gets rewritten into neutral feature lists. What was "I hated the onboarding, it took me four days and I nearly gave up" becomes an entry under difficulties encountered. The fact survives. The stance does not, and stance is frequently the thing a downstream agent needs in order to behave correctly.

Worth sitting with what that costs in a product. An agent that knows a user found onboarding painful will offer help before being asked, will avoid recommending the same path to a colleague at the same company, and will read a terse reply as friction rather than as disinterest. An agent holding the neutral version knows a fact about a completed event and behaves like it is reading a changelog.

The third mechanism is quieter and probably the most damaging: one narrative arc gets scattered across sections and files, organised by subject rather than by episode, so the decisive updating fact ends up somewhere the reader never sees alongside the thing it updates. Good filing separates related things whenever the filing scheme cuts across the narrative, which it usually does.

What weakens this, said before anyone else says it

Thirty-two questions in a single run carry the headline comparison. The authors are explicit about the resolution: gaps of a few questions on the small tiers are, in their words, suggestive rather than settled. Thirteen questions moved. That is a real effect on a small sample, not a settled magnitude.

There is a stronger objection, and it comes from the same paper. When the store was rebuilt with a more capable management agent, the score recovered from 37.5 to 56.3, and the authors read that as a limitation of the backbone model rather than of the curated representation itself. The sign test on that recovery comes in at p = 0.073, so it is directional rather than decisive, and 56.3 still sits well below the dump's 78.1. But the honest reading is that a better curator narrows the gap substantially. The search side of the same hypothesis, whether a better reader over the same curated store closes the rest of it, was not tested.

What survives all of that is the direction and the mechanism, not the size of the number. The thirteen-of-thirteen result does not depend on the magnitude at all; the facts were present regardless of how many questions moved. And the three failure modes are structural properties of resolving a chronology into a description, which is what any distillation step does by definition.

Three other papers arriving at the same place

On its own the finding would be thin. It is not on its own.

Ctx2Skill distils long, unseen context into a generated SKILL.md and prepends it. The gains are real, taking GPT-4.1 from 11.1% to 16.5% on a 1,899-task benchmark. The detail that matters here is in the formalism rather than the results: the raw context is supplied alongside the generated skill in every single condition. The skill is an addition to context and never a replacement for it, and the paper contains no ablation that removes the context, because removing it was never the design. Even with both, the best configuration leaves roughly three quarters of tasks unsolved.

ContinualSkillBench ran the comparison that most closely resembles the choice a team is actually making. Agents worked through a hundred ordered subtasks per domain and could create and modify their own skill files as they went. Averaged over three domains on GPT-5.3-Codex, an independent baseline that resets everything scores 0.466; plain in-context learning, which keeps the history and the feedback but cannot write skills, scores 0.605; and the skill-maintaining condition scores 0.602. Do not read the 0.003 gap as a result, because with one run per cell it is a coin flip. Read the authors' own conclusion, which is that explicit skill maintenance provides no consistent aggregate advantage over in-context learning. Where the skills did win was on rigid output formats, which is a real and useful capability that I have written about separately in what skills are actually good for. Where in-context learning won was on the open-ended, rubric-scored work, in all three domains they tested.

Do Personalized Skills Help Coding Agents? mined skills from individual developers' own coding-agent histories and tested them on those developers' later tasks. Against a generic skill pooled across everyone, a developer's own personalised skill scored −8.00, −3.81, −7.20 and only +5.67 across bins defined by how many task-relevant prior sessions that developer had. Personalisation only wins in the top bin, and the bins hold three, sixteen, eleven and twelve tasks on a single seed, so treat the shape rather than the values. The cleaner statistic sits in the appendix: six of thirteen developers improved with their own skill, while the generic skill helped eleven of thirteen. Roughly twelve sessions of history is not enough evidence to distil a person into a procedure. A memory layer needs no such threshold, because it is not trying to generalise; it is trying to remember.

The last corroboration comes from the people building the delivery infrastructure rather than from anyone studying memory. @skills, a protocol paper accompanied by a crawl of 56,804 published skills, scopes a skill to procedural knowledge throughout and draws the boundary explicitly in its related work: retrieval over facts is implicit and similarity-based, whereas skill references are explicit and deterministic, which is what reliability requires for instructions as opposed to facts. Their own framing separates the two cases, and the paper says nothing about user state, history or personalisation because those were never in scope. It is a design argument rather than a measured one, and the author discloses that he founded the company operating the hub the paper recommends, so weigh it accordingly. I include it because it is the clearest statement of the boundary I found, and it comes from someone with every incentive to draw the circle around skills as widely as possible.

What follows from this architecturally

One substrate is fine. The filesystem-memory authors argue for exactly that, describing declarative memory and skills as one filesystem memory carrying different content, and nothing in this corpus refutes it. Files are files.

One write policy is not fine, and that is where the distinction actually lives.

A skill store wants distillation. Its whole value is that a procedure gets generalised out of specific episodes, so the agent does the thing the same way every time. It wants versioned commits, a replay gate before any widening of scope is accepted, and retirement on measured contribution, all of which the rest of this literature is converging on independently.

A memory store wants the opposite discipline at write time. Corrections recorded rather than overwritten. Supersession preserved as a relationship between two statements rather than resolved into one. Chronology kept, because chronology is the cheapest possible encoding of which fact replaced which. Stance kept alongside content, because a downstream agent that knows a user disliked something behaves differently from one that knows the thing happened.

Concretely, the same event gets written down twice and differently. The skill store learns that when a user reports an onboarding failure, the agent should check the integration logs before suggesting a reinstall, and that procedure is written once, versioned, and applies to everyone. The memory store records that this particular user hit that failure on a Tuesday in March, said they nearly gave up, and later said the second attempt went fine. Neither entry substitutes for the other. Ask the skill store what this user thinks and it has nothing; ask the memory store what to do about it and it has nothing.

Both of those write policies have a literature behind them now, and the skill side of it is further along than most teams realise; the governance machinery for versioning, replay-gating and retiring skills is covered in self-evolving skill libraries.

Those two policies are in direct opposition. Distillation is what makes a skill useful and it is precisely what makes a memory wrong. Running one store with one policy means choosing which of the two jobs to do badly, and the evidence above says the memory job is the one that breaks first and least visibly, because the facts are all still sitting there and every diagnostic you have will tell you the retrieval is fine.

That is the practical warning. A team routing memory through skills will not see a retrieval failure, will not see missing data, and will see an agent confidently answering with a preference the user abandoned two months ago.

Testing this on your own stack

None of the above requires taking my word for it, and the experiment is small enough to run in an afternoon.

Take fifty exchanges from a real user's history where something changed: a preference reversed, a project got cancelled, a role changed, a decision got revisited. Write down, by hand, what the correct current answer is for each. Then ask your system two kinds of question about the same fifty exchanges. The first kind is factual and stable, of the form what did this user tell us about X. The second kind is stateful, of the form what does this user currently want regarding X, where the honest answer changed at some point in the record.

Score both. If the two scores are close, your write path is preserving supersession and you can stop reading. If the stateful score is materially worse, open the failures and check the one thing that decides what to do next: was the superseding fact present in the retrieved context. When it is present and the answer is still wrong, no amount of retrieval work will help you, and the fix is in how the material is written down rather than in how it is found.

The second half of that check is the part teams skip, and it is the part that tells you which budget to spend.

What would change my mind

The search-side test the authors did not run. Their own hypothesis has two halves, and only one has been examined: a stronger writer recovered the score from 37.5 to 56.3. Nobody has yet put a stronger reader over the same curated store. If a better search agent closes the remaining gap to the verbatim dump, then the failure is a retrieval problem after all, the whole argument above weakens considerably, and the right answer is to spend on retrieval rather than on representation.

I do not think it will, because the thirteen-of-thirteen result already shows a competent reader finding the right material and still getting the answer wrong. But that is an inference from one experiment on thirty-two questions, and it is the experiment I would want run before anyone rebuilds anything on the strength of this piece.

---

Every figure in this piece is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, where the full claim register, including the eight claims that did not survive verification, is published in the open. We build Synap, which is a memory layer for agents, so read the argument above with that in mind and check the register.