# When Agent Skills Go Stale: Why a Saved Skill Can Do Worse Than No Skill, How to Repair Only the Broken Part, and How to Gate an Agent's Edits to Its Own Skills

> A saved agent skill can do worse than no skill once the environment changes: in an AWS study published 17 November 2026, a skill that added 4.7 points to Claude Opus 4.7 on Terminal-Bench 2 left it 11.4 points below no skill after one change. How to find the stale part by replaying recorded checks, repair only what conflicts (Haiku tax-form accuracy 30.5% to 97.5%), gate an agent's self-edits by what they break and count memory pollution (Google's GUARD), reject risky updates (Safe Personal RSI, 20 of 21), and write skills from memory only past an evidence bar (Perplexity).

_Gaurav Dadhich · 2026-11-20_

\# When Agent Skills Go Stale: Why a Saved Skill Can Do Worse Than No Skill, How to Repair Only the Broken Part, and How to Gate an Agent's Edits to Its Own Skills \*Published 20 November 2026. Every figure below was read from the paper or post it is attributed to on 20 November 2026; all four sources were published between 17 and 18 November 2026.\* A saved agent skill can do worse than having no skill at all once the environment it was written for changes. In \[When Agent Skills Go Stale\](https://arxiv.org/abs/2611.22525), a 17 November 2026 paper from AWS Agentic AI and the University of Southern California, a skill that raised Claude Opus 4.7's pass rate on Terminal-Bench 2 by 4.7 points (from 92.1% to 96.8%) dropped it to 78.1% after a controlled change to the environment, which is 11.4 points below the same model with no skill (89.5%). The skill had not become wrong everywhere. One saved value no longer matched the environment, and the agent followed it anyway. The fix the paper tests is narrow on purpose: replay checks recorded from earlier successful runs, find the part of the skill that conflicts with what the environment now returns, repair only that part, validate it, and rebuild from scratch only when the conflict cannot be pinned down. On historical IRS tax forms, that lifted Claude Haiku 4.5's field accuracy with a stale skill from 30.5% to 97.5%. Two other papers and one production write-up from the same week cover what happens when the agent maintains its skills itself. Google Cloud's \[GUARD\](https://arxiv.org/abs/2611.23476) promotes a self-edit only when what it fixes outweighs what it newly breaks, and it reports memory pollution next to accuracy. \[Safe Personal RSI\](https://arxiv.org/abs/2611.23651) puts every update to a personal assistant's instructions through risk and utility checks and rejected 20 of 21 deliberately risky update attempts. \[Perplexity's self-improving skills\](https://www.perplexity.ai/hub/blog/self-improving-skills) write a new skill from a user's memory only after the same behaviour has appeared twice, and an eight-week randomized test on production traffic reported 11% fewer tool-call failures. The sections below take each one in the order an engineer hits the problem. ## What a stale agent skill is A skill goes stale when a change in the environment makes some of what it saved invalid while the rest stays useful. That definition, from the AWS paper, matters because of the second half: a stale skill is mostly right. A software update renames one API endpoint and leaves the rest of the workflow intact; an \[MCP\](https://modelcontextprotocol.io) server adds a new connection requirement, so one saved connection setting is now wrong while every other step in the procedure still applies. A skill is a cache of the environment it was written in. It stores the paths, identifiers, field names and assumptions about API behaviour that were true on the day it was authored, so that the agent can skip rediscovering them. Skipping rediscovery is where the savings come from, and it is also why the skill has no way of noticing that the thing it skipped has moved. ## How much damage a stale skill does The AWS team compared skill reuse before and after a single controlled change in four settings: \[Terminal-Bench 2\](https://www.tbench.ai), historical IRS tax forms, \[AppWorld\](https://appworld.dev) and \[ScienceWorld\](https://github.com/allenai/ScienceWorld), using Claude Opus 4.7 and Claude Haiku 4.5. Each change touched one thing a skill could have saved, so every loss can be traced to one stale commitment. | Setting and model | No skill | Skill, original environment | No skill, changed environment | Stale skill, changed environment | |---|---|---|---|---| | Terminal-Bench 2, Opus 4.7 | 92.1% | 96.8% (+4.7) | 89.5% | 78.1% (11.4 below no skill) | | Terminal-Bench 2, Haiku 4.5 | 44.7% | 87.4% (+42.6) | 44.7% | 56.1% (+11.4 over no skill) | Pass rates for the cache-heavy skill variant, from Table 1 of \[arXiv 2611.22525\](https://arxiv.org/abs/2611.22525). On IRS forms the stale cache-heavy skill also left Opus 12.5 points below no skill. Several findings in the paper are worth carrying into any design review. A previously helpful skill can cut task performance by more than 10 points. It can raise execution cost by nearly 40%: on ScienceWorld, the stale skill left Opus 13 points below no skill and needing 38.9% more environment steps. In other settings task success held while the savings shrank, so a dashboard that tracks only success rate would show nothing wrong. And the damage depends on the model: the same stale Terminal-Bench skill still helped Haiku by 11.4 points, because Haiku without a skill was weak enough that a partly wrong skill was still better than none. Agents do partly rescue themselves at runtime. On the tax forms, both models read the current form before writing in most runs, but reading did not mean using: across both skill styles, Haiku still wrote to a field that no longer existed in 36 of the 80 runs where it had inspected the form, against 7 of 73 for Opus. Recovery also costs turns, and it fixes nothing for next time, because the skill file is unchanged at the end of the run. ## Which parts of a skill go stale: paths, field names, endpoints The changes that broke skills in the study were ordinary operations work: software and configuration updates, renamed or renumbered form fields, renamed API endpoints and an object moved to a different room in a simulated world. The parts of a skill that broke were the ones that stored a concrete value, such as a file path, a field identifier, a schema, a package choice or a connection setting. Authoring style decides how badly a skill ages. The paper wrote every skill twice from the same material: a procedural version that keeps discovery steps and fallbacks ("look up the field list, then fill"), and a cache-heavy version that strips them and stores the answers. Cache-heavy skills saved more up front (on Terminal-Bench, Opus used 66.5% fewer agent turns with the cache-heavy skill against 34.9% fewer with the procedural one) and lost more after the change: the cache-heavy skill lost 16.1 points of its benefit for Opus on Terminal-Bench, while the procedural skill's benefit grew from 1.1 to 7.9 points, and on IRS forms the losses were 31.3 points against 12.6. In the authors' words, "procedural lookup helps recover changed paths and field names." The practical rule: keep the lookup and cache its result as a hint, not as the instruction. A line such as "the export lives in \`reports/q3/\` (confirm with \`ls reports/\`)" costs one call and survives a reorganisation; "write to \`reports/q3/export.csv\`" does not. The same holds outside code, where repository layout, package manager, test command, deploy flow and team conventions all move, a pattern \[Saqueib's write-up on dev.to\](https://dev.to/saqueib/claude-code-skills-need-maintenance-not-just-a-good-first-draft-3bae) breaks down as structural, standards, product-context and tooling rot. ## How to find the stale part without being told what changed Selective maintenance, as the AWS paper calls it, needs no changelog. It works in four moves: 1. \*\*Record checks while the skill works.\*\* From the skill's saved content and its successful executions, build checks (does this path exist, does this endpoint answer, does the form still have the field the skill writes to) and store the result each one gave in the original environment. 2. \*\*Replay the checks with no model calls.\*\* When the skill is next used, or on a schedule, run the same checks against the current environment. This is cheap because nothing is generated. 3. \*\*Separate harmless drift from conflict.\*\* Not every difference matters. The method asks whether an observed difference conflicts with something the skill actually relies on. 4. \*\*Repair locally, validate, and rebuild only as a fallback.\*\* Patch the affected content and validate the patch; if the conflict cannot be localised or the repair does not validate, rebuild the skill's instructions and code. The checks are the skill's own tests. A skill without them is a cache with no invalidation, and the first sign of a problem is a worse answer. ## Repair only what conflicts, and rebuild only when you cannot Compared with reusing the stale skill, selective maintenance raised Haiku's tax-form field accuracy from 30.5% to 97.5% (the no-skill baseline was 4.6%), raised Haiku's Terminal-Bench pass rate with the stale skill from 56.1% to 64.9%, and cut ScienceWorld environment steps by up to 37% in a case where the stale skill already scored perfectly but took the long way. Mean execution cost fell in all eight model and setting combinations. It is not free and not always a win. On the tax forms, the tokens saved on later tasks paid back the one-time model cost of maintenance after roughly 21 to 42 uses, compared with retiring the skill and going back to no skill. On AppWorld, the procedural Haiku skill scored lower after maintenance (56.9% against 62.7%). The study also covers one change at a time, two models from one family and a single maintenance event, which the authors list as limits. Real environments change several things at once. Repairing beats deleting for a reason the paper makes concrete: most of a stale skill is still right. A full rewrite throws away the parts that were working along with the one that was not, and an agent that regenerates a skill from scratch can reintroduce errors the old version had already fixed. ## When the agent edits its own skills, gate by what the edit breaks Self-improving agents add a second way for skills to go wrong: the edit itself. A rule written to fix last week's failures can break tasks that already worked, a change of workflow or topic can overwrite earlier skills, and unverified additions pile up. GUARD, from Google Cloud (\[arXiv 2611.23476\](https://arxiv.org/abs/2611.23476), 17 November 2026), names these as local overfitting, catastrophic forgetting and memory pollution. GUARD's answer is to score each candidate edit by direction rather than by one number. It turns every task into checks written before execution, grades runs against them, and computes what it calls directional regret, which separates what an edit newly broke from what was already unsolved. Rules the agent has mastered are pinned. Several candidate edits are tried in parallel, and a gate promotes the winner only when its gains outweigh the regressions it causes, narrowing an over-broad rule with a precondition instead of deleting it. On the hosted EvoHarnessBench SDK track (43 test tasks on an evolving calendar tool, with Gemini 4 in every role), GUARD reached 80.8% accuracy against 71.9% for the agent with no self-improvement, a gain of 8.89 points, and 6.28 points above Meta-Harness, the strongest baseline in that table at 74.5%. It used 59,651 tokens per task against 131,032 for the base agent, which is 54.5% fewer (Meta-Harness used 78,705). All 19 promotions it made across its execution-gated benchmarks were non-regressive. Those are the authors' own results on their chosen benchmarks. The pattern travels without the paper's machinery. Before an agent's edit to a skill goes live, replay the tasks the old skill already passed; reject the edit if any of them now fails, however much it improves the new case. The question for the edit is "what did this break", asked before "what did this fix". A second problem sits under the gate: the agent that proposes a change is optimising to pass whatever checks it can see, a point \[Harry Floyd makes about release engineering for self-improving agents\](https://harryfloyd.substack.com/p/self-improvement-is-release-engineering). Checks recorded from earlier successful runs help here, because the proposer did not write them for this edit. ## Count memory pollution, not only accuracy Memory pollution is the build-up of unverified, stale or conflicting rules in what an agent carries forward. It does not show on an accuracy chart until it is bad, because each added rule is individually plausible. GUARD reports it as a separate metric on PAST-Bench, a cross-session benchmark that scores write precision, recall, update correctness, retention and pollution. The numbers show why accuracy alone misleads. The aggressive rewriters in the comparison (RRSI and Meta-Harness) reached update correctness of 0.901 to 0.903, which looks excellent, while tripling pollution to 0.500 overall and 1.000 on procedural tasks. Methods that only append kept pollution at the 0.167 floor but held on to stale clauses after updates, with procedural update correctness of 52.9% and 46.4%. GUARD reported 90.1% update correctness on procedural tasks with zero procedural pollution, and overall it led update correctness (0.924) while matching the 0.167 pollution floor, with zero pollution on 10 of 12 task families. If you run a self-improving loop, track two numbers per update: did the agent get the change right, and how much stale or conflicting text did it leave behind. A loop that scores well on the first and badly on the second is filling its own context with rules that will fail later. The same failure in a memory store, where a newer statement quietly overrides a better one, is covered in our post on \[agent memory versioning and rollback\](https://www.maximem.ai/blog/agent-memory-versioning). ## A gate for risky updates to a personal assistant Safe Personal RSI (\[arXiv 2611.23651\](https://arxiv.org/abs/2611.23651), 18 November 2026) is a technical report on update evaluation for Kicker, a personal health assistant. It limits self-improvement to changes in the assistant's instructions, with model weights and runtime code fixed, and puts each proposed change through a procedure: test the proposed and current instructions, repeat the failing cases, temporarily undo parts of the change to guide repairs, and check both safety and task performance before adoption. If no version passes, the current instructions stay in use. Across three models and two deliberately risky edits, on a benchmark of 29 email, calendar, health and marketplace cases, the procedure rejected 20 of 21 update attempts: 12 on risk checks and 8 on utility checks. An admitted repair on GPT-5.6 Luna cut attack violations from 10.7% to 0% and raised safe task completion from 89.7% to 100%. The authors also report, to their credit, that a broad handwritten authority rule matched those outcomes, so clear requirements can still be written as plain rules; the procedure's value is in catching the updates nobody thought to write a rule for. Two parts of that design carry over to any agent that edits itself. Keeping the current version is the default outcome, not a failure state. And safety gets its own checks, separate from task success, because an update can make an assistant more helpful and more willing to disclose something in the same change. Poisoned inputs reaching an agent's saved instructions are a related route, covered in \[how to prevent memory poisoning attacks on AI agents\](https://www.maximem.ai/blog/memory-poisoning-attacks-ai-agents). ## Write skills from memory only past an evidence bar Perplexity's write-up (\[Self-improving skills\](https://www.perplexity.ai/hub/blog/self-improving-skills), 18 November 2026) describes how Brain, the self-improvement system in Perplexity Computer, writes personal skills from a user's memory and session history. The rules are specific: - \*\*Creating a skill\*\* needs two independent occurrences of the same reusable behaviour. - \*\*Updating a skill\*\* can rest on a single grounded occurrence. - \*\*The agent proposes, it does not edit.\*\* Each proposal records the change, the reasoning and the evidence; deterministic checks confirm it is well formed and applies, and an LLM judge audits a sample. - \*\*One change per skill per run\*\*, and a cap of 10 automatically created skills per user. At the cap, adding a skill means retiring one and justifying why the new one is more useful. - \*\*Built-in skills shared by all users\*\* change only if every test scenario passes with the revised skill and at least one fails with the original, scenarios built from control sessions are replayed to confirm nothing has stopped working, and an engineer approves the pull request. Perplexity ran an eight-week randomized test on production traffic. 72.9% of users with automatic skill improvement got at least one new personal skill, against 6.5% of users without it who added one by hand. Tool-call failures fell by 11% across all Computer activity (about 7% excluding the system's own background runs), medium or high dissatisfaction threads per user fell by 7.6%, and Perplexity saw no measurable change in cost or response time. These are Perplexity's own measurements of its own product, reported as relative changes, and the first comparison is automatic creation against manual creation rather than two equal arms. The evidence bar is the part to copy. A rule seen once is an anecdote; writing it into a skill makes it part of every future run. Requiring a second independent occurrence before creating a skill, and a cap that forces retirement, is how a self-writing system avoids the pattern Addy Osmani describes in \[Audit your Agent files\](https://addyosmani.com/blog/audit-your-agent-files/), where a rule is added after every error until the file is too long to follow. ## A maintenance checklist for skills you own For skills you write and maintain yourself, the four sources reduce to a short list: - \*\*Store a lookup next to every concrete value\*\* (path, field, endpoint, setting), so the skill can rediscover what it cached. - \*\*Record checks while the skill works\*\*, and replay them on a schedule or before use, with no model calls. - \*\*Repair the conflicting part and validate it\*\*; rebuild only when the conflict cannot be localised. - \*\*Gate every edit, human or agent, on the tasks the old version passed.\*\* Reject anything that breaks one. - \*\*Measure pollution alongside accuracy\*\* for any loop that writes to its own skills or memory. - \*\*Require repeated evidence before creating a skill\*\*, cap the library, and keep the current version when nothing passes. - \*\*Give each skill an owner, a last-reviewed date and the change events that should trigger review\*\*, such as a package manager switch or a repository restructure. Distributing updates to installed copies is a separate, solved problem: Joost de Valk's \[self-updating skills\](https://joost.blog/self-updating-agent-skills/) post shows a \`SessionStart\` hook that runs \`npx skills update\` outside the context window. One more line is worth drawing. A skill is the right home for procedure that stays true and the wrong home for facts that get replaced, which is the argument of \[We do not need a memory layer. We have skills.\](https://www.maximem.ai/memory-vs-agent-skills) and of our research note on \[whether agent skills can replace memory\](https://www.maximem.ai/research/agent-skills/skills-vs-agent-memory). Every concrete value cached in a skill is a fact that can be replaced. Maximem Synap, our memory layer, treats each memory as a separate statement so that it "can notice that a new one contradicts an old one, work out which should win, keep the old one linked to its replacement, and tell you why it changed." Skills need the same discipline, and when a self-evolving library uses an LLM judge to decide what to keep, the judge's reliability sets the ceiling, as our note on \[self-evolving skill libraries\](https://www.maximem.ai/research/agent-skills/self-evolving-skill-libraries) shows. ## Frequently asked questions ### Can an agent skill make performance worse than no skill? Yes. In the AWS study, a skill that added 4.7 points to Claude Opus 4.7 on Terminal-Bench 2 left it 11.4 points below the no-skill baseline after one controlled change to the environment, and a stale tax-form skill left Opus 12.5 points below no skill. The stale part was a small piece of saved state, such as a path, field name or setting. ### How do I know which part of a skill is stale? Record checks from the skill's successful runs (paths that existed, endpoints that answered, fields that were present) and replay them against the current environment without model calls. Differences that conflict with what the skill relies on point at the stale part. The AWS method does this with no description of what changed. ### Should I rewrite a stale skill or repair it? Repair the conflicting part and validate it; rebuild only when the conflict cannot be localised or the repair fails validation. Selective repair raised Claude Haiku 4.5's tax-form field accuracy from 30.5% to 97.5% with a stale skill, and most of a stale skill is usually still correct. ### What is memory pollution in self-improving agents? Memory pollution is the build-up of unverified, stale or conflicting rules in what an agent carries forward between runs. Google Cloud's GUARD paper measures it separately from accuracy on PAST-Bench, where methods that rewrote aggressively reached about 90% update correctness while tripling pollution. ### When should an agent create a new skill from memory? Only when the same reusable behaviour has appeared more than once. Perplexity's Brain requires two independent occurrences to create a skill, allows an update from one grounded occurrence, applies one change per skill per run, and caps each user at 10 automatically created skills. \*Sources, all read on 20 November 2026: Akgül et al., \[When Agent Skills Go Stale\](https://arxiv.org/abs/2611.22525) (arXiv, 17 November 2026); Zhang, Kang et al., \[GUARD\](https://arxiv.org/abs/2611.23476) (arXiv, 17 November 2026); Pei et al., \[Safe Personal RSI\](https://arxiv.org/abs/2611.23651) (arXiv, 18 November 2026); Perplexity Engineering, \[Self-improving skills\](https://www.perplexity.ai/hub/blog/self-improving-skills) (18 November 2026).\*

---

Source: [https://www.maximem.ai/blog/agent-skills-go-stale](https://www.maximem.ai/blog/agent-skills-go-stale)
