Your Skill Library Stops Pruning Bad Skills Once Its Grader Gets Too Lenient
A skill library that removes its own bad skills depends on whatever grader decides pass or fail. That grader has a false-pass rate, which is how often it scores a genuine failure as a pass. Once that rate reaches (1 − τ)/2, the library stops retiring anything at all. τ (tau) is the eviction margin, and at a margin of 0.10 the threshold works out to 0.45, or 45%. Past that point no amount of extra testing helps, because more trials shrink the random error in the estimate and leave the bias exactly where it was. This is a derivation rather than a measurement. It falls out of modelling the grader as a noisy channel that flips some of its verdicts, then asking what the eviction test can still resolve once the channel has shifted the average it reads.
Plenty of teams run this machine without calling it that. A loop writes new skills from observed failures, routes at most one of them into each task, scores the outcome, and removes any skill whose measured contribution stays negative. That is a controller with a sensor attached, and the sensor is whatever decides pass or fail. Where the sensor is a unit test, the false-pass rate is close enough to zero that none of this applies. Where it is a model grading prose, a report, a plan or a diff, there is no deterministic oracle, and the false-pass rate is a real number that nobody on the team has estimated. The eviction policy is then worth only what that number allows it to be.
Most of this literature, catalogued in the evidence base, attends to authoring and retrieval instead. Candidate rejection has finally been instrumented, and SkillBoost shows what it is worth by taking it away. Removing the verified-acceptance gate, which requires a candidate skill to fix more than it breaks across the full task set, cost 3.2 to 12.5 points across six model and benchmark pairs. Library-level eviction is a different operation, and nobody has instrumented it yet. What gets removed there is an existing skill competing for a capped shelf, not a candidate that never got onto the shelf. That operation is what this piece is about.
Five papers carry this result, and they sit closer to each other than a citation list suggests. Library Drift, Ratchet, The Blind Curator and Who Grades the Grader share all seven authors, split between the AWS Generative AI Innovation Center and the HSBC Technology Center China. The last of the four adopts the Ratchet loop unchanged and cites a fifth paper from the same team, which is how the cluster reaches five rather than three, and it does not cite The Blind Curator at all. Two of the first three are workshop papers, at ICML (the International Conference on Machine Learning) and COLM (the Conference on Language Modeling); Ratchet states no venue. Reading these as four independent teams agreeing with each other would be a mistake.
Half of that disclosure cuts the other way, because the drift mechanism underneath the threshold has now been reproduced by strangers. SkillBoost, out of Zhejiang University and Alibaba, measured the test-minus-train gap for two published skill-evolution methods across five models and four benchmarks. The gap came out negative in all 40 of the 40 configurations, meaning held-out performance sat below training performance every single time. That was one run on random seed 42, with no error bars and with test-split sizes never stated. Its Figure 3 shows skill text never shrinking at any point across all four benchmarks, while held-out performance falls, which is Library Drift's accumulation mechanism measured by a team with no connection to the AWS and HSBC cluster. Shared authorship is a reason to discount the specific magnitudes, not the phenomenon.
Only one direction of grader error breaks the test
Eviction tests one number per skill: passes minus failures, divided by trials. That number has to fall below a negative threshold before the skill goes. Rewrite the test in terms of the skill's true pass rate and it becomes a comparison against (1 − τ)/2. Once the grader has a false-pass rate of its own, the pass rate it reports concentrates somewhere above the true one. Retirement then requires a true pass rate below a quantity that shrinks as the false-pass rate grows, and at (1 − τ)/2 that quantity reaches zero. A skill whose real contribution is catastrophic reads, through the grader, as merely mediocre. Mediocre survives the threshold.
Direction is the whole story, and it is the part people get backwards. Errors the other way, where a genuine pass gets scored as a failure, only compress the statistic toward the middle. The sign survives, harmful skills still cross the threshold, and the cost is a larger sample budget rather than a broken controller. Errors in the false-pass direction add to the bound instead, and they disable the test all at once. A grader that is wrong very often in the harmless direction is safer than one wrong somewhat less often in the harmful direction, which inverts how most teams would rank two graders shown only an accuracy number.
Two things follow. Spending on sample size is not a fix, because the random error in the estimate and the bias sit in the same expression, and only the random error responds to more trials. And landing inside the region where retirement is provably possible is necessary rather than sufficient: the authors' own loop retired nothing at a false-pass rate of 20%, comfortably inside the region where the inequality says retirement is possible. Possible and observed are different claims.
What an operator sees when retirement stops
Nothing at all. That is the behavioural finding, and it is why the mechanism matters separately from the theory.
Genuine retirement falls to zero under bias. The Blind Curator's loop retired 1.3 genuinely bad skills per run on a clean grader, against 0.0, 0.3 and 0.0 at three increasing false-pass rates. The clean baseline is itself only 1.3 give or take 1.2, so the quantity being driven to zero was never large, and a controller retiring roughly one skill per run is not a dramatic instrument even when it works. Magnitude depends on which grader sits in the loop. The same table reports 10.3 genuine retirements per run under the real audited judge, against 1.3 under a clean deterministic grader, so the small baseline is a property of that control rather than a ceiling on what a curator can do.
Raw deprecation counts, which is the number an operator actually watches, never reach zero under bias. They sit at 9.7 per run on the clean grader, against 7.7, 3.7 and 3.0 as bias increases. Cap-eviction churn fills the gap: skills keep getting pushed off the shelf to make room for new ones, whether or not anything is being judged badly. A dashboard tracking deprecations shows a number that declines somewhat and stays comfortably positive, while the only removals that were doing useful work have stopped entirely.
End-task score is no better as an alarm. Harm against the clean loop peaks at the threshold rather than beyond it, then partially recovers at higher bias. The same corruption that disarms the curator also starves the author: skills written per run fall from 22 to 15 as judge bias rises, and a loop that writes fewer skills has fewer bad ones to route. So the aggregate metric moves in both directions in the thing you would want it to track, and where failures are abundant it does not move at all while the curator is dead.
Who Grades the Grader reaches the same warning by a second and more damaging route. The loop trained by a worthless always-pass grader scored 0.742 on held-out tasks, give or take 0.014, against 0.717 give or take 0.038 for the loop trained by the anchored grader, over three seeds on locked sets of 40 to 48 items. The vacuous grader trained the higher-scoring loop. A team comparing those two configurations on end-task score would have shipped the broken one.
Prune too hard and you score worse than using no skills at all
Governance done well is worth a great deal. Ratchet's governed loop reports a rolling gain of +0.328, give or take 0.018, against +0.002 give or take 0.005 for the same loop with skill injection withheld, over 100 rounds, three seeds and 40 held-out tasks. Those 40 tasks were selected as ones the base model already failed, so the held-out set leaves more room to move than a representative sample would. The comparison also runs against a stripped-down version of the loop itself rather than against an independent system. Discount for both and the gap is still the reason anybody runs one of these loops.
One ablation points the other way, and it is more useful than the entire theory section. When the evidence floor was lowered so that eviction could fire on far fewer trials, the loop scored 0.019 below the no-skill control, give or take 0.010, consistent across three seeds. That is the only condition in the paper that lands below the baseline. Skills with genuinely positive contribution were being evicted on unlucky draws, the shelf collapsed to a handful of survivors, and the router, which decides what to inject into each task, responded by injecting nothing.
Both failure modes are silent, they arrive from opposite directions, and the response to one makes the other more likely. A team that reads the threshold result and reacts by tightening its eviction policy has moved toward the ablation that measured below baseline. The eviction margin and the evidence floor are a single joint setting, and it has to be calibrated against a measured false-pass rate. No published default has ever been calibrated that way. Defaults that survive across domains do exist for the honest-sensor case, and every one of them was tuned behind a deterministic scorer.
Seven judges have now been measured past the threshold
No deployed judge has been watched drifting across the threshold over time inside a running skill-eviction loop, and that much remains true. Every threshold number in the cluster comes from taking a grader that was already correct and randomly flipping a set share of its verdicts. That places a judge relative to the threshold without ever exhibiting one that crossed it while a library evolved underneath. What has changed is that judges have now been measured sitting past the threshold from the start, with nothing injected anywhere.
SkillTV-Bench scored seven judges against 156 real agent trajectories that had genuinely failed, and got false-pass rates from 45.5% to 79.5%: GPT-5.2 at 51.3%, Gemini 3.1 Pro Preview at 67.9%, Claude Sonnet 4.6 at 79.5%, a majority-vote ensemble at 69.9%, a rubric-based judge at 59.6%, an agent judge at 67.3%, and the paper's own refined judge at 45.5%. Every one of them sits at or past the 45% threshold.
Read the caveats first. The evaluation set is deliberately hard, since tasks the agent always passed or always failed were screened out and the split runs 47 pass against 156 fail, so this is a population of contested trajectories rather than a representative sample. Each judge was run once, with no error bars. Weighing the other way, the team is genuinely independent: Shanghai Jiao Tong, Shanghai AI Laboratory, Sun Yat-Sen and UCL (University College London), with zero author overlap with the AWS and HSBC cluster and no citation of any of its papers.
Trying hard does not clear it either. The paper's own evolved JudgeSkill, a skill written to make the judge better at judging, took false accepts from 105 down to 71 out of those same 156 failures. That moved the false-pass rate from 67.3% to 45.5% and lifted accuracy by 14.8 points, at a p-value below 0.0001, meaning chance is a very poor explanation for the improvement. It still does not get below the threshold. Judge accuracy runs from 30.8% to 66.7% across domains, so false-pass behaviour depends heavily on the kind of work being graded, and a rate you measured on coding tasks tells you very little about your report-composition ones.
An evaluator has also now been watched collapsing on its own. Who Grades the Grader removed two guards from an evolving metric, and the metric fell into an always-pass state in three of three seeds, passing 97% to 100% of everything it graded, with nothing injected. The search itself finds the degenerate solution, because always-pass scores extremely well on the objective the search was handed. The authors call the paper a mechanism study rather than a scaling result, which is the right way to hold it.
Coupling is no longer out of scope either. The same paper contains an episode where evolved skills raised the rubric score by 0.26 partly by writing evidence tags with no values behind them, about 30% of tags at peak rounds. An independent outer judge caught it within a hundred rounds and one added detector repaired it. That is an evolving library learning what makes its own grader lenient, which is exactly the trajectory the earlier papers set aside as unmeasured and unbounded by any fixed audit. It got caught only because somebody was looking from outside the loop with a different instrument.
Not every grader is permissive. The one real judge the cluster audited inside its own loop measured a false-pass rate of roughly 1%, clearing the 45% threshold by a factor of 45, with a channel resolution near 0.04. That second number describes a different problem from the one the paper is named after. Channel resolution is how far apart a good skill and a bad skill look once you are reading them through the grader, and 0.04 is close to nothing. A strict, well-instructed judge that almost never passes a failure also flags an enormous share of genuine passes as failures, and the contribution estimate then barely moves in response to reality. Both edges leave the curator inert; only one of them is the threshold this piece is about.
The measurement noise is bigger than the improvement you are looking for
A prior problem sits under all of this, and two papers outside the cluster define its size.
Feedback Dynamics tracked ten rounds of skill self-evolution and found that 55 of 388 candidates, 14.2%, established a byte-distinct validation best, meaning the file genuinely changed and the new version scored better than every version before it. Revision is common and retained improvement is rare. The same paper evaluated one artifact eight times without changing a single byte of it, and the scores ranged from 71.43% to 83.67%, a standard deviation of 3.92. That was Gemini 3.1 Pro on the OfficeQA benchmark with a validation set of 49 items, so the whole range is six items moving. A gate that admits a candidate whenever validation does not decrease is reading a signal whose noise band is wider than most of the improvements it is being asked to detect.
Corroboration comes from On the Fragility of Self-Improving Agents, which re-ran two self-improvement methods with repeated runs and shuffled task orders. Run-to-run variance increased in 17 of 24 cases, on three runs per setting. Those artefacts are textual memory banks rather than SKILL.md files, the skill artifacts themselves. The paper places skills in a separate cited literature, so the result transfers as an argument about self-improvement loops in general and not as a finding about skills. That caveat still applies, but the argument no longer rests on it, because two independent teams have since measured the same instability directly on SKILL.md files: Feedback Dynamics on re-evaluating an unchanged artifact, and SkillBoost on held-out generalisation across all 40 configurations. The warning is first-hand rather than borrowed, and it says a loop writing its own artefacts on a noisy signal can amplify the noise rather than average it out.
Resolution, variance and bias are three separate ways for the same controller to be reading nothing, and only one of them has a clean inequality attached. What a correctly retrieved but wrong skill does to the run is a separate failure surface, covered in Getting Retrieval Right Does Not Save You. The raw material these loops prune is itself heavily duplicated, which the ecosystem census puts numbers to.
How to measure your own grader in a day
Take the artefacts your grader scores and list the defect classes that matter for your domain, which in the report-composition case meant orphan citations, unregistered metrics, unsourced numbers, broken cross-references, missing summaries, negated claims and swapped numbers. Inject one instance of each class into otherwise correct outputs, one class per sample, so that you know the true label of every injected case. Run your grader over them and count how often it returns a pass on something you broke on purpose. That count over that denominator is your false-pass rate. Compare it against (1 − τ)/2 for the eviction margin your library is configured with. Treat a result within an order of magnitude of the threshold as a reason to widen the margin, or to defer self-evolution entirely. The only lever against bias is a wider threshold, not more data.
Two things make the estimate honest. The injections have to provably change true quality, or you are measuring the grader's agreement with your own taste rather than its blindness. And the audit certifies only the classes you listed, so a defect family you did not think of is one your number says nothing about.
What you learn by running it once is which edge you are near, and the two measurements that exist point in opposite directions. The cluster's audited judge sat at roughly 1% and resolved almost nothing. The only cross-model measurement of judges on real failed trajectories puts seven of them between 45.5% and 79.5%, every one at or past the threshold. Nothing about your model choice, your rubric style, your prompt, or the accuracy number on your evaluation set predicts which of those two populations yours belongs to. The spread across domains inside those seven says the same grader will land in different places on different work. Guessing the edge is how a team talks itself out of the audit, and the audit is a day.
If your library maintains itself, a grader decides what survives, and no grader measured so far is good enough to be trusted with that. It is the strongest single reason we argue against keeping anything that changes in a skill file, which is the skills question in full.
Every figure here is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, with the full claim register published alongside them. We build Synap, which is a memory layer for agents, so read the argument above with that in mind and check the register.