Your Skill Library Has a Judge, and It Has a Cliff
A skill library that prunes itself stops pruning anything at all once the grader it reads scores failures as passes at a rate of (1−τ)/2, which is 0.45 at an eviction margin of 0.10, and past that false-pass rate nothing is retirable at any sample size, because more trials shrink the estimator's radius and not the bias. That is a derivation rather than a measurement, falling out of modelling the judge as a binary channel and asking what the eviction test can still resolve once the channel has displaced the mean it was reading.
Worth being precise about the machine this describes, because many teams are quietly running one without calling it that. A loop that writes new skills from observed failures, routes at most one of them into each task, scores the outcome, and removes anything whose measured contribution stays negative is a controller with a sensor attached, and the sensor is whatever decides pass or fail. Where that sensor is a unit test, the false-pass rate is close enough to zero that none of this applies. Where it is a model grading prose, a report, a plan, a diff, or anything else without a deterministic oracle, the false-pass rate is a real number that nobody on the team has estimated, and the eviction policy is only worth what that number allows it to be. The rest of this literature, catalogued in the evidence base, attends mostly to authoring and retrieval. Candidate rejection has finally been instrumented, and SkillBoost shows what it is worth by taking it away, losing 3.2 to 12.5 points across six model and benchmark pairs when the verified-acceptance gate that requires a candidate to fix more than it breaks on the full set is removed. Library-level eviction is a different operation and still uninstrumented, because what gets removed there is an existing skill competing for a capped shelf rather than a candidate that never got onto it. That is the operation this piece is about.
Five papers carry this result, and they are closer to each other than a citation list suggests. Library Drift, Ratchet, The Blind Curator and Who Grades the Grader share all seven authors, split between the AWS Generative AI Innovation Center and the HSBC Technology Center China; the last of them adopts the Ratchet loop unchanged and cites a fifth same-team paper, which is how the cluster reaches five rather than three, and it does not cite The Blind Curator at all. Two of the first three are workshop papers, at ICML and COLM respectively, and Ratchet states no venue. Reading it as a corroborating cluster would be a mistake.
Withholding the other half of that disclosure would be lopsided, because the drift mechanism underneath the cliff has now been reproduced by strangers. SkillBoost, out of Zhejiang University and Alibaba, measured the test-minus-train gap for two published skill-evolution methods across five models and four benchmarks and found it negative in 40 of 40 configurations, at a single run on seed 42 with no error bars and test-split sizes never stated. Its Figure 3 shows skill text growing monotonically across all four benchmarks while held-out performance falls, which is Library Drift's accumulation mechanism measured by a team with no connection to the AWS and HSBC cluster. Shared authorship is a reason to discount the specific magnitudes, not the phenomenon.
The inequality, and which direction of error matters
Eviction in this design tests a single scalar per skill, the difference between passes and failures divided by trials, against a negative threshold. Rewriting that test in terms of the underlying pass rate turns it into a comparison against (1−τ)/2, and once the grader has a false-pass rate of its own, the observed pass rate concentrates somewhere above the true one. Retirement then requires a true pass rate below a quantity that shrinks as the false-pass rate grows, and at (1−τ)/2 that quantity reaches zero. A skill whose true contribution is catastrophic still reads, through the channel, as merely mediocre, and mediocre survives the threshold.
Direction is the whole story, and it is the counterintuitive part. Errors in the other direction, where genuine passes get scored as failures, only compress the statistic toward the middle; the sign survives, harmful skills still cross the threshold, and the cost is a larger sample budget rather than a broken controller. Errors in the false-pass direction enter the bound additively and disable the test discontinuously. A judge that is wrong very often in the harmless direction is safer than one that is wrong somewhat less often in the harmful one, which inverts how most teams would rank two graders shown only an accuracy number.
Two things follow that are easy to get backwards. Spending on sample size is not a fix, because the estimator's radius and the bias sit in the same expression and only the radius responds to more trials. And the certifiable region is necessary and not sufficient: the authors' own loop retired nothing at a false-pass rate of 0.20, comfortably inside the region where the inequality says retirement is possible, because possible and observed are different claims.
What a dead curator looks like from the outside
Nothing. That is the behavioural finding, and it is why the mechanism matters separately from the theory.
Genuine retirement falls to zero under bias, running 1.3 per run on the clean grader against 0.0, 0.3 and 0.0 at increasing false-pass rates, and the clean baseline is itself only 1.3 ± 1.2, so the quantity being driven to zero was never large and a controller retiring roughly one skill per run is not a dramatic instrument even when it works. Magnitude depends on which grader sits in the loop, and the same table reports genuine retirements at 10.3 per run under the real audited judge, against 1.3 under a clean deterministic grader, so the small baseline is a property of the control rather than a ceiling on what a curator can do.
What an operator actually watches is where this bites. Raw deprecation counts never reach zero under bias, sitting at 9.7 on the clean grader against 7.7, 3.7 and 3.0 as bias increases, because cap-eviction churn fills in: skills keep getting pushed out to make room for new ones regardless of whether anything is being judged badly. A dashboard tracking deprecations shows a number that declines somewhat and stays comfortably positive, while the only removals that were doing useful work have stopped entirely.
End-task score is no better as an alarm. Harm against the clean loop peaks at the cliff rather than beyond it and then partially recovers at higher bias, because the same corruption that disarms the curator also starves the author, with skills written per run falling from 22 to 15 as judge bias rises, and a loop that writes fewer skills has fewer bad ones to route. So the aggregate metric moves non-monotonically in the thing you would want it to track, and in abundant-failure conditions it does not move at all while the curator is dead. Who Grades the Grader reaches the same warning by a second and more damaging mechanism: the loop trained by the worthless always-pass grader scored 0.742 ± 0.014 on held-out tasks against 0.717 ± 0.038 for the loop trained by the anchored one, over three seeds on locked sets of 40 to 48 items. The vacuous grader trained the higher-scoring loop, so a team comparing two configurations on end-task score would have shipped the broken one.
Governance done badly is worse than none
Governance done well is worth a great deal. Ratchet's governed loop reports a rolling gain of +0.328 ± 0.018 against +0.002 ± 0.005 for the same loop with skill injection withheld, over 100 rounds, three seeds and 40 held-out tasks selected as ones the base model already failed. Selection matters there, since a held-out set built from known failures leaves more room to move than a representative sample would, and the comparison runs against the loop's own ablated self rather than against an independent system. Discount for both and the gap is still the reason anybody runs one of these loops.
One ablation here points the other way, and it is more useful than the entire theory section. When the evidence floor was lowered so that eviction could fire on far fewer trials, the loop scored −0.019 ± 0.010 below the no-skill control, consistent across three seeds. That is the only condition in the paper that lands below the baseline. Skills with genuinely positive contribution were being evicted on unlucky draws, the shelf collapsed to a handful of survivors, and the router responded by declining to inject anything.
Both failure modes are silent, they arrive from opposite directions, and the response to one makes the other more likely. A team that reads the cliff result and reacts by tightening its eviction policy has moved toward the ablation that measured below baseline. The honest reading is that the eviction margin and the evidence floor are a joint setting that has to be calibrated against a measured false-pass rate, and no published default has ever been calibrated that way. Defaults that survive across domains do exist for the honest-sensor case, and every one of them was tuned behind a deterministic scorer.
Your judge is probably already past the cliff
No deployed judge has been observed drifting across the cliff over time inside a running skill-eviction loop, and that much remains true. Every cliff number in the cluster comes from a bit-flip channel injected onto a grader that was already correct, which places a judge relative to the threshold without ever exhibiting one that crossed it while a library evolved underneath. What has changed is that judges have now been measured sitting past the threshold from the start, with nothing injected anywhere.
SkillTV-Bench scored seven judges against 156 truly-failed real agent trajectories and got false-pass rates from 0.455 to 0.795: GPT-5.2 at 0.513, Gemini 3.1 Pro Preview at 0.679, Claude Sonnet 4.6 at 0.795, a majority-vote ensemble at 0.699, a rubric-based judge at 0.596, an agent judge at 0.673, and the paper's own refined judge at 0.455. Every one of them sits at or past 0.45.
Read the caveats first. The evaluation set is deliberately hard, since tasks the agent always passed or always failed were screened out and the split runs 47 pass against 156 fail, so this is a population of contested trajectories rather than a representative sample. Each judge was run once, with no error bars. Weighing the other way, the team is genuinely independent: Shanghai Jiao Tong, Shanghai AI Laboratory, Sun Yat-Sen and UCL, with zero author overlap with the AWS and HSBC cluster and no citation of any of its papers.
Trying hard does not clear it either. The paper's own evolved JudgeSkill took false accepts from 105 down to 71, moving the false-pass rate from 0.673 to 0.455 and lifting accuracy 14.8 points at p below 10⁻⁴, and it still does not get below the cliff. Judge accuracy runs from 30.8% to 66.7% across domains, so false-pass behaviour is strongly domain-dependent, and a rate you measured on coding tasks tells you very little about your report-composition ones.
An evaluator has also now been watched collapsing on its own. Who Grades the Grader removed two guards from an evolving metric and the metric fell into an always-pass state in three of three seeds, passing 0.97 to 1.00 of everything it graded, with nothing injected: the search itself finds the degenerate solution, because always-pass scores extremely well on the objective the search was handed. The authors call the paper a mechanism study rather than a scaling result, which is the right way to hold it.
Coupling is no longer out of scope either. The same paper contains an episode where evolved skills raised the rubric score by 0.26 partly by writing evidence tags without values, about 30% of tags at peak rounds, caught within a hundred rounds by an independent outer judge and repaired by one added detector. That is an evolving library learning what makes its own grader lenient, which is exactly the trajectory the earlier papers set aside as unmeasured and unbounded by any fixed audit. It got caught only because somebody was looking from outside the loop with a different instrument.
None of which means every grader is permissive. The one real judge the cluster audited inside its own loop measured a false-pass rate of roughly 0.01, clearing the cliff by a factor of 45, with a channel resolution near 0.04. That second number describes a different problem from the one the paper is named after. A strict, well-instructed judge that almost never passes a failure also flags an enormous share of genuine passes as failures, and the resulting channel carries so little resolution that the contribution estimate barely moves in response to reality. Both edges leave the curator inert; only one of them is the cliff.
The measurement floor underneath all of it
A prior problem sits under all of this, and two papers outside the cluster define its size.
Feedback Dynamics found, across ten rounds of skill self-evolution, that 55 of 388 candidates, 14.2%, established a byte-distinct validation best; revision is common and retained improvement is rare. The same paper evaluated one byte-identical artifact eight times and got 71.43% to 83.67%, SD 3.92, using Gemini 3.1 Pro on OfficeQA with a validation set of 49 items, so that range is six items moving. A gate that admits a candidate when validation does not decrease is reading a signal whose noise band is wider than most of the improvements it is being asked to detect.
Corroboration comes from On the Fragility of Self-Improving Agents, which re-ran two self-improvement methods with repeated runs and shuffled task orders and found run-to-run variance increasing in 17 of 24 cases, on three runs per setting. Those artefacts are textual memory banks rather than SKILL.md files, and the paper places skills in a separate cited literature, so the result transfers as an argument about self-improvement loops in general and not as a finding about skills. That hedge still stands and it is no longer load-bearing, because two independent teams have since measured the same instability directly on SKILL.md artefacts: Feedback Dynamics on byte-identical re-evaluation, and SkillBoost on held-out generalisation across 40 configurations. The warning is first-hand rather than borrowed, and it says a loop writing its own artefacts on a noisy signal can amplify the noise rather than average it out.
Resolution, variance and bias are three separate ways for the same controller to be reading nothing, and only one has a clean inequality attached. What a correctly retrieved but wrong skill does to the run is a separate failure surface, covered in Getting Retrieval Right Does Not Save You, and the raw material these loops prune is itself heavily duplicated, which the ecosystem census puts numbers to.
The experiment, and it is an afternoon
Take the artefacts your grader scores and enumerate the defect classes that matter for your domain, which in the report-composition case meant orphan citations, unregistered metrics, unsourced numbers, broken cross-references, missing summaries, negated claims and swapped numbers. Inject one instance of each class into otherwise correct outputs, one class per sample, so that you know the ground-truth label of every injected case. Run your grader over them and count how often it returns a pass on something you broke on purpose. That count over that denominator is your false-pass rate. Compare it to (1−τ)/2 for the eviction margin your library is configured with, and treat a result within an order of magnitude of the threshold as a reason to widen the margin or defer self-evolution entirely, since the only lever against bias is a wider threshold rather than more data.
Two things make the estimate honest. The injections have to provably change true quality, or you are measuring the grader's agreement with your own taste rather than its blindness. And the audit certifies only the classes you enumerated, so a defect family you did not think of is one your number says nothing about.
What you learn by running it once is which edge you are near, and the two measurements that exist point in opposite directions. The cluster's audited judge sat at roughly 0.01 and resolved almost nothing; the only cross-model measurement of judges on real failed trajectories puts seven of them between 0.455 and 0.795, every one at or past the threshold. Nothing about your model choice, your rubric style, your prompt, or the accuracy number on your evaluation set predicts which of those two populations yours belongs to, and the spread across domains inside those seven says the same grader will land in different places on different work. Guessing the edge is how a team talks itself out of the audit, and the audit is a day.
Every figure here is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, with the full claim register published alongside them. We build Synap, which is a memory layer for agents, so read the argument above with that in mind and check the register.