Skills Fail From Confusability, Not From Count
Fifty near-identical skills are harder to retrieve from than two hundred well-separated ones, and the number everybody asks about is the wrong number.
The controlled version of this comes from Demystifying Agent Skills, one of the thirty-five papers behind The Agent Skills Evidence Base, which built candidate pools of matched size around each task and varied only how alike the distractors were. At a pool of a hundred skills, top-1 precision runs at 53.4% on semantically similar pools against 84.1% on random pools and 93.2% on dissimilar ones, and that contrast is an offline embedding diagnostic whose output was never passed to execution. Pool size is held constant across the three conditions. Similarity is doing all of the work.
Two things need saying before the argument goes anywhere, and they weaken it. The first is that the dramatic collapse in the same study, execution-time actual-use precision falling from 29.6% to 3.3% as pools grow from five skills to a hundred, did not carry a matching collapse in outcomes: downstream success moved only 36.4% to 39.3% over the same sweep. The second is the authors' own summary of why, which is that "exact ground-truth skill invocation is neither sufficient nor strictly necessary for success." Agents inspect several candidates, use parts of the wrong one, and finish the task anyway. Retrieval precision and task success are loosely coupled at best in this data, and any piece that presents a precision curve as a performance curve is overselling.
Worth reading it the other way round. The layer everybody reports on, task success, held roughly flat while the layer underneath it degraded by an order of magnitude, which means the degradation was invisible to the only metric most teams collect. That is the finding. It is a statement about instrumentation rather than about outcomes, and it is exactly the kind of thing that stays quiet until the task distribution shifts and the slack runs out. What a correctly retrieved skill then does to the implementation is a different failure entirely, covered in Getting Retrieval Right Does Not Save You.
What confusability means in a real library
The authors state the mechanism plainly: "pool size contributes to the difficulty, but semantic confusability is the more important stressor for identifying the correct procedural artifact." Their offline diagnostics degrade gently with size and steeply with similarity, and the gap between those two curves is the practical content of the result.
Every library that grows organically grows toward confusability rather than away from it. Teams do not add a hundred unrelated skills; they add the fourth variant of a deployment procedure, the third document-conversion helper, a second PDF extractor that handles scanned pages, a copy of an existing skill with two lines changed for a different client. The distribution of a mature internal library looks like the similar-pool condition and nothing like the random-pool condition, which is the condition most retrieval intuitions were built on.
So the governing question for library design is not how many skills an agent can hold. It is how many mutually confusable skills sit in any one neighbourhood of the embedding space, and whether the descriptions that separate them say anything a retriever can act on. Two hundred skills spread across twenty distinct capabilities is a straightforward retrieval problem. Fifty skills that are all some flavour of "process this document" is a hard one, and it stays hard as the library shrinks.
That claim now has independent replication at fixed bundle size. From Relevance to Execution Utility gave an agent bundles of identical size and varied only composition, and a gold bundle succeeded on 13 of 20 tasks against 2 of 20 for a half-mixed one on the first benchmark, 17 against 3 on a second and 20 against 6 on a third, at twenty queries per cell on a single run, with the authors noting the outcomes are caused by composition rather than count. Same size, different neighbours, and success moves by a factor of five or more. Those bundles were constructed rather than retrieved.
What a production library actually scores
Comparative Approaches to Agent Retrieval over Large Skill Libraries measured a real one at Praetorian, and the headline is a hybrid BM25 and dense ranker reaching hit@5 of 0.735 with a 95% half-width of ±8.0 points at n = 117 queries, on a library of 639 skills at the point retrieval was measured, later 690 at graph-build time, against an 875-entry retrieval catalogue. Top-1 is 0.504, which is to say the ranker puts the right skill first about half the time on a library most teams would consider small. Roughly a quarter of realistic queries are unserved at rank five by the production ranker, which is the authors' own reading of their number, and the honest framing of the whole paper is a negative result: a typed knowledge graph layered on top of that ranker did not close the gap.
The stratum that matters most is the one that degrades furthest. On indirect queries, where the user describes a symptom rather than naming the capability, hybrid hit@5 falls to 0.628. Direct queries are the easy case and the rare one; a user who can name the skill they need barely needs retrieval at all. The queries that justify having a library in the first place are the ones the ranker handles worst.
Both of these are single-organisation numbers on a single embedding model, and the authors say so, disclaiming any claim about other libraries, scales or backends. The ranker reads only the one-line name and description from each file's frontmatter, never the body, which is a deliberate choice that a sibling effort supported and contemporaneous work at larger scale contradicts. Treat the shape rather than the value.
Why skill files are harder to retrieve than ordinary documents
There is a structural reason the numbers look worse than general retrieval benchmarks would lead you to expect, and SkillSight named it. Skill documents share heavy boilerplate: capability descriptions, invocation interfaces, execution conditions, usage instructions, all phrased in the same register because they were written against the same specification. That shared background leaks into dense relevance scores, so a candidate can score highly by matching the template while mismatching the operation, the target and the constraint.
The measurement is the burial rate. On a skill-retrieval benchmark, roughly 1.24% of the corpus is ranked above the gold skill on average, against 0.05% to 0.42% on standard text-retrieval benchmarks, using one embedding model. At tens of thousands of candidates that is a large absolute number of hard negatives sitting between the query and the right answer, and it is a property of how skill files are written rather than of how many there are.
That is an unusually actionable diagnosis. The fix is not a bigger retriever. It is either calibrating the shared background out of the scoring, as that paper does without any training, or writing descriptions that carry the discriminating detail instead of the ceremonial detail. A frontmatter line that says what makes this skill different from its four neighbours is worth more than a paragraph explaining what a skill is.
The evaluation trap, which is the transferable part
The most reusable result in this literature costs nothing to apply. The same ranker on the same catalogue scored hit@5 of 0.946 on a 37-query set written by the author of the system, against 0.735 on a non-echo set built by a separate team with an integrity test that rejects queries echoing a skill description.
Take the size of that gap seriously. Only the queries changed between the two conditions, since the ranker, the catalogue and the metric were held identical, so the entire difference is a property of how the questions were written. An author-written evaluation would have reported near-perfect retrieval and hidden the headroom entirely. It also produced a false equivalence: on the author-written set a lexical baseline and the production ranker score identically, while on the non-echo set they separate by a wide margin. An evaluation set that flatters your system will also mislead you about which system to keep, and that second failure is the expensive one, because it survives into the next architecture decision.
Anyone evaluating a skill library can reproduce the trap in an afternoon by writing the queries themselves and then having someone who has never seen the library write another set from user transcripts. If the two sets agree, the descriptions genuinely carry discriminating information. If they diverge, every retrieval number reported so far has been measuring vocabulary overlap between two documents written by the same person.
Gate on what can run, not on what is relevant
There is a second axis that relevance scoring cannot see, now measured on a benchmark as well as in production, and the benchmark side is the more surprising half.
The same study asked whether retrieval relevance predicts anything about execution, and it does not: relevance predicting whether executing the retrieved bundle earns any verifier reward at all comes in at AUROC 0.435 and 0.418, with Spearman −0.089 and −0.131, over 288 single-run rollouts across 72 tasks, and a 10,000-shuffle permutation test places the observed value inside the null interval of 0.427 to 0.572, so the honest claim is no evidence of signal rather than anti-correlation. Used as a gate on what to run, relevance scores 0.447 against random's 0.500. Relevance is silent about execution, not opposed to it.
In Wix's customer-care assistant, a deterministic gate sits between semantic matching and the model's selection decision, evaluating each candidate skill's own hard exit conditions against authoritative account state and dropping the ones that cannot complete. Over 756,641 user messages, that gate blocked 59.4% of topically relevant skill-message pairs as not executable for that account, across ten skills in one topic family inside a production library whose total size the authors deliberately do not disclose.
Nearly six in ten of the candidates a relevance-ranked retriever would have offered were impossible to execute. Not unhelpful, not lower-ranked: impossible, because the account had no domain connected, or the subscription state made the action unavailable. No amount of embedding quality surfaces account state, since whether a subscription is active is a fact about the caller, not about the text of the skill.
Executability more broadly is partly learnable, and the two families of gate that follow are not interchangeable. A learned utility gate in the same study, trained to predict whether a rollout earns reward, reaches AUROC 0.717 under full-information replay, and it buys cost rather than correctness: it discards 39% of available reward to save 68% of calls, with two-sided error, and its discrimination falls to 0.557 ± 0.077 under realistic deployment feedback, where rewards are only observed for the calls you actually make. A deterministic precondition gate of the Wix kind is sound by construction, because it reads an authoritative fact and drops only candidates that provably cannot complete. One is not a replacement for the other.
The combined effect on context is 90.5% relative to exposing all ten skills to every message, of which the gate contributes 59.1% and semantic matching removes 76.9% of the all-message baseline first, so the two stages are sequential and their savings are not additive. The counterfactual is the part that matters for correctness rather than cost: replaying conversations with the gate removed, the model selected a skill that production had blocked in a risk-enriched cohort, which the authors are careful to describe as a conditional rate that cannot be extrapolated to all traffic. What the study does not do is vary library size at all; the menu is fixed at ten skills, and "at scale" in that paper refers to traffic volume rather than skill count. The cost side of loading decisions is a separate argument with separate evidence, in Skills Do Not Save You Tokens.
What to do with a library that is getting bigger
Stop counting. A skill count tells you nothing about whether retrieval will work, because the same count can describe a trivially separable library or a pathologically confusable one, and the evidence above says separation is the variable that moves precision.
Audit for neighbourhoods instead. Embed every description in your library, cluster them, and read the clusters: any group of skills whose descriptions a human cannot tell apart in one line is a group a retriever cannot tell apart either, and the repair is to merge them into one skill with branching guidance or to rewrite the descriptions around what distinguishes them. Then check what fraction of your candidates could even run for the caller before they reach the model, because a filter on executable state removes candidates that no relevance model will ever demote. Build the query set from user transcripts rather than from your own head, and measure the indirect queries separately, since the average will hide them.
None of that requires a bigger library or a smaller one, as far as retrieval is concerned. Count exerts its own pressure downstream of retrieval: in When Self-Evolution Backfires, a self-evolving agent's pass@1 peaks at 62% with 105 skills in context and falls to 50% at 179, against 46% for three hand-written skills, which is a context-contamination effect with no retrieval step involved, on one evolution run per configuration. So count is the wrong variable for retrieval and a live one for context. Retrieval requires the library to be shaped so that the boundaries between its entries survive being compressed into a single embedding, which is a writing problem wearing a retrieval problem's clothes. The teams that will scale a skill library furthest are the ones treating each description as an interface contract rather than as documentation.
Every figure here is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, with the full claim register published in the open. We build Synap, a memory layer for agents, so read the argument with that in mind and check the register.