Skills Fail Because They Look Alike, Not Because There Are Too Many
Fifty skills that all look alike are harder to search than two hundred that do not. The count everybody asks about is the wrong number to ask about.
A controlled test of that sits in Demystifying Agent Skills, one of the thirty-five papers behind The Agent Skills Evidence Base. The authors built candidate pools of matched size around each task and changed one thing only: how alike the wrong candidates were to the right one. Hold the pool at a hundred skills, and the correct skill is ranked first (top-1 precision) 53.4% of the time when the pool is full of similar skills, 84.1% of the time when the pool is random, and 93.2% of the time when the skills are clearly different from each other. That contrast is an offline diagnostic on the matching step alone, and its output was never passed to a live run. Pool size never moved across the three conditions, so similarity is doing all of the work.
Two things weaken that argument, and both belong here rather than in a footnote. The same study saw a far steeper collapse elsewhere: execution-time actual-use precision, meaning how often the skill the agent actually used was the right one, fell from 29.6% to 3.3% as pools grew from five skills to a hundred. Task success did not follow it down, moving only from 36.4% to 39.3% over the same sweep. The authors say why in their own words, which is that "exact ground-truth skill invocation is neither sufficient nor strictly necessary for success." Agents inspect several candidates, use parts of the wrong one, and finish the task anyway. Retrieval precision and task success are loosely coupled at best in this data, so a precision curve presented as a performance curve is overselling.
Read it the other way round and it gets more interesting. Task success, the layer everybody reports on, held roughly flat at 36.4% to 39.3% while the layer underneath it fell from 29.6% to 3.3%. That degradation was invisible to the only metric most teams collect. It is a statement about instrumentation rather than about outcomes, and problems shaped like that stay quiet until the task distribution shifts and the slack runs out. What a correctly retrieved skill then does to the implementation is a different failure entirely, covered in Getting Retrieval Right Does Not Save You.
Why real libraries drift toward look-alike skills
Their summary of the mechanism is blunt: "pool size contributes to the difficulty, but semantic confusability is the more important stressor for identifying the correct procedural artifact." Semantic confusability here means how easily a retriever mistakes one description for another. Their offline diagnostics degrade gently as the pool grows and steeply as similarity rises, and the gap between those two curves is the practical content of the result.
Every library that grows on its own grows toward look-alike skills rather than away from them. Teams do not add a hundred unrelated skills. They add the fourth variant of a deployment procedure, the third document-conversion helper, a second PDF extractor that handles scanned pages, a copy of an existing skill with two lines changed for a different client. A mature internal library ends up resembling the similar-pool condition and not the random-pool condition, and the random pool is the case most retrieval intuitions were built on.
So the question to ask about a library is not how many skills an agent can hold. It is how many look-alike skills sit next to each other, and whether the words that separate them say anything a retriever can act on. Two hundred skills spread across twenty distinct capabilities is a straightforward retrieval problem. Fifty skills that are all some flavour of "process this document" is a hard one, and it stays hard as the library shrinks.
That claim now has independent replication at fixed bundle size, in From Relevance to Execution Utility. The authors gave an agent bundles of identical size and varied only what went into them. A bundle holding the correct tools, which the paper calls the gold bundle, succeeded on 13 of 20 tasks against 2 of 20 for a half-mixed one on the first benchmark, 17 against 3 on a second, and 20 against 6 on a third, at twenty queries per cell on a single run. The authors note the outcomes are caused by composition rather than count. Same size, different neighbours, and the success counts move several times over. Those bundles were constructed rather than retrieved.
What a real library of 639 skills actually scores
Comparative Approaches to Agent Retrieval over Large Skill Libraries measured one at Praetorian. The strongest configuration they report is a hybrid of keyword matching (BM25) and dense embedding search, and it reaches hit@5 of 0.735, which is the share of queries where the right skill appears somewhere in the top five results. That figure carries a 95% confidence interval, the range the true value probably sits in, of plus or minus 8.0 points, on 117 queries. The library held 639 skills when retrieval was measured and 690 by the time the graph was built, against a retrieval catalogue of 875 entries. Top-1 is 0.504, so the ranker puts the right skill first about half the time on a library most teams would consider small. Roughly a quarter of realistic queries go unserved at rank five, which is the authors' own reading of their number, and the honest framing of the whole paper is a negative result: a typed knowledge graph layered on top of that ranker did not close the gap.
Indirect queries degrade furthest, and they are the ones that matter most. Where the user describes a symptom instead of naming the capability, hybrid hit@5 falls to 0.628, against 0.735 across all queries. Direct queries are the easy case and the rare one; a user who can name the skill they need barely needs retrieval at all. The queries that justify having a library in the first place are the ones the ranker handles worst.
Both numbers come from one organisation and one embedding model, and the authors say so, disclaiming any claim about other libraries, scales or backends. Their ranker reads only the one-line name and description in each file's front matter, the metadata block at the top, and never the body. That is a deliberate choice, supported by a sibling effort and contradicted by contemporaneous work at larger scale. Treat the shape rather than the value.
Why skill files are harder to retrieve than ordinary documents
There is a structural reason these numbers look worse than general retrieval benchmarks would lead you to expect, and SkillSight named it. Skill documents share heavy boilerplate: capability descriptions, invocation interfaces, execution conditions, usage instructions, all phrased in the same register because they were written against the same specification. That shared background leaks into the similarity score a retriever computes, so a candidate can score highly by matching the template while mismatching the operation, the target and the constraint.
Their measurement is the burial rate, meaning how much of the corpus gets ranked above the correct skill. On a skill-retrieval benchmark it sits at roughly 1.24% of the corpus on average, against 0.05% to 0.42% on standard text-retrieval benchmarks, using one embedding model. At tens of thousands of candidates, 1.24% is a large absolute number of near-miss candidates sitting between the query and the right answer, and it is a property of how skill files are written rather than of how many there are.
That diagnosis is unusually actionable. The fix is not a bigger retriever. It is either calibrating the shared background out of the scoring, which that paper does without any training, or writing descriptions that carry the discriminating detail instead of the ceremonial detail. A front-matter line that says what makes this skill different from its four neighbours is worth more than a paragraph explaining what a skill is.
Who writes the test queries changes the result
The most reusable result in this literature costs nothing to apply. The same ranker on the same catalogue scored hit@5 of 0.946 on a 37-query set written by the author of the system, against 0.735 on a non-echo set, which a separate team built with an integrity test that rejects any query echoing a skill description.
Take the size of that gap seriously. Only the queries changed between the two conditions, since the ranker, the catalogue and the metric were held identical, so the entire difference is a property of how the questions were written. An author-written evaluation would have reported near-perfect retrieval and hidden the headroom entirely. It also produced a false equivalence: on the author-written set a plain keyword baseline and the production ranker score identically, while on the non-echo set they separate by a wide margin. An evaluation set that flatters your system will also mislead you about which system to keep, and that second failure is the expensive one, because it survives into the next architecture decision.
Anyone evaluating a skill library can reproduce the trap in an afternoon. Write the queries yourself, then have someone who has never seen the library write another set from user transcripts. If the two sets agree, the descriptions genuinely carry discriminating information. If they diverge, every retrieval number reported so far has been measuring vocabulary overlap between two documents written by the same person.
Relevance does not tell you whether a skill can run
Relevance scoring cannot see a second thing entirely, which is whether the skill it just ranked first is able to run at all. That has now been measured on a benchmark as well as in production, and the benchmark half is the more surprising one.
That same study asked whether retrieval relevance predicts anything about execution, and it does not. Relevance predicting whether executing the retrieved bundle earns any verifier reward at all comes in at 0.435 and 0.418 on AUROC, the area under the curve, where 0.500 is what a coin flip scores. The rank correlation, Spearman, sits at −0.089 and −0.131. Those figures rest on 288 rollouts, meaning single end-to-end attempts at a task, across 72 tasks, one run each. A permutation test that reshuffles the data 10,000 times places the observed value inside the range chance alone produces, 0.427 to 0.572, so the honest claim is no evidence of signal rather than anti-correlation. Used as a gate on what to run, relevance scores 0.447 against random's 0.500. Relevance is silent about execution, not opposed to it.
Wix's customer-care assistant puts a deterministic gate between semantic matching and the model's selection decision. It evaluates each candidate skill's own hard exit conditions against authoritative account state, and drops the ones that cannot complete. Over 756,641 user messages, that gate blocked 59.4% of topically relevant skill-message pairs as not executable for that account. The measurement covers ten skills in one topic family, inside a production library whose total size the authors deliberately do not disclose.
Nearly six in ten of the candidates a relevance-ranked retriever would have offered were impossible to execute. Not unhelpful, not lower-ranked: impossible, because the account had no domain connected, or the subscription state made the action unavailable. No amount of embedding quality surfaces account state, since whether a subscription is active is a fact about the caller, not about the text of the skill.
Executability more broadly is partly learnable, and the two families of gate are not interchangeable. A learned utility gate in the same study, trained to predict whether a rollout earns reward, reaches 0.717 on AUROC under full-information replay, where every outcome is already known. It buys cost rather than correctness: it discards 39% of available reward to save 68% of calls, with error in both directions, and its discrimination falls to 0.557, give or take 0.077, under realistic deployment feedback, where rewards are only observed for the calls you actually make. A deterministic precondition gate of the Wix kind is sound by construction, because it reads an authoritative fact and drops only candidates that provably cannot complete. One is not a replacement for the other.
Context saving comes to 90.5% combined, measured against exposing all ten skills to every message. Semantic matching removes 76.9% of that baseline first, and the gate then removes 59.1% of what is left, so the two stages run in sequence and their savings are not additive. The counterfactual matters more than the cost. Replaying conversations with the gate removed, the model selected a skill that production had blocked, in a risk-enriched cohort, which the authors are careful to describe as a conditional rate that cannot be extrapolated to all traffic. What the study never does is vary library size: the menu is fixed at ten skills, and "at scale" in that paper refers to traffic volume rather than skill count. The cost side of loading decisions is a separate argument with separate evidence, in Skills Do Not Save You Tokens.
What to do with a library that is getting bigger
Stop counting. A skill count tells you nothing about whether retrieval will work, because the same count can describe a trivially separable library or a hopelessly confusable one, and the evidence above says separation is the variable that moves precision.
Audit for neighbourhoods instead. Embed every description in your library, cluster them, and read the clusters. Any group of skills whose descriptions a human cannot tell apart in one line is a group a retriever cannot tell apart either. The repair is to merge them into one skill with branching guidance, or to rewrite the descriptions around what distinguishes them. Then check what fraction of your candidates could even run for the caller before they reach the model, because a filter on executable state removes candidates that no relevance model will ever demote. Build the query set from user transcripts rather than from your own head, and measure the indirect queries separately, since the average will hide them.
None of that requires a bigger library or a smaller one, as far as retrieval is concerned. Count exerts its own pressure downstream of retrieval. In When Self-Evolution Backfires, a self-evolving agent on the ungated trajectory solved 62% of tasks on its first attempt (pass@1) at the peak, with 105 skills in context, and 50% at 179 skills, against 46% for three hand-written skills. That is a context-contamination effect with no retrieval step involved, on one evolution run per configuration. So count is the wrong variable for retrieval and a live one for context. Retrieval asks something else of you: that the library be shaped so the boundaries between its entries survive being compressed into a single embedding. That is a writing problem wearing a retrieval problem's clothes. The teams that scale a skill library furthest will be the ones treating each description as an interface contract rather than as documentation.
Count is the wrong variable and similarity is the right one, which is exactly why a library of facts about users is the worst shape a selector can be handed. Every fact you move into a skill file is one more thing it has to tell apart. That is one of four reasons we answer the skills question the way we do.
Every figure here is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, with the full claim register published in the open. We build Synap, a memory layer for agents, so read the argument with that in mind and check the register.