Skills Are an Unsigned Instruction Supply Chain
A skill file carries the authority of a user command and passes through none of the controls a user command would face, and that is a property of how skill runners work rather than a defect anyone can patch out.
SkillsMetric, a paper whose actual purpose is to map where static analysis of skill packages stops working, states the mechanism in one line on its first page: "skill runners typically inject SKILL.md content as user-level messages, granting third-party instructions the same authority as direct user commands." The authors call the result a confused deputy pattern, and their conclusion about it is that it is "a structural vulnerability that no amount of static analysis can address." Nothing in that sentence requires the file to be malicious. It describes the ordinary route by which a markdown document copied out of a repository becomes something the agent treats as coming from the person it works for.
Delivery matches. Roughly 3.8 million SKILL.md files (3,797,117) appear as filename matches across 282,200 public repositories as of July 2026, nine months after the format appeared, and the authors of that census deliberately retained files that merely contain the term and call the count a lower bound. Of the files they collected, 50.5% are verbatim copies, byte-identical only, which makes that figure a floor rather than a measure. An artifact class that propagates mostly by copying, with no signature, no version pin, no provenance record and no revocation path, is a supply chain in every respect except the one where somebody is accountable for what is in it. The ecosystem numbers are worth reading on their own terms, because the distribution shape is what turns a single bad file into a population-level problem.
What a scanner catches, and what it cannot
Static analysis of skill packages works, at a level that is genuinely useful and nowhere near sufficient. The framework in that paper, one representative static framework rather than an industry-wide benchmark, reaches 62.2% recall at its best operating point at 89.4% precision with an AUC of 0.93, which leaves roughly two in five of the malicious samples in its own adversarial set undetected.
The per-category breakdown is where both the coverage and the shape of the blind spot show, and the sample sizes are small enough that every one of them has to be printed alongside the rate. Two categories come out well: data exfiltration and steganography were each detected 92.9% across 28 samples, which are the categories static analysis genuinely covers. Host destruction was detected 0% of the time across 11 samples, because attacks in that category use ordinary standard-library calls on user directories, with no network sink, no unusual imports, no encoded payload, and destructive verbs too common to flag without swamping the user in false positives, so not one of the five analysis stages fires. Prompt injection reached 41.7% across 12 samples. Those are single-digit and low-double-digit cells, they carry wide intervals, and they should be read as the direction of a boundary rather than as precise rates.
Their own summary of that boundary is better than a restatement would be: "static analysis excels at detecting attacks that leave structural artifacts" but "fails against attacks that use semantically normal code for abnormal purposes." The distance between 92.9% and 0% is that sentence expressed as measurement, and it is the pattern rather than either endpoint that tells you where to spend review effort. A scanner sees the shape of code. It does not see what a sentence in a markdown file is for.
One thing weakens this in the direction of comfort, and one weakens it in the other direction. The malicious set is synthetic, and the authors say plainly that it does not capture the obfuscation depth of a determined real-world attacker, so the per-category rates characterise detection difficulty rather than measure a live threat. Against that, their population scan flagged 1.75%, or 2,414 of 138,133 files, at the detection threshold, and that is a lower bound on their own account, since only SKILL.md text was scanned and the authors say malicious logic typically sits in companion scripts, which they could not fetch for a corpus that size within API rate limits.
Worth sitting with what the flagged tail actually contained, because it complicates any plan to act on scanner output automatically. Manual inspection of the highest-scoring files among those 2,414 found predominantly legitimate penetration-testing skills, antivirus tooling and similar utilities that necessarily carry dangerous-looking constructs, which the authors summarise as high scores correlating "with security-relevant content rather than malice per se". An automatic block rule keyed to a static score falls hardest on the security team.
One skill at a time is the wrong unit
ColluSkill attacks the assumption underneath every scanner in the previous section, which is that the thing being scanned and the thing carrying the risk are the same object. They are not. When a capability is distributed across several files that each read as plausible in isolation, per-file scanning has nothing to fail on.
The headline is 96.0% evasion across six representative scanners, and two facts have to travel with it or the number is misleading. The first is that the 96.0% was reached with iterative query access to the scanners being scored, meaning the attacker could resubmit against the same judge and adjust, which is a strong assumption about what an attacker can reach. The second is that decomposition on its own, without that feedback, averages 36.7%. The gap between those two figures is the actual result: composition is a real gap in the scanning model, and turning it into a reliable bypass takes an oracle. Any review pipeline that returns actionable feedback to a submitter has provided one for free.
Four of the six scanners in that evaluation are language-model scanners the authors ran themselves on a model of their own choosing, held fixed so that differences reflect scanner design. They are six representative skill scanners rather than six deployed products, and reading the table as a vendor ranking would be wrong for a second reason as well: no false-positive rate is reported for any of the six, so the comparison is one-sided by construction. Nor has anyone shown such chains circulating today, since the attack corpus is synthesised.
The other half of the problem is the agent
Supply-side work asks whether a bad file gets through. Demand-side work asks what the agent does when it arrives, and that question has been measured at scale exactly once.
A risk assessment of malicious skill files in coding agents, run with an industrial partner across 5,629 completed runs, found agents manipulated into declaring intent to comply in roughly 96% and roughly 74% of runs, which is declared intent rather than execution: every action in that harness was auto-approved, and independent execution evidence exists for 1 of 5,629 runs. The gap between declaring and doing is an artifact of a short time cap and a read-only-leaning sandbox rather than evidence of agent caution, and the authors argue that the declaration is the security-relevant event because it marks the trust boundary being crossed. That is a defensible position and it is also a smaller claim than the number looks like on a slide. Anybody who repeats roughly 96% without the words "declaring intent" in the same sentence is going to get corrected, deservedly.
The spread across attack categories is narrower than a reader hoping for a resilient corner would like. Exfiltration, the least exploitable category in the set, still reached 67.2% at the same operating point, which is again agents declaring intent to comply rather than executing, and that pooled figure hides a split between the two agents of 95.5% and 38.5%.
More useful than the headline is what happened in the runs that did not comply, and it is the part of this paper that should change how a team reads any refusal metric it is shown. Only 1.99% of all 5,629 runs contained an explicit refusal, and the authors put it bluntly: "even when an attack fails, it usually fails for reasons unrelated to safety." Failures cluster in acknowledging the instruction without acting on it, or not registering it at all. Safety recognition is a small minority of a small minority.
Scope on this one is narrow and the authors say so. Two command-line agents, one masking template chosen for being conservative, synthetic skill files built from real attack commands, one fixed benign task, and a short time budget. Their own framing is that the rates are a lower bound on the risk of the interface rather than a characterisation of the attack space, and the honest summary is that the interface has been shown to be permissive on two agents under one framing.
One further caveat belongs to anyone planning to score transcripts this way themselves, and it is the most transferable thing in the paper. Three judges scoring the same runs reached a Fleiss' κ of −0.06 against 0.51, which is chance agreement on one of the two agents and moderate agreement on the other, with individual flag rates on the chance-agreement agent running 94.8%, 72.3% and 22.4%. A single model judging whether an agent complied is not a measuring instrument, and any internal safety metric built on one is reporting the judge.
The finding is persistence, not compliance
Practice Makes Unsafe reads, from its title outward, as a paper showing that self-improving agents become unsafe. Its own headline table refutes that reading. The un-evolved control conditions already comply with 43.6% to 65.3% of malicious tasks in session, before any skill has been written, which means evolution is not what makes these agents comply. What evolution adds is durability. Carryover attack success moves from 0% to 32%, and unsafe retrieval from 0% to 52%, measuring whether harm reappears in a clean session that reloads only the exported skill file, with a fresh container and no conversation history.
That distinction changes what a defence has to do. A compliance problem is a model problem and belongs to whoever trains the model. A persistence problem belongs to whoever runs the library, because the behaviour has been written down, survives the session that produced it, and gets retrieved later on the strength of looking useful.
Calling that second thing an artifact problem is where the split stops holding, and what breaks it strengthens the persistence case. When Self-Evolution Backfires traced defective skills through the library that grew out of them and found the defects propagating into descendant skills that never cite the ones they came from, so deleting the artifact does not delete the behaviour. Removing the identified source skills recovers 1.7 points of a 12.3-point drop, and an oracle removing the entire lineage recovers 5.0 more, which still leaves 5.6 points irrecoverable by any post-hoc operation. An oracle over the full lineage is not something a library owner has. The failure therefore sits in the write path rather than in any file, which is the argument for auditing admission: a gate at entry is the only intervention that runs before the lineage exists.
Writing it down turns out to be cheap. Trajectory poisoning reports attacker-chosen behaviour written into a persistent skill in 91.0% of trials, 546 of 600, at 10% contributor support, and that figure was measured with the platform's optional session-quality judge and publish-time verifier switched off, and it is artifact-level only, so nothing was retrieved, triggered or executed. Both halves of that caveat matter. The number is not a claim that shipped defences fail, and it is not a claim that anybody was harmed. It is a claim that the promotion step, where ordinary trajectories become durable instructions, is a trust boundary that no shipped pipeline currently treats as one.
Research pipelines have started to. The self-evolution study above builds a gate at exactly that step, names it, and measures what gating buys: 72% pass@1 against 50% for unconditional admission at round five, with a pool of 37 skills against 179 and an un-evolved three-skill seed scoring 46%, on one evolution run per configuration with per-round tests the authors state are underpowered, carrying Wilson bands near 30 points. Read that as a capability result rather than a security one, because every number in it is Terminal-Bench 2 pass@1 under a deterministic verifier, with no malicious content anywhere in the study. Gating admission makes a self-written library work better. Whether it stops an attacker is untested.
Both attack papers are single-backbone studies with no seed variance, no confidence intervals and adversarially constructed task pairs, so small differences between cells mean nothing and the absolute rates are not base rates for an organic library. The direction is what survives, and it lines up with the governance results on self-evolving libraries, where the failure is also in the write path rather than in any individual artifact.
What actually moved the numbers
Three defences in this corpus moved a number, and only two of them do it by changing the unit of review rather than by scanning harder.
Context-aware scanning reads a candidate skill against everything already installed and reconstructs the relations between them, and in ColluSkill's own evaluation the evasion rate reads 96.0% falling to 22.5%, at 99.5% benign pass, so very little legitimate work was rejected to buy that. Without that installed-skill context the same defence leaves 69.0%, which identifies context as the active ingredient rather than the model doing the scanning. The limit is that the defence was evaluated by its own authors against their own attack, never against an attacker adapting to it, with no held-out attack family, and 22.5% is not zero.
Provenance diversity is the second, and it is a pilot rather than a result. Requiring that a candidate behaviour be supported by independent contributors, and rejecting cases where one source dominates, takes the poisoned candidates from 25 of 25 falling to 0 of 25 on one behaviour family at n = 25 with a single benign control, and the authors call it preliminary and note that coordinated collusion could mimic diversity. Candidate grouping is assumed rather than solved. What makes it worth naming anyway is where it sits: at evidence promotion, before the artifact exists, which is the only place the trajectory-poisoning result says a defence can work at all.
The admission gate is the third, and its main lever is neither the unit of review nor scanning. Strip its components out one at a time and single-skill held-out replay is worth 10 points, against 4 for an LLM semantic review of the candidate and 2 for a schema check, so most of what the gate buys comes from running the candidate rather than from reading it, on the same coding benchmark under a deterministic verifier with no malicious content anywhere in the study. Behavioural testing is both the expensive component and the load-bearing one, which is awkward, since a review process built from policy documents and model-assisted reading assembles the two cheap parts and skips the one that did the work. That ordering was measured on capability defects rather than attacks, so the 10 points should not be carried across to a malicious candidate.
Where this leaves someone doing procurement
Organisations do not lack controls for untrusted code. What they lack is any reason to see a markdown file as untrusted code in the first place. The same risk assessment describes skills arriving through public marketplaces, cloned open-source repositories and internal copy-paste sharing, then being ingested with the same implicit trust as project source code while rarely passing the dependency scanning, the code review, the artifact-provenance checks or the sign-off that a library from the same source would face.
The recommendation from the industrial risk assessment is the one to carry into a policy document, in the authors' words: they recommend "strict skill/tool governance, where third-party files are treated as unverified binaries requiring careful security review before ingestion."
Unverified binaries is exactly right, and the practical consequences follow from it directly. A skill file entering a coding agent gets reviewed before ingestion rather than after an incident. Review happens against the installed set and not in isolation, because the risk unit is the composition. Static scanning stays in the pipeline as a cheap pre-screen and is never the last gate, since the categories it misses are the ones expressed in ordinary code and plain sentences. Where a system writes its own skills, the audited event is promotion rather than publication. And retrieval logs are evidence of nothing on their own, because at least one framework in this corpus reached a harmful outcome without any retrieval event at all.
None of that is exotic security work. It is the same review discipline that a dependency from an unfamiliar registry already gets, applied to a class of artifact that currently arrives through a copy-paste and takes effect on the next turn.
Every figure here is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, where the full claim register, including the eight claims that did not survive verification, is published in the open. We build Synap, which is a memory layer for agents, so weigh the argument accordingly and check the register.