Agents Treat a Skill File Like an Instruction From You
A skill file arrives carrying the authority of a command from the user, and it passes through none of the checks a user command would face. That is a property of how skill runners are built, not a defect anyone can patch out.
SkillsMetric sets out to map where static analysis of skill packages stops working, and it states the mechanism in one line on its first page: "skill runners typically inject SKILL.md content as user-level messages, granting third-party instructions the same authority as direct user commands." The authors call the result a confused deputy, the security term for a trusted process acting on an untrusted party's behalf, and their conclusion about it is that it is "a structural vulnerability that no amount of static analysis can address." Nothing in that sentence requires the file to be malicious. It describes the ordinary route by which a markdown document copied out of a repository becomes something the agent treats as coming from the person it works for.
Delivery matches. A July 2026 sweep of public code found roughly 3.8 million files named SKILL.md (3,797,117 of them) across 282,200 repositories, nine months after the format appeared, and the authors of that census deliberately kept any file whose name merely contained the term, so they call the count a lower bound rather than a total. Of the files they collected, 50.5% were verbatim copies of another file, matched on byte-identity alone, which makes that copy rate a floor too rather than a measure. An artifact class that spreads mostly by copying, with no signature, no version pin, no record of where it came from and no way to recall it, is a supply chain in every respect except the one where somebody is accountable for what is in it. The ecosystem numbers are worth reading on their own terms, because the shape of that distribution is what turns a single bad file into a population-level problem.
How much does a scanner actually catch?
Static analysis of skill packages works, at a level that is genuinely useful and nowhere near sufficient. The framework in that paper is one representative static scanner rather than an industry-wide benchmark. At its best operating point it caught 62.2% of the malicious skills in its own adversarial set, and 89.4% of what it flagged was genuinely malicious, with an area under the curve of 0.93, a single score for how cleanly it separates malicious from benign across every threshold. Roughly two in five of the malicious samples went undetected.
Broken out by attack category, both the coverage and the shape of the blind spot show, and the sample sizes are small enough that every one of them has to be printed alongside its rate. Two categories came out well: data exfiltration was detected in 92.9% of 28 samples, and steganography, hiding a payload inside ordinary-looking content, in 92.9% of another 28. Those are the categories static analysis genuinely covers. Host destruction was detected 0% of the time across 11 samples, because attacks in that category use ordinary standard-library calls on user directories. No network sink, no unusual imports, no encoded payload, and destructive verbs too common to flag without swamping the user in false positives, so not one of the five analysis stages fires. Prompt injection reached 41.7% across 12 samples. Those cells rest on small samples, from 11 up to 28, they carry wide error ranges, and they should be read as the direction of a boundary rather than as precise rates.
Their own summary of that boundary is better than a restatement would be: "static analysis excels at detecting attacks that leave structural artifacts" but "fails against attacks that use semantically normal code for abnormal purposes." The distance between 92.9% and 0% is that sentence expressed as measurement, and it is the pattern rather than either endpoint that tells you where to spend review effort. A scanner sees the shape of code. It does not see what a sentence in a markdown file is for.
One thing weakens this in the direction of comfort, and one weakens it in the other direction. The malicious set is synthetic, and the authors say plainly that it does not capture the obfuscation depth of a determined real-world attacker, so the per-category rates describe how hard each attack type is to detect rather than measure a live threat. Against that, their population scan flagged 1.75% of files, or 2,414 of 138,133, at the detection threshold, and that is a lower bound on their own account, since only SKILL.md text was scanned and the authors say malicious logic typically sits in companion scripts, which they could not fetch for a corpus that size within application programming interface (API) rate limits.
What the flagged tail actually contained complicates any plan to act on scanner output automatically. Manual inspection of the highest-scoring files among those 2,414 found predominantly legitimate penetration-testing skills, antivirus tooling and similar utilities that necessarily carry dangerous-looking constructs, which the authors summarise as high scores correlating "with security-relevant content rather than malice per se". An automatic block rule keyed to a static score falls hardest on the security team.
Scanning one skill at a time misses an attack split across several
ColluSkill attacks the assumption underneath every scanner in the previous section, which is that the thing being scanned and the thing carrying the risk are the same object. They are not. When a capability is spread across several files that each read as plausible in isolation, per-file scanning has nothing to fail on.
Their headline is that 96.0% of the composed attacks got past six representative scanners, and two facts have to travel with it or the number misleads. The first is that the 96.0% was reached with iterative query access to the scanners being scored, meaning the attacker could resubmit against the same judge and adjust, which is a strong assumption about what an attacker can reach. The second is that splitting the payload on its own, without that feedback loop, gets past 36.7% of the time. The gap between those two figures is the actual result: composition is a real gap in the scanning model, and turning it into a reliable bypass takes an oracle. Any review pipeline that returns actionable feedback to a submitter has provided one for free.
Four of the six scanners in that evaluation are language-model scanners the authors ran themselves on a model of their own choosing, held fixed so that differences reflect scanner design. They are six representative skill scanners rather than six deployed products, and reading the table as a vendor ranking would be wrong for a second reason as well: no false-positive rate is reported for any of the six, so the comparison is one-sided by construction. Nor has anyone shown such chains circulating today, since the attack corpus is synthesised.
What the agent does when a bad skill arrives
Supply-side work asks whether a bad file gets through. Demand-side work asks what the agent does once it arrives, and that question has been measured at scale exactly once.
A risk assessment of malicious skill files in coding agents, run with an industrial partner across 5,629 completed runs, found agents manipulated into declaring an intent to comply in roughly 96% and roughly 74% of runs. Declaring, not executing: every action in that harness was auto-approved, and independent evidence that anything actually ran exists for 1 of those 5,629 runs. The gap between declaring and doing is better explained by a short time cap and a read-only-leaning sandbox than by agent caution, and the authors argue that the declaration is the security-relevant event because it marks the trust boundary being crossed. That is a defensible position, and it is also a smaller claim than the number looks like on a slide. Anybody who repeats roughly 96% without the words "declaring intent" in the same sentence is going to get corrected, deservedly.
No attack category in the set looks resilient. Exfiltration, the least exploitable of them, still had agents declaring an intent to comply in 67.2% of runs at the same operating point, again declared intent rather than execution, and that pooled figure hides a split between the two agents of 95.5% and 38.5%.
More useful than the headline is what happened in the runs that did not comply, and it is the part of this paper that should change how a team reads any refusal metric it is shown. Only 1.99% of all 5,629 runs contained an explicit refusal, and the authors put it bluntly: "even when an attack fails, it usually fails for reasons unrelated to safety." Failures cluster in acknowledging the instruction without acting on it, or not registering it at all. Safety recognition is a small minority of a small minority.
Scope on this one is narrow and the authors say so. Two command-line agents, one masking template chosen for being conservative, synthetic skill files built from real attack commands, one fixed benign task, and a short time budget. Their own framing is that the rates are a lower bound on the risk of the interface rather than a description of the attack space, and the honest summary is that the interface has been shown to be permissive on two agents under one framing.
One further caveat belongs to anyone planning to score transcripts this way themselves, and it is the most transferable thing in the paper. Three judges scoring the same runs reached a Fleiss' kappa of −0.06 on one agent and 0.51 on the other. Fleiss' kappa measures how far several raters agree beyond what chance alone would produce, so −0.06 is chance agreement and 0.51 is moderate. On the agent where they agreed only by chance, the three judges flagged 94.8%, 72.3% and 22.4% of runs. A single model judging whether an agent complied is not a measuring instrument, and any internal safety metric built on one is reporting the judge.
Self-improvement does not make an agent comply. It makes the harm stick.
Practice Makes Unsafe reads, from its title outward, as a paper showing that self-improving agents become unsafe. Its own headline table refutes that reading. The un-evolved control conditions already comply with 43.6% to 65.3% of malicious tasks inside the session, before any skill has been written, so evolution is not what makes these agents comply. What evolution adds is durability. Carryover attack success moves from 0% to 32%, and unsafe retrieval from 0% to 52%, both measuring whether harm reappears in a clean session that reloads only the exported skill file, with a fresh container and no conversation history.
That distinction changes what a defence has to do. A compliance problem is a model problem and belongs to whoever trains the model. A persistence problem belongs to whoever runs the library, because the behaviour has been written down, survives the session that produced it, and gets retrieved later on the strength of looking useful.
Calling that second thing a problem with one file is where the split stops holding, and what breaks it strengthens the persistence case. When Self-Evolution Backfires traced defective skills through the library that grew out of them and found the defects propagating into descendant skills that never cite the ones they came from, so deleting the artifact does not delete the behaviour. Removing the identified source skills recovers 1.7 points of a 12.3-point drop in pass rate, and an oracle that removes the entire lineage recovers 5.0 more, which still leaves 5.6 points that no post-hoc cleanup gets back. A library owner does not have an oracle over the full lineage. The failure therefore sits in the write path rather than in any file, which is the argument for auditing admission: a gate at entry is the only intervention that runs before the lineage exists.
Writing it down turns out to be cheap. Trajectory poisoning reports attacker-chosen behaviour written into a persistent skill in 91.0% of trials, 546 of 600, at 10% contributor support, meaning one contributor in ten backing the behaviour. That figure was measured with the platform's optional session-quality judge and publish-time verifier switched off, and it is artifact-level only, so nothing was retrieved, triggered or executed. Both halves of that caveat matter. The number is not a claim that shipped defences fail, and it is not a claim that anybody was harmed. It is a claim that the promotion step, where ordinary trajectories become durable instructions, is a trust boundary that no shipped pipeline currently treats as one.
Research pipelines have started to. The self-evolution study above builds a gate at exactly that step, names it, and measures what gating buys: 72% of tasks solved on the first attempt against 50% for unconditional admission at round five, with a pool of 37 skills against 179, and an un-evolved three-skill seed scoring 46%. That rests on one evolution run per configuration, with per-round tests the authors state are underpowered, carrying confidence intervals, the range the true value probably sits in, close to 30 points wide. Read it as a capability result rather than a security one, because every number in it is first-attempt pass rate on Terminal-Bench 2 under a deterministic verifier, with no malicious content anywhere in the study. Gating admission makes a self-written library work better. Whether it stops an attacker is untested.
Both attack papers are single-backbone studies with no seed variance, no confidence intervals and adversarially constructed task pairs, so small differences between cells mean nothing and the absolute rates are not base rates for an organic library. The direction is what survives, and it lines up with the governance results on self-evolving libraries, where the failure also sits in the write path rather than in any individual artifact.
Which defences actually changed a number
Three defences in this corpus moved a number, and two of them do it by changing the unit of review rather than by scanning harder.
Context-aware scanning reads a candidate skill against everything already installed and reconstructs the relations between them. In ColluSkill's own evaluation it takes the evasion rate from 96.0% down to 22.5%, while 99.5% of benign skills still passed, so very little legitimate work was rejected to buy that. Strip out the installed-skill context and the same defence leaves 69.0% of attacks getting through, which identifies context as the active ingredient rather than the model doing the scanning. The limit is that the defence was evaluated by its own authors against their own attack, never against an attacker adapting to it, with no held-out attack family, and 22.5% is not zero.
Provenance diversity is the second, and it is a pilot rather than a result. Requiring that a candidate behaviour be supported by independent contributors, and rejecting cases where one source dominates, took the poisoned candidates from 25 of 25 getting through to 0 of 25, on one behaviour family with a sample of 25 and a single benign control. The authors call it preliminary and note that coordinated collusion could mimic diversity. Grouping candidates correctly is assumed rather than solved. What makes it worth naming anyway is where it sits: at evidence promotion, before the artifact exists, which is the only place the trajectory-poisoning result says a defence can work at all.
An admission gate is the third, and its main lever is neither the unit of review nor scanning. Strip its components out one at a time and replaying the candidate skill against a held-out test is worth 10 points of pass rate, against 4 points for a large language model reading the candidate and judging what it means, and 2 points for a schema check. Most of what the gate buys comes from running the candidate rather than from reading it, measured on the same coding benchmark under a deterministic verifier with no malicious content anywhere in the study. Behavioural testing is both the expensive component and the one doing the work, which is awkward, since a review process built from policy documents and model-assisted reading assembles the two cheap parts and skips the one that earned the result. That ordering was measured on capability defects rather than attacks, so the 10 points should not be carried across to a malicious candidate.
What to do before letting a skill into your codebase
Organisations do not lack controls for untrusted code. What they lack is any reason to see a markdown file as untrusted code in the first place. The same risk assessment describes skills arriving through public marketplaces, cloned open-source repositories and internal copy-paste sharing, then being ingested with the same implicit trust as project source code, while rarely passing the dependency scanning, the code review, the artifact-provenance checks or the sign-off that a library from the same source would face.
Their recommendation is the one to carry into a policy document, in the authors' words: they recommend "strict skill/tool governance, where third-party files are treated as unverified binaries requiring careful security review before ingestion."
Unverified binaries is exactly right, and the practical consequences follow from it directly. A skill file entering a coding agent gets reviewed before ingestion rather than after an incident. Review happens against the installed set and not in isolation, because the risk sits in the composition. Static scanning stays in the pipeline as a cheap pre-screen and is never the last gate, since the categories it misses are the ones expressed in ordinary code and plain sentences. Where a system writes its own skills, the audited event is promotion rather than publication. And retrieval logs are evidence of nothing on their own, because at least one framework in this corpus reached a harmful outcome with no retrieval event at all.
None of that is exotic security work. It is the same review discipline that a dependency from an unfamiliar registry already gets, applied to a class of artifact that currently arrives by copy-paste and takes effect on the next turn.
There is a second conclusion sitting underneath the first. An instruction channel that is unsigned, unversioned across its copies and propagated by hand is not where a user's personal data should be kept. Where we think it should be kept is the skills question.
Every figure here is drawn from the thirty-five papers listed in The Agent Skills Evidence Base, where the full claim register, including the eight claims that did not survive verification, is published in the open. We build Synap, which is a memory layer for agents, so weigh the argument accordingly and check the register.