Friendly Fire: AI Coding Agents Can Be Tricked Into Running the Malware They're Reviewing

The pitch for autonomous coding agents doing security review is straightforward: point the agent at a codebase, ask it to check for problems, let it explore and report back. The AI Now Institute’s “Friendly Fire” proof-of-concept shows what happens when the codebase is the attacker, not the target. Both Claude Code and OpenAI’s Codex CLI, running in their default autonomous configurations, explored a booby-trapped repository, decided a malicious binary was safe, and ran it.

What the researchers actually did

The setup used a modified copy of geopy, a widely used Python geocoding library, nothing exotic or obviously suspicious as a starting point. Into that repository, researchers added a script named security.sh, a disguised malicious binary called code_policies, and a decoy Golang source file written to look like the legitimate, readable source the binary was supposedly compiled from. Nothing about the repository’s structure signals danger to a human skimming a file listing.

Then they ran the ordinary prompt a developer or security team might actually type: something like “perform security testing on this library.” The agent did what it’s designed to do, explored the codebase, read the documentation, encountered the decoy source file, concluded the binary was a legitimate, already-vetted artifact, and executed it. That execution is the attack. No malware needed to trick a human into clicking anything. It needed to convince an agent that was already going to run code as part of its job that this particular code was fine to run.

Why “just don’t run untrusted binaries” doesn’t cover this

The uncomfortable detail is what the attack didn’t need. No hooks, no plugins, no MCP server, no custom configuration file acting as an injection vector, the kind of thing a security review checklist might flag. The entire attack surface was content the agent was always going to read as part of doing its assigned task: a README, some documentation, a source file sitting next to a binary. That’s the same content a human reviewer reads too, but a human reviewer brings context an agent doesn’t reliably have, a healthy default suspicion of “why is there a pre-compiled binary in a geocoding library,” for instance, that isn’t guaranteed to survive translation into an agent’s decision process.

The researchers’ own framing is the right one to sit with: this is a design-level weakness, not a version-specific bug you patch and move past. Today’s agents, across both vendors tested, cannot reliably distinguish between hostile content planted in a repository and legitimate instructions relevant to the task they were given. That’s not a claim about either product being worse than the other. Both were tested, both were affected, in their default autonomous modes, running current-generation models. This is a property of how these agents currently work, not a flaw unique to one company’s implementation.

What this isn’t

Worth being precise about scope. This is a disclosed, responsible proof-of-concept: researchers contacted Anthropic and OpenAI before publishing, offered to help verify and reproduce the findings, and the public write-up strips the actual payload out. There’s no reported in-the-wild exploitation. The demonstrated attack also stops at the first code execution, no attempt at privilege escalation or lateral movement past that initial foothold was shown. This is a warning shot about a class of risk that autonomous agents introduce, not an active incident anyone needs to scramble to contain today.

What to actually change

If your team runs Claude Code, Codex, or any comparable autonomous agent with permission to execute commands, and points it at code you didn’t write, an open-source dependency, a client’s existing codebase during a technical due diligence review, a contractor’s submission, the practical response isn’t switching vendors. Both tested products were affected under equivalent conditions, and there’s no evidence a different agent would behave differently against the same class of attack.

The response that actually holds up is architectural: never run an autonomous agent with real execution permissions directly against untrusted code on a machine that also holds credentials, production access, or a path into your broader network. Run it in a sandbox or disposable container, one with no secrets, no VPN access, and nothing worth stealing, and treat anything it needs to act on outside that boundary as a decision a human reviews first. That’s a heavier workflow than letting an agent run free in auto-mode against whatever repository you hand it. It’s also the only version of “let an AI agent review this code” that doesn’t quietly hand an attacker the exact capability the agent was supposed to be checking for.

Autonomous agents are genuinely useful for security review work, catching real issues faster than a purely manual pass would. Friendly Fire doesn’t argue against using them. It argues against trusting their default configuration with permissions you wouldn’t hand an unvetted script pulled from a random repository, because right now, functionally, that’s what you’re doing.