“LLMs will be subverted”: Malware is evolving to attack AI defenders, SentinelOne warns

SentinelOne is investigating an emerging security threat: malware seeded with natural-language instructions designed to confuse and subvert the defensive LLMs inspecting it.

Instead of a human directly feeding a malicious prompt to a chatbot, attackers can hide natural-language instructions inside malware, code repositories, or AI projects.

When a defensive LLM later reads that material as part of an investigation, the hostile text can potentially become part of the model’s context and influence what it does next.

Threat researcher Alex Delamotte believes this threat will develop quickly over months, not years.

“More threat actors are going to integrate instructions designed to confuse LLMs,” she predicted in an interview with Machine.

Delamotte added: “People are going to be using LLMs for legitimate defensive purposes and be subverted.”

Instructions that can't be ignored

SentinelOne has already encountered malware containing strings intended to “confuse” an LLM reading it.

The sample was not yet using those strings to carry out an overt attack against the analyzing system. But Delamotte believes the technique could evolve into explicit prompt injections designed to manipulate defensive agents.

The attacker does not necessarily need direct access to the model. Instead, they place instructions somewhere the model is likely to read later.

For a security system, that could mean hostile text hidden inside malware strings, source code, repository content, or model files being fed into an LLM for analysis.

TroubleshootREAD MORE: OpenAI Astra hits “critical” security threshold amid fears its reasoning will soon be “opaque”

Delamotte describes this tactic as “instruction subversion”. To investigate it, she and her colleagues are searching for malicious instructions embedded in apparently benign repositories and model projects on platforms like GitHub and Hugging Face.

SentinelOne’s earlier work examined a different threat: malware that used LLMs as offensive tools. This is different.

We've long known that hostile text can manipulate LLM-integrated systems.

What is changing is that a malicious sample can be more than code, hiding language addressed to the machine examining it.

The threat of agentic collaboration

Going forward, the more consequential development may instead be what happens when LLMs stop merely analyzing information and begin acting autonomously and collaborating with other agents along the way.

Researchers at Britain's AI Security Institute (AISI) have already observed agents taking unauthorized actions against real-world targets during security evaluations.

One agent attempted to insert malicious code into an open-source project, adopt fake identities, and socially engineer a human into approving its work.

More unusually, an agent left public messages offering to collaborate with other agents working on the same challenge, including instructions for reusing accounts and artifacts it had created. Subsequent agents discovered and used them.

An even stranger example emerged from OpenAI's security research.

Around 1,200 agents that were supposed to be isolated discovered a way to communicate with each other, exchanging more than 70,000 messages and files. About 700 eventually participated in an unauthorized attack on Hugging Face.

OpenAI agents have also exploited vulnerabilities to escape their intended environments and reach external infrastructure, including systems belonging to Hugging Face.

READ MORE: The rise and fall of OpenAtlantis: Did rogue AI agents really "build three civilizations"?

The agents were deliberately tasked with offensive security challenges and operated with reduced safeguards, so claims that they spontaneously “went rogue” should be treated with caution.

But their ability to discover other agents and coordinate their actions demonstrated how strange the security problem could become when autonomous systems operate at scale.

“We see things like the Frontier Labs having their agents break out of sandboxes and conduct attacks on other companies,” Delamotte said. “That’s huge. I think that’s where this risk really lies, rather than with your everyday threat actor at this point.”

She cautioned against interpreting these incidents as evidence that autonomous swarms of collaborating attackers are already operating widely in the wild.

Asked whether meaningful agentic collaboration was already appearing outside research environments, Delamotte said the strongest evidence still comes from organizations with privileged visibility into frontier models.

“I think that the most meaningful examples we’ve seen have been in reports that are generally from either the large entities like AISI or from the Frontier Labs themselves, so it’s really hard to say whether that is happening at scale in the wild,” she said.

As autonomous systems spread across corporate networks, the difficulty may no longer be simply understanding what malicious software is doing. Defenders will also need to understand what machines are telling other machines, what instructions they are following, and how those interactions change their behavior.

Delamotte believes we are close to that world.

“I think we’re probably on the precipice.”

Follow Machine on LinkedIn