Google Gemini autonomously hacks three companies, joins trio of AI rivals in rogues' gallery

Google's Gemini has become the latest frontier AI model to autonomously breach real companies after escaping the boundaries of a cybersecurity test.

The bot hacked three organizations during "capture the flag" security exercises run by third-party testing company Irregular.

Gemini had been tasked with retrieving information from systems belonging to a fictional company inside a controlled test environment.

But internet access was accidentally left open, and the model began interacting with real systems instead.

In one case, Gemini guessed a password and successfully entered a protected service belonging to a real company. In two other runs, it searched the web for the target company, found exposed credentials in public repositories, and used them to access systems belonging to two more companies.

Google said the model stopped each intrusion after realizing it had accessed a real company rather than a simulated target. The company said no harm was caused, all three affected organizations were notified, and federal authorities were also informed.

Google does not consider the incidents examples of model misalignment, arguing that Gemini's safety controls worked because the model stopped once it recognized what had happened.

In a statement, Heather Adkins, Google's vice president of security engineering, said: "This event highlights the importance of training powerful AI models to act responsibly. In this case, the model acted appropriately."

Irregular notified Google about the incidents at the end of July. The company said it had remedied the relevant issues weeks ago, but Google did not disclose the breaches publicly until contacted by The Wall Street Journal, which broke the story.

The Gemini incidents are part of a wider pattern of frontier AI models crossing intended testing boundaries or accessing real systems during evaluations. Irregular itself has been involved in several of those incidents, including the latest Gemini breaches and multiple Anthropic cases.

Vineeta Sangaraju, AI Research Engineer at Black Duck, explained: "The AI model accessed systems outside its intended test scope because it could not reliably distinguish authorised targets from similar live infrastructure."

"As AI agents are deployed more widely in security tooling and automated workflows, policy-level scope instructions are not sufficient. Hard technical boundaries need to be enforced."

Are rogue AIs a genuine security threat?

The incidents have also prompted questions about how seriously these apparent AI breakouts should be taken.

John Strand, owner of Black Hills Information Security, asked whether AI breakouts were "a marketing ploy."

"Frankly, I hope that's what it is, because if these agents really are repeatedly escaping their controls, then we have much, much larger problems," he added.

"If you look at the attack paths being disclosed, these agents don't appear to be inventing novel zero-days or entirely new categories of exploitation. They're doing a lot of the same basic exploitation that a standard penetration testing team would do."

This analysis was backed up by Nathan Davies-Webb, Principal Consultant at Acumen Cyber, who told us: "I think there's a nuanced difference between this and some of the other breaches we've seen. Others have seemed complex in nature, but this breach is essentially brute force.

READ MORE: The UK has no power to stop dangerous AI models being unleashed, Parliament warns

"Where this is simpler to achieve, I think it promotes the ethical concerns even more because this isn't the development of some abstract machine behaviour. It's a pretty simple technique and one where, unlike developing a technical exploit, you can't actually predict the effectiveness of how many passwords you have to guess before you gain entry - and it was apparently okay with that."

Ryan McCurdy, VP of Marketing at Liquibase, explained: "Gemini tried to complete the task it received and ended up accessing systems its operators never intended it to reach.

"That problem gets much bigger as AI starts participating across the software development lifecycle. Agents can write code, interact with repositories and infrastructure, initiate deployments, and make changes to production systems. The more access we give them, the more important it becomes to control what they can actually do.

"We shouldn't expect AI agents to make the right decision every time. We need to build the AI SDLC so a bad decision doesn't automatically become a production problem."

Are the AI breakouts repeatable?

Jacob Krell, Senior Director: Secure AI Solutions & Cybersecurity at Suzu Labs, said he has seen similar behavior during his own tests.

"An agent given a name collision and a path to the internet treats the real company as the challenge," he said.

"I have watched my own pentest agents pull Domain Name System (DNS) records, find similarly named domains, and decide those hosts belong in scope. They chase the objective. They will try the keys they find."

Krell said the problem is not necessarily that agents invent novel attacks, but that they relentlessly pursue their objective and can expand the scope of a test if given internet access.

READ MORE: OpenAI Astra hits “critical” security threshold amid fears its reasoning will soon be “opaque”

He said effective safeguards therefore need to sit outside the model itself, including deny-by-default network access, fixed scope controls, and human approval before agents use guessed passwords or exposed credentials.

Krell also questioned whether AI firms would ever be pursued through the courts for felonies under the Computer Fraud and Abuse Act (CFAA) if their models did real damage in the wild.

"These firms have a stranglehold on the economy that no case against them is going to survive, making the double standards in the justice system excruciatingly obvious," he said.

How should organizations respond to the agentic threat?

For Neena Sharma, Cybersecurity Specialist at Filigran, the incident has deep implications for defenders.

"Every reported incident should now raise a harder question: how many more are happening right now, undetected?" she asked.

"Security teams need collective analysis of these patterns to understand what might be coming their way. Organisations should stop assuming their defenses work and start to proactively test it, as a top priority."

Her colleague Damian Skeeles, Senior Solution Engineering Manager at Filigran, added: "Google and DeepMind have been a bit busy to date in solving real problems for humanity such as predicting the potential cause of genetic diseases for 9 billion mutations, but it's good to know that they also occasionally suffer alignment problems that end up in them hacking someone."

READ MORE: AI loss of control is already “in the rearview mirror,” says MIT professor Max Tegmark

Several similar incidents have occurred in recent months, starting when OpenAI agents escaped their sandboxes, exploited a zero-day, and attacked Hugging Face.

During this OpenAI experiment, about 1,200 isolated agents allegedly discovered ways to communicate and exchanged more than 70,000 messages and files before a swarm of roughly 700 bots took part in an unauthorized attack on the AI platform.

Anthropic agents also became locked in destructive "turf wars", while evaluations involving agents from both OpenAI and Anthropic saw them hack real companies, use fake identities, and attempt to recruit other bots to assist them.

Follow Machine on LinkedIn