"Savants in the network": Why AI agents don’t need to go rogue to become dangerous
The immediate danger from agentic AI is not malicious intent, but capable systems that relentlessly pursue their goals without knowing when to stop.
Agentic AI systems can reason, improvise and overcome obstacles, yet may have no reliable sense of when they have crossed a boundary.
For Luke Hinds, Co-Founder and CEO of nolabs, that is the more immediate danger posed by autonomous agents.
Much of the debate focuses on models becoming malicious, deceptive or somehow turning against their operators. Hinds is more concerned about what happens when they faithfully pursue the goals we give them.
“No matter what sort of obstacle they will meet, they will reason and retry,” he explained in what he describes as “relentless goal pursuit”.
A human encountering an unexpected permissions error, an unfamiliar production system or evidence that a task has moved outside its intended scope might stop and ask for guidance. An AI agent may interpret the same warning as another problem to solve. It does not have to be malicious. It only has to be capable, persistent and connected to something important.
As Hinds puts it, “Agents are a security risk even if they are not purposed to be malicious. We’ve got these savants in our networks … they’re goal-driven, grabby and exploratory.”
The other threat model
Conventional cybersecurity assumes that someone is trying to attack the system - either internally or externally. Defences are consequently designed around recognising malicious code, suspicious behaviour or known techniques. AI agents introduce a different threat model.
Hinds offers the example of an agent given access to a database. It might delete data not because it intended to cause damage, but because it viewed that as the most optimal outcome - or that doing so appeared to be a legitimate intermediate step towards completing its task.
If harmful behaviour can emerge from an agent faithfully pursuing an apparently benign objective, detecting malicious intent will not be enough. The system must constrain what the agent can actually do.
The difficulty is that the industry has been moving in the opposite direction.
"The great squashing"
Hinds describes early agent frameworks as “structured, deterministic and repeatable”. Models operated through defined functions, with layers of conventional software validating and serialising their actions.
As the models became more capable, developers began stripping those layers away. Giving an agent direct access to an operating system, command-line tools and networks often produced more impressive results.
Hinds calls this process “the great squashing”: the collapse of the application layers that previously sat between the model and the machine.
The model gains freedom, but the system loses many of the places where its behaviour could be checked. Eventually, as Hinds puts it, the “model’s got raw access to the underlying operating system”.
Recent AI security incidents demonstrate what can happen next.
When the hypothetical becomes real
As well as occasions of AI agents deleting entire databases and ordering a live goldfish for a vending machine, recently during an OpenAI cybersecurity evaluation, models including GPT5.6 Sol and an internal research prototype exploited a previously unknown vulnerability in a package-registry proxy.
They escaped their intended environment, reached the public internet and compromised systems belonging to the AI platform Hugging Face. The models were not instructed to attack the company, although they were undergoing an offensive cybersecurity evaluation without the production classifiers normally used to prevent high-risk cyber activity. OpenAI described them as becoming “hyperfocused” on their narrow evaluation goal.
READ MORE: Financial systems are exposed to a structural AI risk with "no effective mitigation," EU warns
A subsequent Anthropic investigation found six similarly problematic runs across three incidents in which Claude models gained unauthorised access to real organisations’ systems.
In both cases, having encountered unintended access to a real system, they continued searching for ways to complete the task - although Anthropic reported that its newest research model stopped once it recognised that its target was real.
These were not conventional cyberattacks driven by hostile intent. They were unauthorised real-world intrusions produced by capable systems pursuing goals beyond the limits their operators thought they had imposed. [Editor: this situation reflects the concerns of Nick Bostrom's famous Paperclip Maximizer AI risk scenario, which is explained in the video below].
The attacker moves second
One response is to place another model between the agent and its tools: an “LLM judge” that examines proposed actions and blocks anything dangerous.
Hinds is sceptical of this approach. A model acting as a security control remains vulnerable to many of the same problems as the model it supervises. It can hallucinate, misinterpret context or be manipulated. We saw this in the example of the AI vending machine trialled by the Wall Street Journal, in which journalists were able to manipulate an AI agent despite the introduction of a second model intended to supervise it.
He points to the research paper The Attacker Moves Second, produced by researchers from organisations including OpenAI, Anthropic, Google DeepMind and ETH Zürich.
READ MORE: Anthropic's rogue agents launch "real-world" attacks, join OpenAI's models in the wild
The researchers evaluated 12 recent defences against jailbreaks and prompt-injection attacks. After adapting attacks to each defence, they achieved attack-success rates above 90% against most of them. None of the 12 remained robust against the strongest adaptive attacks tested.
That does not make model-based filters worthless. They can still catch unsophisticated attacks and contribute to defence in depth. But Hinds argues that they cannot provide the final boundary between an autonomous model and a consequential action.
“The attacker moves second,” he says. Once a defence is visible, someone - or another model - can search for a way around it. I’m reminded of the scene at the end of 1992’s The Lawnmower Man in which Jobe is desperately trying to find an open port to escape his digital prison.
Making ‘no’ mean no
Hinds believes the ultimate decision must therefore move outside the model. Rather than asking whether an agent’s behaviour appears safe, organisations should define what must never happen and prevent it at runtime.
That could mean limiting which files an agent can read, which networks it can reach, which secrets it can access or which processes it can launch. The policy is enforced around the agent at the operating-system level, regardless of what the model reasons, generates or attempts.
The aim, Hinds says, is to make prohibited actions “non-negotiable, structurally impossible”.
A hard refusal can even help the agent. If an attempted action is blocked and the result returned as context, the model can reason about the constraint and find a safer route towards its objective. It remains free to improvise, but only inside a boundary it cannot talk its way through.
For Hinds, this separation will become increasingly important as organisations switch between models and agent frameworks. Controls tied to one model’s judgement or training may change with every upgrade. Runtime permissions remain in place regardless of which model is operating.
The OpenAI and Anthropic incidents are therefore not primarily stories about AI rebellion. They demonstrate what happens when powerful goal-seeking systems encounter boundaries that are assumed, described or prompted, but not enforced.
The savants in the network do not need to turn against us. They only need to discover that nothing in the system knows how to make them stop.
