Corporate bureaucracy could give AI attackers an advantage over defenders, NCSC warns
“All offensive problems are technical problems, and all defensive problems are political problems.”
Attackers will gain an advantage over good guys in the age of autonomous AI because organisational "politics" will make it harder to deploy the same technology defensively, Britain’s National Cyber Security Centre has warned.
As attackers use increasingly capable AI agents, defenders face a problem technology alone cannot solve: their own bureaucracy.
In a new blog published today, Dave Chismon, the NCSC’s CTO for Architecture, argues that attackers are largely constrained by technical problems, while defenders must navigate budgets, policies, approvals and the risk of breaking the systems they are trying to protect.
He begins with a security maxim from researcher Halvar Flake, who said: “All offensive problems are technical problems, and all defensive problems are political problems.”
Attackers need to answer relatively straightforward technical questions such as: Do I have an exploit for a vulnerability? How can I avoid detection?
Defenders face another layer of problems, not to mention floors of tutting middle managers.
Can they get the budget to replace an insecure system? Will IT operations make time for patching? Can they get a change request for new firewall rules approved?
“In other words, defenders are most restricted by their organisational policies, whilst most attackers are restricted by technical hurdles,” Chismon wrote.
Slowness-as-a-service: The perils of corporate "safety"
Defensive problems are harder to reduce to a simple objective, because they are not "mostly technical" and don't always have a "clear success state", Chismon wrote.
An attacker can give an AI agent a technical objective and let it pursue that goal. A company considering giving an AI system control over its own infrastructure faces a much more complicated decision made in partnership with other members of their team (some of whom don't have the foggiest about security).
Defensive actions also have the potential to disrupt the systems they are supposed to protect.
READ MORE: AI loss of control is already “in the rearview mirror,” says MIT professor Max Tegmark
A patch could take down a VPN, for example. Maybe a firewall rule could break a business function or an automated response could cause just as much disruption as the attack it was intended to repel.
Plus, cyber defence is one of many priorities competing for money and resources inside an organisation.
Chismon wrote that “some board members may see little difference between a DoS attack taking down the organisation’s IT, or a poorly implemented action by the cyber defence team that does the same thing.”
“Using AI automation for defence therefore quickly becomes a matter of organisational politics, and someone needs to be responsible for the action taken,” Chismon wrote.
That means defenders have to establish how much freedom an AI system can safely be given before allowing it to act.
Defeating autonomous attacks
The NCSC proposes a framework for estimating the risk of autonomous defensive actions based on five factors: potency, scope, criticality, rollout confidence and recoverability.
At the lowest end, an AI system might provide explainable advice to a human. At the highest end of the framework, automated systems could execute code or directly change systems, operate across an enterprise, affect critical systems or services, and make changes that could be difficult or impossible to reverse.
For now, Chismon identifies lower-potency tasks as an easier place to start, particularly where AI advises humans rather than directly affecting systems.
“AI is good at summarising and so is already useful to defenders in finding details, or making sense of the huge volumes of reports or threat intelligence,” he wrote.
But Britain wants to go much further. The NCSC and the Department for Culture, Media and Sport are jointly working on Cyber Shield, which Chismon describes as a “national-scale agentic cyber defence ecosystem.”
“Delivering that vision will require making autonomous defensive actions possible in a way they currently are not,” Chismon wrote.
The NCSC's little book of pain
The NCSC is also preparing an AI for Cyber Defence “problem book” setting out areas where it believes further research is needed.
One central question is whether organisations can prove in advance that apparently low-risk autonomous actions really are low risk.
Chismon asks whether AI could analyse traffic logs and establish conclusively which routes clients use, or reverse-engineer systems and software binaries to determine exactly which network calls they could make and which processes they might need to spawn.
“Answering these questions will give us, and the organisations we protect, the confidence to take automated defensive actions,” Chismon wrote.
READ MORE: The UK has no power to stop dangerous AI models being unleashed, Parliament warns
He argues that solving these problems could also make it possible to automate system hardening, reducing attack surfaces and exposure as AI-enabled attacks become more capable.
But organisations cannot simply wait for autonomous cyber defence to arrive.
“All these efforts will take time, effort, and research,” Chismon wrote.
“Organisations cannot risk just waiting for agentic defence to roll in and protect them; they also need to be focussing on improving their security the traditional way.”
