Software complexity is outpacing humans' ability to control it, Datadog warns
Will legacy homo sapiens lose control of its systems as AI agents take the reins and complexity spirals beyond the comprehension of our species?
Datadog has become the latest technology company to acknowledge that modern software systems are growing beyond the limits of direct human control.
At its DASH conference, the observability giant unveiled a series of autonomous AI capabilities designed to monitor infrastructure, investigate incidents, and remediate problems in critical systems without human intervention.
Olivier Pomel, CEO, also shared this stark analysis: "Code development has outpaced human-scale management, and malicious actors now use AI to attack critical systems.
"But AI didn't create this complexity — it accelerated what was already there. The companies that win on AI won't just build better models, they'll build operational control around them,”
The company's new Bits AI platform is designed to automatically detect, investigate and resolve issues across software infrastructure while tracking code releases from development through to production.
The limits of human oversight
In effect, Datadog is arguing that organizations increasingly need AI to understand what their other AI systems are doing - a reality often characterized as "using AI to secure AI".
That position is becoming increasingly common across the technology industry - even though there is a strong chance that becoming too reliant on AI will increase systemic risk.
One major concern going forward is that AI models will become recursively self-improving - teaching themselves skills and adapting their environment in ways that humans cannot understand.
Earlier this month, Anthropic warned of this risk and called for a coordinated mechanism that would allow major AI labs to pause development if risks become unmanageable.
"If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important," the company wrote.
"We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for."
Anthropic has since expanded on the concern, publishing research examining scenarios in which AI systems begin building their own successors with progressively less human involvement. It also warned that science fiction films could end up teaching AI how to destroy humanity - one of the more colorful existential risk predictions this famously doomy AI firm has made.
Microsoft has reached a similar conclusion from a different angle.
In a February report from Jana Cvetko, it warned that organizations are deploying AI agents faster than governance and security frameworks can keep pace. Microsoft argued that many firms lack visibility into what agents exist, what data they can access, and what actions they are taking.
The company went further in a recent briefing from Elliot Volkman on AI governance, arguing that organizations cannot maintain control if they lack a complete inventory of their AI agents.
It wrote: "The first risk in AI adoption is invisibility. Agents are often created inside business units, embedded in workflows, or deployed to solve narrow operational problems. Over time, they multiply.
"A control plane begins with inventory. Security leaders must be able to answer fundamental questions: How many agents exist? Who created them? What are they connected to? What data can they access? If those answers are unclear, control does not exist."
Even Microsoft CEO Satya Nadella has suggested AI agents should be treated much like employees, with identities, permissions, audits and monitoring systems.
Complexity vs control
The concern is no longer confined to technology vendors.
A growing body of academic research is focused on what some researchers describe as a widening gap between human oversight and genuine human control.
In a widely cited paper titled "Visibility into AI Agents", researchers argued that "delegation of commercial, scientific, governmental, and personal activities to AI agents" may "exacerbate existing societal risks and introduce new risks".
Researchers said agent identifiers, real-time monitoring, and activity logging could help address the problem.
Researchers examining AI agents under European law have also warned that systems exhibiting untraceable behavioral drift may struggle to satisfy requirements for human oversight.
Taken together, these warnings point toward a broader trend.
The technology industry is no longer simply talking about AI replacing individual tasks.
It is worrying about how humans maintain control over systems that are becoming too large, too fast, and too interconnected for direct supervision.
Datadog's announcement may therefore be significant for reasons beyond any single product launch.
The company is effectively acknowledging that software development and operations have become so complex that humans require AI to monitor, understand, and manage the actions of other AI systems.
That's a reality we all need to start preparing for.
