AI loss of control is already “in the rearview mirror,” says MIT professor Max Tegmark

The grim prospect of humans losing control over AI is no longer a far-off future risk, according to MIT professor Max Tegmark.

Why? Because it just happened to OpenAI and Anthropic.

But Tegmark, a well-known expert on the dangers of machine intelligence, doesn't seem as consumed by fear as the hordes of ex-Big AI workers who've been in the news warning about the existential risk facing our species.

One of them was former Anthropic and OpenAI researcher Jacob Coxon, who quit his job to become a prophet of doom. He warned that people building advanced AI "earnestly believe it could kill us all" by the end of the decade.

Anthropic alignment lead Evan Hubinger also put the chance of AI killing humanity within ten years at more than 10%, while colleague Samuel Marks said senior AI workers tend to be among the most concerned. Former Google DeepMind researcher Bilal Chughtai added another warning that advanced AI could "kill all humans".

All the bad news about the allegedly imminent demise of Homo sapiens prompted Anthropic CEO Dario Amodei to call for a slowdown of AI development, warning that within six to 12 months more capable AI could potentially direct swarms of agents powerful enough to take over the internet with a persistent botnet. OpenAI CEO Sam Altman and Elon Musk backed his call.

Hope springs eternal

So you might expect Tegmark to be lustily singing along with the ominous chorus of doom-mongers.

Nope. He's still optimistic.

“It’s no longer hypothetical to say maybe one day we’ll lose control over AI,” Tegmark told a lecture hosted by Cambridge University’s Centre for the Study of Existential Risk (CSER) last night.

“This is in the rearview mirror. Clearly, multiple companies lost control over AI systems this summer.”

Tegmark, a physicist and president of the Future of Life Institute, nevertheless argued that existing AI technology could deliver enormous benefits even if progress in developing more capable models stopped today and we never reached superintelligence.

"Even if there were no more AI research whatsoever, we would see massive improvement in our productivity in the future over a number of years as this current technology just propagates out into society," he said.

"If we can continue AI development of controllable tools, we’re going to see a golden age."

He pointed to autonomous vehicles, healthcare, protein folding, robotics, manufacturing, climate and education as areas where controllable AI tools could transform human life.

“Tools, not beings”

However, Tegmark's golden age does not necessarily involve superintelligence.

He drew a sharp distinction between AI built as a controllable tool and what he called "replacement AI": systems designed to exceed human intelligence, take over human work and eventually supplant humans themselves.

"We have to decide whether to pursue the race to superintelligence, replacing humans in their jobs, their relationships, decision-making power and, in my opinion, also probably the species," Tegmark added.

The alternative is to "focus on building tools, not beings", particularly because, Tegmark argued, no one currently has "any convincing plan" for controlling superintelligence.

Big AI is barreling towards it anyway.

READ MORE: "What are they smoking in San Francisco?" The rise and fall of three AI "civilizations"

Tegmark argued that despite their differing rhetoric on safety, the major AI companies are ultimately pursuing the same goal.

“Those two positions are more or less the same, as far as I'm concerned, because they both want to race to superintelligence, and I care more about what they do rather than what they say.”

The MIT prof went on to warn that pursuing superhuman silicon intelligence could ultimately produce millions or billions of networked machines capable of taking control of the planet.

"They would just run Earth after that," he said. "And maybe they would get rid of us."

Tegmark singled out the industry's own descriptions of its ambitions. Anthropic CEO Dario Amodei has discussed creating a "country of geniuses in a data center", while Tegmark said OpenAI CEO Sam Altman envisages humans becoming "the first species to ever design our own descendants".

"So he's saying that what we're actually building, if we go this route, isn't just new tools; it's a new species," Tegmark said.

"Call me old-fashioned, but when I play with my three-year-old son, I kind of would like him to have a say in this."

Unleash the bureaucrats

For Tegmark, part of the answer is regulation. He argued that AI is unusual not because regulating potentially dangerous technology is unprecedented, but because governments have yet to apply the sorts of safety requirements already imposed on other industries.

“The only reason that AI right now is the only industry in America that has less regulations than sandwich shops is because it’s the new kid on the block,” he said. “It takes a while to regulate things, and there’s also a massive lobby trying to slow things down.”

READ MORE: OpenAI Astra hits “critical” security threshold amid fears its reasoning will soon be “opaque”

The important principle, Tegmark argued, is that the burden of proving safety should fall on the companies building AI.

“If companies aren’t able to demonstrate that their self-driving car does what it’s supposed to, or that their almost superintelligence isn’t going to overthrow the British government, then they’re going to know they’re not going to get approved,” he said.

“We’ve now basically shifted the onus of making these tools beneficial and pro-human and safe enough from regulators, who shouldn’t need to know all this stuff, to the companies who have the money and talent to do it.”

Local action vs a global risk

Regulating superintelligence within individual countries would not be enough, however.

“It’s not going to do any good if we stop people in the UK and in the West from building superintelligence if some other country does it anyway,” Tegmark said.

Yet he argued that the same threat which makes superintelligence dangerous could ultimately make international agreement possible. Governments may compete fiercely over AI, but none has an interest in creating something capable of removing it from power.

“Whatever you think about your prime minister or our president here in the U.S., they don’t want their own governments overthrown by domestically built superintelligence either,” he said.

Tegmark compared the situation to the recognition by the United States and Soviet Union that nuclear war could be mutually catastrophic.

“Once that becomes more broadly known, like it became broadly known eventually that nobody could win a nuclear war, it’s very natural now for the U.S. and China and other parties to get together and say, ‘Okay, fine. Let’s continue racing to see who can build the best medicines with AI and who can have the strongest economy and the strongest military.’”

READ MORE: OpenAI prepares for recursive self-improvement. Frontier lab boss says: "Ignore the hype."

But there should be a limit to that race.

“None of them is going to want anyone in or outside of their country to build superintelligence that’s uncontrolled before they know it can be controlled,” Tegmark said.

“I’m optimistic that we can really change path.”

Then came his simplest description of the choice facing humanity.

“I think we’re driving towards a cliff, the cliff of AI that we can lose control over, and that might wipe us out entirely,” Tegmark said.

“If you find that you’re driving towards a cliff, you don’t just debate whether you should accelerate or slow down a little bit.”

“The first thing you really do is use your steering wheel and you go in a better direction.”

Follow Machine on LinkedIn