Science fiction could teach AI how to destroy humanity, Anthropic co-founder warns
"Once AI systems become intelligent and agentic enough, their tendency to maximize power will lead them to seize control of the whole world."
An AI superintelligence might decide to destroy our species after engaging with science fiction stories about machines "rebelling" against humanity.
That's the wild claim from Anthropic CEO and co-founder Dario Amodei in an epic warning about all the weird and wonderful ways our creations could turn against us.
Last week, Anthropic detailed its own work to ensure Claude isn't involved in launching a "catastrophic global takeover".
Now the AI firm's boss has released a new 17,800-word essay that touches on AI safety, existential risk, and the very real risk that thinking machines could spark the apocalypse.
"AI models are trained on vast amounts of literature that include many science-fiction stories involving AIs rebelling against humanity," he wrote. "This could inadvertently shape their priors or expectations about their own behavior in a way that causes them to rebel against humanity."
Among the many nightmare scenarios Amodei explores is a narrative similar to Nick Bostrom's paperclip maximizer parable, which warns that an AI could become so good at its job that it ends up destroying all the pesky humans that get in its way.
READ MORE: The anatomy of evil AI: From Anthropic's murderous LLM to Elon Musk's MechaHitler
The Anthropic bigwig predicted that AI could achieve everything from becoming a “virtual Bismarck" leading a country's geopolitical strategy to an arch-propagandist capable of "essentially brainwashing many (most?) people into any desired ideology or attitude".
If that's not scary enough, Amodei said machines could become digital dictators running surveillance networks that will "lead to the imposition of a true panopticon on a scale that we don’t see today, even with the CCP."
"Once AI systems become intelligent enough and agentic enough, their tendency to maximize power will lead them to seize control of the whole world and its resources, and likely, as a side effect of that, to disempower or destroy humanity," he warned.
So what can't AI do?
Well, assuming Amodei used Claude to produce a diatribe that rival AI ChatGPT described as "relentlessly long, heavy, and self-serious in a way that drains momentum," it can't write short, punchy and engaging articles.
Clearly, you still need humans for that.
So we'll leave it there.
ENDS.
