“It doesn’t matter”: Big AI refuses to estimate p(doom) at NYC Council hearing

Big AI companies have dramatically refused to provide an estimate for p(doom) — the existential risk posed by artificial intelligence — after being put under oath at a major New York City Council hearing examining whether stronger safeguards are needed to control the technology.

Representatives of OpenAI, Anthropic, Google, and Meta were each asked to quantify the risk posed by AI in a worst-case scenario during a rare Committee of the Whole hearing on October 5, which brought together the full 51-member City Council to examine the dangers of rapidly advancing AI. 

The questioning was led by New York City Council Speaker Julie Menin, who has spearheaded a package of 10 proposed AI laws covering measures including independent third-party validation, whistleblower protections and mechanisms allowing humans to shut down dangerous systems. 

The proposals include legislation requiring third-party validation and a shutdown capability for certain AI models

Menin's first target was OpenAI.

“As you all four are under oath, we want to hear how each of you quantify the risk imposed by AI in the worst case catastrophic scenario,” Menin said.

OpenAI’s Morgan Dwyer, head of policy development and operations, replied: “I don't know. I also don't think it matters whether it's 1% or 10% or a 20% chance that something catastrophic will go wrong. None of these levels is remotely acceptable, we should not train models that we cannot make an extremely strong case that we can keep under human control.”

Menin immediately pushed back and said: "To say you don't know and it doesn't matter is flippant at best. This idea that if you're a pharmaceutical company and you're developing a drug, and you say, ‘I don't know if it's going to kill people,’ I honestly am incredulous at that answer.”

She added: "If you can't quantify what the safety risk is, how do you know that the product is safe?”

Misanthropic AI development?

Menin then put the same question to Logan Graham, head of Anthropic’s Frontier Red Team, who has previously described his team’s job as building an “early warning system” for advanced AI risks. 

Graham said Anthropic has spent four years focusing on issues like biological security and cybersecurity, but did not provide a p(doom) estimate.

He said: "Policies like responsible scaling policies have so far been fairly effective at taking a cautious approach. However, now we have bigger and bigger threats that might come if we don't properly safeguard them as the models get more and more capable.”

“You say that you want to be more transparent, but you're not really addressing the question. I mean, the question is, can you quantify the risk of something cataclysmic happening?”

Graham pointed to Anthropic’s risk reports, threat modelling and internal forecasting, but did not provide a percentage. 

“Troubling at best"

Menin next turned to Shane Cahill, Meta’s AI policy director, who also leads privacy and AI legislation work at the company. 

Cahill said: "We have a multi-layered approach to this, where teams evaluate before deployment, including adversarial testing, applying safeguards, and deploying when only risks are mitigated. We set this out in our public Meta superintelligence scaling framework and our preparedness reports that go alongside the release of models.”

When asked directly to quantify the catastrophic risk of AI, Cahill replied: “Speaker, I don't wish to be imprecise in relation to the question of quantifying, I don't have our framework in front of me right now, but I'd be pleased to follow up with you afterwards.”

Menin called the failure to provide a number “troubling at best.” 

READ MORE: The UK has no power to stop dangerous AI models being unleashed, Parliament warns

Finally, Alice Friend, Google’s head of AI and emerging technology policy, argued that assigning probabilities to future catastrophes was not yet scientifically rigorous. 

Friend said: “We take catastrophic risks and their possibility extremely seriously, and we address that in a few ways. One way is that we perform research into producing our own safety frameworks for artificial general intelligence, so we have an AGI safety and security approach, we also have a frontier safety framework, which is our protocols for monitoring our models under development for dangerous capabilities."

Google said its benchmarks can measure model capabilities, but catastrophic risk cannot yet be quantified rigorously because there is no reliable historical reference point on which to base a probability estimate.

Menin then summed up the exchange: “So, basically I'm going to take it then that neither of the four of you, no company here can quantify the risk of something cataclysmic happening.” 

What's your p(doom)?

The refusal is striking because senior figures in the AI industry have previously been willing to put numbers on broadly similar risks, although definitions of p(doom) vary substantially and the estimates are not directly comparable.

Anthropic CEO Dario Amodei, for instance, has repeatedly discussed a roughly 10% to 25% chance of outcomes going catastrophically wrong on a civilizational scale. In a more recent interview, he was confronted directly with his previous estimate of a “10 to 25% chance of civilizational collapse” and did not disavow it.

Amodei has stressed that such figures cover a wider range of disastrous outcomes than literal human extinction, which is one reason comparisons between different p(doom) estimates can be misleading

Elon Musk, whose SpaceXAI operation did not appear at the hearing, has previously put the probability that AI “goes bad” at around 10% to 20%.

READ MORE: AI loss of control is already “in the rearview mirror,” says MIT professor Max Tegmark

Google DeepMind chief Demis Hassabis has warned that advanced AI systems could become difficult to control, but has said the risk is currently too uncertain to quantify.

In a 2025 interview, Hassabis said: "You know, I don’t have a p(doom) number because I think it would imply a level of precision that is not there. 

"I don’t know how people are getting their p(doom) numbers. I think it’s a little bit of a ridiculous notion. What I would say is it’s definitely non-zero and it’s probably non-negligible. So that in itself is pretty sobering."

Meta chief Mark Zuckerberg, meanwhile, has taken a more optimistic and market-led view of existential AI risk.

Zuck has argued that competition between AI systems and companies can help keep power in check, and that policymakers should be wary of measures that slow development or centralize superintelligence.

The exchange over p(doom) exposed a simple question at the center of AI safety.

Has Big AI really been overegging the danger as part of some bizarre "doomfishing" marketing campaign - or are we actually doomed?

If the latter is true, it would certainly be a rather awkward confession for any corporate mouthpiece. So perhaps you can't blame them for dodging the question.

Follow Machine on LinkedIn