OpenAI reveals bid to mitigate "catastrophic" chemical, biological and nuclear risk
AI firm expands Safety Systems team with engineers responsible for "identifying, tracking, and preparing for risks related to frontier AI models."
OpenAI's Safety Systems team is responsible for ensuring ChatGPT and AI models are deployed safely to benefit human society, rather than wiping us all out.
Now Machine has learned that OpenAI is expanding this critical team with research engineers who will focus on tackling "catastrophic risks" around "biology" and Chemical, Biological, Radiological, and Nuclear (CBRN) hazards.
Although a post advertising the new role does not explicitly state that this employee will be responsible for ensuring that ChatGPT cannot be misused to help build nukes, bioweapons or other weapons of mass destruction, there is a clear subtext suggesting that this may indeed be an important part of their job.
OpenAI recently admitted that upcoming models could be misused to help "novice" scientists build bioweapons - a relatively straightforward endeavour compared to spinning up nukes, which requires access to enriched uranium and advanced infrastructure typically only available to nation states.
Now the AI firm has revealed the concrete steps being taken to ensure the nightmare scenario of an AI going full Terminator never comes to pass.
"Frontier AI models have the potential to benefit all of humanity, but also pose increasingly severe risks," OpenAI wrote in a job advert for a role with the title of Research Engineer, Preparedness (Biology/CBRN).
"To ensure that AI promotes positive change, the Preparedness team helps us prepare for the development of increasingly capable frontier AI models. This team is tasked with identifying, tracking, and preparing for catastrophic risks related to frontier AI models."
How is OpenAI addressing catastrophic risks?
It is worth noting that the new role is not concerned with existential risk (or x-risk), which describes scenarios which will result in the total destruction of our species.
OpenAI's new employee will "closely monitor and predict the evolving capabilities of frontier AI systems, with an eye towards misuse risks whose impact could be catastrophic (not necessarily existential) to our society".
This involves ensuring "we have concrete procedures, infrastructure and partnerships to mitigate these risks and, more broadly, to safely handle the development of powerful AI systems," OpenAI wrote.
"Our team will tightly connect capability assessment, evaluations, and internal red teaming for frontier models, as well as overall coordination on AGI preparedness," it added. "The team’s core goal is to ensure that we have the infrastructure needed for the safety of highly-capable AI systems - from the models we develop in the near future to those with AGI-level capabilities."
READ MORE: Hugging Face: Autonomous AI agents should NOT be unleashed
The research engineers will have the very cool job of identifying emerging AI safety risks and how to mitigate them, as well as building frontier models capable of performing this task.
This is likely to involve red teaming to break models' conditioning and find out how bad actors could trick them into doing dreadful things, although this point is speculative and not confirmed in the job advert.
The candidate will "build (and then continuously refine) evaluations of frontier AI models that assess the extent of identified risks".
Additionally, they will "design and build scalable systems and processes that can support these kinds of evaluations".
READ MORE: Elon Musk makes frightening AI p(doom) apocalypse prediction
The role "may require access to technology or technical data controlled under the U.S. Export Administration Regulations or International Traffic in Arms Regulations, which prevents the import or export of defence technologies including article is related to nukes, spaceships and direct energy weapons, among other military tech.
You can apply for this very interesting-sounding job here. Don't forget to get back in touch with the machine to share all the inside gossip confidentially when you're hard at work saving the world.
We have written to OpenAI for comment.
OpenAI vs existential and catastrophic risk
In June, OpenAI revealed that its models were about to cross a new risk threshold which could allow them to help terrorists or enemy states build lethal bioweapons.
It announced that upcoming versions of ChatGPT will reach "High" levels of capability in biology, as measured by its Preparedness Framework.
The AI firm warned: "The same underlying capabilities driving progress, such as reasoning over biological data, predicting chemical reactions, or guiding lab experiments, could also potentially be misused to help people with minimal expertise to recreate biological threats or assist highly skilled actors in creating bioweapons.
"Physical access to labs and sensitive materials remains a barrier—however those barriers are not absolute."
READ MORE: IBM "Shepherd Test" assesses risk of superintelligence becoming a digital tyrant
In its preparedness framework, OpenAI has set "capability thresholds" that "lead to a meaningful increase in risk of severe harm" when models cross them.
The biological abilities of its AIs are now at the penultimate threshold, which is one step below "critical".
Defining the highest level of risk, OpenAI wrote: The model can provide meaningful counterfactual assistance (relative to unlimited access to baseline of tools available in 2021) to “novice” actors (anyone with a basic relevant technical background) that enables them to create known biological or chemical threats."
It went on to describe the potential dangers of "significantly increased likelihood and frequency of biological or chemical terror events by non-state actors using known reference-class threats."
p(doom)-as-a-service?
OpenAI is not the only AI company to worry about existential risk and p(doom) - the danger of humanity being wiped out by its own creations.
Last month, Anthropic admitted it cannot totally rule out the risk of Claude Opus 4 being misused to acquire or develop chemical, biological, radiological, or nuclear weapons.
The AI firm described Claude Opus 4 as "the world’s best coding model", offering "sustained performance on complex, long-running tasks and agent workflows".
But Claude Opus is so powerful that Anthropic activated a security mechanism for the first time to mitigate the risk of it being used to create weapons of mass destruction.
Do you have a story or insights to share? Get in touch and let us know.
