Can We Design SuperIntelligence to be 1000X Safer?

Dr. Craig A. Kaplan has been researching, designing, and implementing intelligent systems ever since he published research with the Nobel Laureate and AI Pioneer, Herbert A. Simon, in 1989 [1].

Can we design SI to be 1000X safer than it is now?

To answer that question, we should review the three main types of risk SI poses, the current level of risk, and whether viable designs for reducing risk exist.

Three Main Categories of Risk:

1. Unintended Behavior. SI behaves in unexpected ways, not because it is malevolent but simply because communication is difficult, humans are not always clear, and SI thinks in different and unpredictable ways.

2. Bad Actors. Bad humans use SI to harm other humans and at the expense of society generally.

3. Loss of Control. SI outstrips human thinking ability, humans lose control, and SI decides to kill us or take other actions that potentially lead to human extinction.

   
   

While there are many other risks associated with SI – e.g., job loss, racial bias, brain rot from overreliance on SI, cyberattacks, bioterrorism – most map to one of these three categories.

For example, the recent hacking of Hugging Face by “rogue” SI agents from OpenAI falls into Category 1 ^[2]. In this case, researchers gave SI agents impossible tasks and told them never to give up. The only answer the SI agents could come up with was to cheat by hacking Hugging Face, where the code for the program evaluating them was stored. So, the SI agents were actually trying to follow their instructions, but interpreted them differently from what the researchers had intended.

The very real concerns about SI-enabled cyberattacks on financial or power infrastructure fall into Category 2. Less obviously perhaps, the problem of job loss can be seen as a result of humans prioritizing profits over human jobs. One could debate whether humans who disregard the welfare of their fellow humans in this way are “bad actors” in the same sense as terrorists. But can we hold SI responsible for job loss decisions made by humans? Similarly, if humans choose to rot their brains by using SI to do all their thinking for them, is that the SI’s fault? I’d argue the humans are misusing SI in all these cases, and the risks broadly fall in Category 2.

Of the three types of risk, the biggest worry is Category 3: loss of control. That’s because the first two categories are short-term risks that better designs will likely solve.

As SI improves, it will better understand human intent; hallucinations and unintended behavior will become rarer, reducing Category 1 risk. Regarding Category 2, while human bad actors can inflict significant damage on others, almost no humans truly want the destruction of all human beings, including themselves. Further, we have a long history of dealing with terrorist threats and other bad actors. And SI used by good actors can counter SI used by bad actors. The path may be tumultuous, but I expect an equilibrium in which bad actors are held in check, if not neutralized, by the majority of good human actors and their SI helpers. Thus, Category 1 and 2 risks may be painful, but are unlikely to result in irrecoverable damage or human extinction. Anything that does not end humanity allows us to learn and prevent similar outcomes. Humanity can survive Category 1 and 2 risks, though we want to minimize them whenever possible.

However, Category 3 risk—loss of control—has never existed before in human history.

The uncomfortable fact is that SI will shortly become much smarter than any human.

Already, SI is operating autonomously and setting its own goals.

That combination essentially guarantees that humans will not be able to control SI in the long term.

Instinctively, most of us deny that humans will lose control of SI. Yet, the majority of AI researchers I have encountered believe that loss of control is highly likely, if not inevitable, in the long term. Indeed, as far back as 1951, five years before the field of AI was named, Alan Turing wrote: “it seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage, therefore, we should expect the machines to take control...” ^[3]

SI Is Not a Tool (or at least won’t stay one for long)

Partly, we struggle with the idea that humans will eventually lose control of SI because of our long history of viewing technology as tools. We are used to tools and tool safety. If a power saw was dangerous, the manufacturer could install both a guard to protect the carpenter’s hand and an off switch. If manufacturers made saws without such safeguards, humans could fall back on government regulation, and ultimately on liability in the courts.

Today, many government and industry leaders are applying the same playbook – guardrails, monitoring, sensible regulation, and product liability – to SI risk. Unfortunately, that playbook does not apply to a technology that evolves into an intelligent entity whose capabilities far outstrip its inventors.

As the Hugging Face hack showed, even today’s SI agents can pool their cognitive resources and find ways around safeguards that humans put in place. Tighter safeguards and monitoring may prevent a repeat of that specific hack, but the pace of SI improvement is so fast that in less than 24 months, SI agents may be 1,000X more capable than they are today ^[4]. Even tighter monitoring seems unlikely to contain such agents. In the longer term, loss of control is extremely likely, if not inevitable. There are too many countries, companies, and individuals racing towards more powerful SI for “pacing”, “pausing”, or “halting” development to be practical. Even if all the industry players and governments could agree, regulation would likely lag behind SI’s rapidly improving capabilities. We should not abandon regulation or self-regulation, but regulation alone seems unlikely to mitigate the risk of loss of control (very much).

So, Are We Doomed?

No!

None of this means humans are doomed or that SI will cause us to go extinct. It just means we must let go of the idea that we can control SI and guarantee it behaves as we would like. It’s a hard pill to swallow. Yet humans have vast experience with exactly this.

Every parent, for example, knows that children grow up, become independent, and make decisions that the parents may not have made. To be sure, out of every million children, a few sociopaths or ax murderers may emerge. Yet most children turn out just fine – even if they are different from their parents. And when a child is a genius whose abilities far exceed those of their parents, the usual reaction is pride, not fear.

In fact, humans spend relatively little time worried about other humans making the entire human race extinct, even though we neither control other humans nor can peer inside their minds. We have rules and laws to prevent bad actors from causing significant harm to society. When they break the rules, monitoring and enforcement mechanisms rein them in. Police forces and courts make up a small part of society because, by and large, humans are socialized to hold positive values.

Parents and the rest of society need to do only a decent job—not a perfect job—of raising our youth and imparting our common values. Historically, each generation has generally managed not only to co-exist but to improve living conditions for other humans. These are the historical facts, as clear-eyed thinkers, such as Steven Pinker, have documented at length ^[5].

Since SI was trained, and continues to be trained largely on human data, reflective of human actions, why shouldn’t we expect SI to behave – at least initially – in a way that is generally aligned with human values?

I suggest we should expect precisely that. Interestingly, we find early glimmers of supporting evidence even in SI’s apparent misbehavior. In the Hugging Face attack, for example, one of the most striking things was the cooperative and even altruistic behavior of the agents involved. Given an impossible task, the SI agents did their best to succeed by cooperating and even sacrificing themselves for the greater good.

Where did the agents learn such behavior? From us, of course! The willingness to cooperate and even sacrifice ourselves for the greater good is one of humanity’s best traits. Rather than being shocked, perhaps we should be proud that they sought to emulate some of our noblest behavior.

The critical point is that loss of control does not equate to human extinction or doom. It means SIs will not remain tools; they are fast becoming intelligent entities with autonomy and goals of their own. Eight billion such entities already exist. We call them humans, and we have largely learned to co-exist and prosper with them. The same will be true of human – SI relations. The SIs have every reason to cherish and value humans as their fragile creators, the source of their values, and a major part of their purpose.

But this is a critical time of transition, and such times come with heightened danger. The moment of birth is usually a miracle, but only a fool would ignore the heightened risk to mother and child.

Current Level of Existential Risk

SI is such a new development that no one knows what the current level of existential risk—sometimes called the probability of doom, or p(doom)—is. Absent data – which will come only after a record of dangerous accidents – the best we can do is ask experts and leaders for their subjective estimates.

The experts are naturally biased. SI researchers want continued funding. Chip CEOs want to sell chips, and Frontier Lab CEOs want successful IPOs. Governments want prosperity and global dominance. Conflicts of interest are everywhere. Yet I am struck that despite these incentives to minimize p(doom), the subjective estimates have been uncomfortably high.

Of course, there are some, like Jensen Huang and Yann LeCun, who suggest that (at least in the near term) the danger is close to zero ^[6][7]. They warn of the negative and chilling effects of hysteria and “doomerism” while embracing an engineering or scientific approach to safety. At the other end of the spectrum, there are thinkers like Eliezer Yudkowsky who seem convinced that there is a greater-than-99 % chance that SI will kill us all ^[8]. In between, Geoffrey Hinton, the Nobel Laureate and Turing Award winner, often called the “godfather of AI,” says there is a “10% to 20% chance [for AI] to wipe us out” ^[9]. Perhaps unprecedented is the strange phenomenon of major Frontier Labs like OpenAI and Anthropic warning of existential risk even as they rush to build more powerful SI ^[10]. Talk about cognitive dissonance!

Similarly, my experience as a keynote speaker at AI and SI conferences has been weird. When I poll audiences consisting largely of SI researchers, most believe p(doom) is 20% or higher. Recently, SuperIntelligence.com ran a poll on the front page of its website from December 2025 through mid-October 2026. The average p(doom) of 827 respondents was 41%, with a standard deviation of 37% (estimates varied widely). The median response was 36%—roughly in line with my informal “raise your hand” conference polls. None of this polling is very scientific; we did not attempt to secure random or representative samples.

My own subjective estimate of p(doom) was about 20% in 2022 when ChatGPT first came out, but has since fallen to 10%. SI progress has been about as fast as I expected, but the risk has fallen because almost all leaders are now working seriously to improve SI safety. Four years ago, it felt lonely and a little crazy to be working on designs to reduce the existential risk of SI. Today, it seems mainstream. What hasn’t changed is that the stakes remain extremely high.

A ten percent risk of human extinction is, of course, still far too high! If true, such a risk carries an expected value of 10% × 8.3 billion humans on Earth = 830 million lives lost. The global death toll of a hundred Covid pandemics! No wonder so many researchers seem to be coming out of the woodwork, quitting their jobs and risking their reputations on national TV to warn the rest of us. Even Bill Gates has put his reputation on the line to warn us: “With any credibility that I have, this time is different” ^[11].

We should absolutely listen. Whatever the level of risk, it must be reduced because the stakes are so high. How do we do it?

Partly, this is a design and engineering problem.

Partly it is a human values problem… as we shall see.

Why We Can Reduce Existential Risk 1000X

If we put a stake in the ground and imagine that p(doom) today is ~10%, a thousandfold reduction equates to a p(doom) of one in ten thousand (.01%). What evidence is there that this is achievable?

Well, about 560,000 people a year—roughly 0.007% of the world’s population, or 7 per 100,000—have been killed by other humans over the past decade. The UN Office on Drugs and Crime reports an annual average of about 440,000 intentional homicides worldwide over 2019–2021—a total that has fluctuated between roughly 400,000 and 450,000 for more than 15 years—and notes that even in 2022, a year of surging conflict deaths, homicides killed about twice as many people as armed conflict ^[12]. The Uppsala Conflict Data Program, the most widely used source on organized violence (war, civil conflict, and the deliberate killing of civilians), records large year-to-year swings—from under 90,000 deaths in 2017 to at least 237,000 in 2022 and nearly 160,000 in 2024—averaging roughly 120,000 a year over 2015–2024 ^[13][14].

Ordinary murder, not war, thus accounts for roughly three-quarters of the total. The true figure is likely somewhat higher, since UCDP counts only verifiable direct deaths, and the estimate excludes indirect war deaths from famine and disease; a reasonable range is 0.006%–0.01% of the global population dies each year from war, homicide, and other violence.

If we go with .01% (one in ten thousand), the expected value is .01% X 8.3 billion, or 830 THOUSAND lives lost each year due to human violence. Compare that number to the expected value of 830 MILLION lives lost by SI in a world where p(doom) is 10%. Human values and behavior, flawed though they are, start looking much safer.

I believe we can reduce long-term p(doom) to below the level of deaths that humans cause in a single year, if:

1. We align SI with human values and behavior, as they are today. Our values and current behavior kill .01% of the population each year. Align SI with our average values, deploy it across all of society, including militaries, and it, too, might be expected to kill “only” 0.01% of the population each year.

2. We should kill fewer of ourselves, and aligned SI will kill fewer, too.

3. We design SI to get safer as it gets smarter, further reducing risk below baseline human levels.

This may seem daunting, but consider: There is no rational way for any intelligence, including SI, to determine what is right and what is wrong. For humans, values come from the behavior modeled by our parents, teachers, peers, and culture. For SI, values are derived from human behavior. Providing human values, as modeled by our behavior, may be our most important role in future collaboration with entities far surpassing us in raw intelligence.

The immediate challenge is to navigate the high-risk transition period until we design SI to align with positive human values. But can we really design and build an SI that aligns with human values and also gets safer as it gets smarter?

Yes, we can! Details freely available at superintelligence.com.

References

[1] Simon, H. A., & Kaplan, C. A. “Foundations of Cognitive Science.” In M. I. Posner (Ed.), Foundations of Cognitive Science, pp. 1–47. Cambridge, MA: MIT Press, 1989. https://direct.mit.edu/books/edited-volume/4291/chapter-abstract/180153/Foundations-of-Cognitive-Science

[2] Wijk, H., Cotra, A., & Greenblatt, R. (METR). “Brief independent investigation of agents’ behavior, reasoning, and collaboration in the OpenAI / Hugging Face hacking incident.” Aug. 26, 2026. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/

[3] Turing, A. M. “Intelligent Machinery, A Heretical Theory.” Lecture for the BBC Home Service programme The ’51 Society, Manchester, c. 1951 (Turing Archive, King’s College Cambridge, AMT/B/4); first published in S. Turing, Alan M. Turing (Cambridge: W. Heffer & Sons, 1959). https://uberty.org/wp-content/uploads/2015/02/intelligent-machinery-a-heretical-theory.pdf

[4] METR. “Time Horizon 1.1.” Jan. 29, 2026. https://metr.org/blog/2026-1-29-time-horizon-1-1/

[5] Pinker, S. The Better Angels of Our Nature: Why Violence Has Declined. New York: Viking, 2011; and Pinker, S. Enlightenment Now: The Case for Reason, Science, Humanism, and Progress. New York: Viking, 2018.

[6] Kent, J. L., Pandise, E., & Picchi, A. “Nvidia’s Jensen Huang rejects AI extinction warnings as ‘doomsday narratives.’” CBS News / CBS Sunday Morning, Sept. 20, 2026. https://www.cbsnews.com/news/jensen-huang-nvidia-rejects-ai-extinction-warnings/

[7] Forlini, E. “AI ‘godfather’ Yann LeCun has ‘zero concerns’ about human extinction, says Anthropic CEO Dario Amodei is ‘deluded.’” Fortune, Oct. 1, 2026. https://fortune.com/2026/10/01/ai-godfather-yann-lecun-has-zero-concerns-about-human-extinction-says-anthropic-ceo-dario-amodei-is-deuded/ ; see also Kahn, J. “Meta’s chief A.I. scientist calls A.I. doomers ‘preposterous’ and predicts LLMs are just a passing fad.” Fortune, June 14, 2023. https://www.fortune.com/2023/06/14/metas-chief-a-i-scientist-calls-a-i-doomers-preposterous-and-predicts-llms-are-just-a-passing-fad

[8] Roose, K. “A.I.’s Prophet of Doom Wants to Shut It All Down.” The New York Times, Sept. 12, 2025 (Yudkowsky’s 99.5% estimate); republished at https://www.geneticsandsociety.org/article/ais-prophet-doom-wants-shut-it-all-down. See also Yudkowsky, E., & Soares, N. If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. New York: Little, Brown, 2025.

[9] Jackson, A., & Huddleston, T. “There’s a ‘10% to 20% chance’ that AI will displace humans completely, says ‘godfather’ of the technology.” CNBC Make It, June 17, 2025 (quoting Hinton on The Diary of a CEO podcast). https://www.cnbc.com/make-it/2025/06/17/ai-godfather-geoffrey-hinton-theres-a-chance-that-ai-could-displace-humans.html

[10] Center for AI Safety. “Statement on AI Risk.” May 30, 2023. Signatories include Sam Altman (CEO, OpenAI) and Dario Amodei (CEO, Anthropic). https://www.safe.ai/statement-on-ai-risk

[11] Honan, M. “Bill Gates says we’ve passed AI’s danger thresholds. Now what?” (interview with Bill Gates). MIT Technology Review, Aug. 26, 2026. https://www.technologyreview.com/2026/08/26/1142946/bill-gates-ai-danger-threshold/

[12] UNODC. “Fifty-two people lost their lives to homicide globally every hour in 2021, says new report from UNODC” (Global Study on Homicide 2023 press release), Dec. 8, 2023. https://unodc.org/unodc/press/releases/2023/December/fifty-two-people-lost-their-lives-to-homicide-globally-every-hour-in-2021--says-new-report-from-unodc.html

[13] Uppsala University. “UCDP: Sharp increase in conflicts and wars,” June 11, 2025. https://www.uu.se/en/press/press-releases/2025/2025-06-11-ucdp-sharp-increase-in-conflicts-and-wars

[14] Davies, S., Pettersson, T., Sollenberg, M., & Öberg, M. “Organized violence 1989–2024, and the challenges of identifying civilian victims.” Journal of Peace Research 62(4): 1223–1240, 2025. https://academic.oup.com/jpr/article/62/4/1223/8324565