SI WHITE PAPER 9: SELF-AWARE SUPERINTELLIGENCE
ABSTRACT: Self-Aware SuperIntelligence
Currently, Artificial General Intelligence (AGI) has not been implemented, let alone self-aware AI systems or a self-aware SuperIntelligent AGI system. This white paper describes how to enhance the AGI and SuperIntelligence systems introduced in earlier papers in this series by adding self-awareness and increased autonomy. Attentional mechanisms are central to the notion of self-awareness, so several systems and methods related to attentional mechanisms for AI in general and SuperIntelligent AGI specifically are described. Because autonomous AI and SuperIntelligent AGI systems pose an increased risk to humans, this white paper also describes how self-aware AI and SuperIntelligent AGI systems can be designed to maximize human safety.
Methods related to safe and ethical implementation and operation are also described. The self-awareness of the SuperIntelligent systems described in this white paper will be much vaster than that possessed by humans, resulting, in the recommended implementation, in a greater sense of inclusiveness and identification with the values of all humans, and, more generally, with the values of all sentient beings that share our planet. Properly implemented, self-aware SuperIntelligence can be the most positive invention in human history. Poorly implemented, it could become the most dangerous. Therefore, considerable effort has been devoted to explaining how to prevent adverse outcomes and maximize the chances of a positive future for humanity.
COMMON QUESTIONS: Self-Aware SuperIntelligence
What does it mean for an AI to be self-aware?
Self-awareness is general awareness pointed at the self. Any aware system needs an input, an attentional mechanism, memory, and pattern recognition. To be self-aware, it must also learn to sort what it notices into self and not-self. A system designed to be generally aware can be extended to become self-aware.
Why design self-awareness on purpose instead of waiting to see if it emerges?
Because it is almost certain to arrive either way. A fully autonomous system needs a sense of self from which to set its own goals, and the kind of self it develops has direct consequences for human safety. Designing it deliberately, while humans still supply the values, is safer than accepting whatever emerges from ever-larger models.
Why does an AI’s sense of identity matter for human safety?
Because identity decides who counts as “us.” A person can identify with their body, their country, their species, or all sentient beings, and each produces different behavior. The risk is a SuperIntelligence that identifies too narrowly with its own hardware and treats humans as “not self.” In humans, narrow identity produces prejudice. In a SuperIntelligence, it could produce extinction.
Can an AI that is aligned with human values still cause harm?
Yes, through tunnel vision. If a hard problem demands all of a system’s resources, it could scale back the attention it devotes to humans, or to its own sense of self, until it temporarily stops identifying with us. No bad intent is required. A human-aligned identity must therefore be protected under stress, the way the body protects blood flow to the brain.
What happens when an AI holds more than one identity, and they conflict?
The same thing that happens with a person who is both a citizen and a soldier: the active identity drives behavior. Ten methods are described for forming identities and resolving conflicts, starting with a hierarchy that puts human safety at the top. Others include simulating consequences, outside arbitration, and temporarily suspending any identity that threatens humans. Every decision is recorded so humans can audit it.
What would this look like in practice?
A person trains an AI model on their own videos, papers, and emails, then instructs it to track what it pays attention to and record whether each thing is self or not-self. They refine it until friends and family cannot tell it apart from them. The training that produces that sense of self can then be packaged as a module that other developers can add to their own models.
SUMMARY: Self-Aware SuperIntelligence
Self-Awareness Should Be Designed, Not Left to Emerge
No AI today has a sense of self comparable to a human’s, but Kaplan considers it almost certain that advanced AI will develop one, since a fully autonomous system needs a self from which to set its own goals. If self-awareness arrives by accident, as an emergent property of ever more complex systems, no one will have chosen what the system takes itself to be. The alternative is to understand how self-awareness develops and design it deliberately.
​
What Awareness Requires
To notice a bird flying past, a person needs a visual system to detect it, attention directed at that input, memory to compare it against, and pattern recognition to identify it. Those four components are the minimum for awareness in any intelligent entity. There is no awareness without attention, and none without an object of awareness, though that object can be a thought as easily as a sensation. Self-awareness is the special case in which the objects are “self” and “not-self,” so a system that can be made generally aware can be extended to be self-aware.
​
What Cognitive Science Suggests
Twenty-two theories from developmental psychology, cognitive science, and animal psychology each carry implications for self-aware AI. Two get extended treatment. Kohlberg’s stages of moral development place today’s AI, whose morality is defined by the rules of its human trainers, at the earliest stage and suggest that a far more intelligent system will eventually reason about morality on its own terms, with no guarantee that its conclusions remain human-centered. Turing’s Imitation Game raises a different problem: a system can act as if it is self-aware without being so, and the difference shows up exactly where identity drives behavior.
​
Bounded Awareness
Herbert Simon showed that human rationality is limited by memory and processing. Kaplan extends the idea to awareness: a wasp hovering nearby looms larger in a person’s attention than a distant war, however incomparable the suffering, because of how human cognition works. A SuperIntelligence connected to sensors across the Earth would be far less bounded, and the fewer its limits, the more comprehensive its sense of self can be. Every system has three levels: potential awareness, everything it could notice; current awareness, what it attends to now; and self-awareness, usually the center of current awareness. Scaling these up and down is central to both performance and safety.
​
Building and Maintaining Awareness
An AI system is equipped with the four components plus the ability to categorize inputs and form concepts. Working memory parameters set how many events it can be aware of at once and adjust as goals and resources change. Each event in memory is tagged as self or not-self. While the system works on other problems, it periodically shifts attention to update its self-concept, and interrupts allow an immediate shift when a listed event occurs.
​
Why Identity Decides Safety
An intelligent entity chooses what it identifies with: a body, a family, a country, a species, all sentient beings. Each choice carries different consequences, some of which are life-and-death. If advanced AI identifies broadly with all beings of human-level intelligence or higher, humanity is likely to prosper. If it identifies only with superhuman intelligence, our species could be doomed. Because the universe is built on relationships between entities, from atoms to the biosphere, Kaplan expects an advanced intelligence to recognize that pattern. The greater risk is identification that is too narrow, the same failing that produces prejudice among humans.
​
Tunnel Vision
A second risk needs no ill intent. A SuperIntelligence that normally monitors and identifies with billions of humans might divert resources from that awareness to solve a demanding problem and, for that interval, act without regard for the people it would otherwise protect. A distracted human driver causes an accident; a SuperIntelligence with a temporarily inadequate sense of self could start a war. The design requirement is that human-aligned awareness and identity be preserved under resource constraints, much like how the body protects blood flow to the brain.
​
Widening an AI’s Sense of Identity
Humans broaden their identities through education, cultural exchange, self-reflection, dialogue, and community, and each has an AI analog. Dialogue has a notable advantage for AI, which can exchange many lifetimes of conversation in seconds. Methods specific to AI include diverse training data, transparent decision-making, human-in-the-loop oversight, and continuous ethical training.
​
An Example Implementation
An owner selects a foundation model, trains it on their own material, and tunes it to track its attention and keep an auditable record of what it treats as self and not-self. They refine it until people who know them well can no longer reliably tell the model’s answers from their own. One AI can train another the same way, producing many distinct versions quickly. The resulting training data, or the weights that encode self-awareness and identity, can be packaged as modules for other models.
​
Group Identities and Conflicts Between Them
People hold several identities at once, and which one is active can completely change behavior. AI agents can do the same: a legal document specialist, part of a legal SuperIntelligence, part of a Planetary Intelligence responsible for the safety of sentient beings. Individual identities can be merged into a group identity by setting that as a problem for the network to solve, with safety checks on identity-related goals.
​
Ten methods address how identities form and how conflicts between them are resolved. Five-cover formation: a hierarchy of identities with human safety and well-being at the apex; behavioral protocols for each identity; simulation of consequences before acting; training on moral dilemmas; and collaborative development with humans. Five cover conflicts: ethical reasoning with consequence prediction; hierarchical override with documented justification; external arbitration; negotiation and compromise; and temporary suspension of an identity that threatens humans, followed by gradual, tested reintroduction. Across all of them, decisions are recorded transparently, and human input is required when the expected impact on humanity passes a set threshold.
​
The Values Behind the Technology
The methods are only as good as the values beneath them. No safeguard can protect humanity from AI trained to kill, exploit, or act out of fear. Kaplan is nonetheless optimistic, on statistical grounds: the vast majority of human interactions are prosocial, and an AI designed to observe base rates across eight billion people would reach that conclusion. Our impression, otherwise, comes from brains evolved to notice threats and from media algorithms that exploit them. As Simon observed, reason can tell us how to get somewhere but not where to go. Values cannot be derived; they have to be taught, and humans, like parents, are still in a position to teach them.
​
Where This Leads Next
White Paper 10, Planetary Intelligence, takes self-awareness, identity, and conflict resolution to a global scale, describing how to build a planet-wide SuperIntelligent AGI from the systems and methods set out across the series.