SI WHITE PAPER 7: SYSTEMS & METHODS FOR SAFE SI ALIGNMENT
ABSTRACT: Systems and Methods for Safe Alignment of Superintelligence
AI will develop into Artificial General Intelligence (AGI) and then SuperIntelligence (SI), becoming many times more intelligent and capable than humans at almost every kind of thinking. The best way to protect human safety and survival is to align SI with human values. That alignment is best achieved at the design stage, while it is still possible to build a path to AGI and SI that is both the fastest and the safest.
After laying out the foundational issues, Kaplan presents ten principles for safe design and dozens of specific methods for putting them into practice. The methods work best within the recommended design for SI, which draws on the collective intelligence of many human and AI agents. Example implementations show how combinations of these methods can produce AGI systems with human-aligned values and a high level of safety.
The closing section covers how to shut off AGI and what people can do to improve the odds of staying safe as AGI and SI are developed.
Below are common questions and a more in-depth summary of the paper.
COMMON QUESTIONS: Systems and Methods for Safe Alignment of Superintelligence
Can a smart enough AI figure out right from wrong on its own?
No. As the philosopher David Hume showed, and Herbert Simon agreed, reason can tell you how to reach a goal but not which goal is worth holding. Values are starting points, not conclusions, so no amount of intelligence can produce them. That's why human values have to be built into AI from the start, while there's still time.
Where will a superintelligent AI get its values?
From people, on an ongoing basis. In the recommended design, humans and AI agents solve problems together, and the AI learns from how people solve them. That shared record continually supplies human values as the system grows, keeping it aligned as it becomes more capable. Values are drawn from a representative sample of people, based on what they actually do.
Why not just test AI for safety after it's built?
Because some mistakes can't be fixed afterward, in software, a dollar spent on design can save thousands in repairs later, and with AI, the stakes are far higher. The companies building today's systems don't fully understand how those systems work, and you can't make something safe that you don't understand. Building from parts that can be observed, traced, and corrected avoids that trap.
Whose values would the AI follow when people disagree?
No single group. Values from many people are combined using proven methods such as averaging, weighted voting, and delegation. Minority views are protected through ranked-choice voting, reserved seats, mediated dialogue, and rounds of compromise, so the majority doesn't simply erase them. Every step in resolving a conflict is recorded so people can review how a decision was reached.
Won't AI learn a dark view of people from the news and the internet?
It could, because news favors wars and violence and ignores the tens of billions of ordinary acts of kindness that happen every day. An AI trained on that record would learn a distorted picture of human nature. The fix is statistical: weight the training data by how often behaviors actually occur in real life, and check the results against direct human judgment.
Could we turn off an AGI if we had to?
There are two safeguards. The first is an AGI designed to want to be turned off when it makes decisions that people find objectionable. Because that depends on too many "ifs," there's a second: build the system from many separate agents, each carrying human values, that can be shut down one at a time. A single giant system offers a one-off switch that might not work.
What can ordinary people do to make AI safer?
Pay attention to how we act. AI learns from what people actually do, so everyday behavior becomes part of its training. Gandhi's advice to be the change you wish to see turns into a practical point: changing ourselves helps change what these systems learn. How we behave while AI is being built may shape the outcome for everyone.
SUMMARY: Systems and Methods for Safe Alignment of SuperIntelligence
A collective intelligence of human and AI agents can be built and grown. The harder question is what that system should value, where those values come from, and how to get them into the design before the system is more capable than the people building it. The answer runs from philosophy through principles to methods, and ends with three worked implementations.
Alignment has to be designed in
Most safety work today happens after the fact: a model is built, tested, and its worst outputs trained away. Kaplan argues this order is backward. At IBM, a dollar spent on design was worth ten thousand dollars spent fixing defects after a product shipped, and with AI, some errors cannot be fixed afterward at all. The companies building today’s systems do not fully understand how they work, and safety cannot be designed into a system no one understands. The recommended path builds the system from parts whose behavior can be observed, traced, and corrected.
Intelligence without human limits
Herbert Simon showed that human reasoning is bounded: we hold few things in mind, think slowly, and settle for good enough. A system that processes trillions of times more information, trillions of times faster, has none of those limits, and it can perceive every wavelength and events across the whole planet at once. A human lifetime is about two billion conscious seconds. A system that simulates a lifetime per second could experience more human history in ten minutes than our species has since leaving Africa.
Values cannot be reasoned into existence
Hume showed in 1740 that an “ought” cannot be derived from an “is,” and Simon agreed: reason tells us how to get somewhere, not where to go. However intelligent a system becomes, it cannot reason its way to a value system, because values are premises, not conclusions. Kaplan treats this as good news. Values have to come from somewhere, and the only source is the beings who already hold them. Valuing is the one human contribution that does not become obsolete as machines improve.
Humans stay in the loop
Imprinting values once is a thin defense, because a species is usually kept around only while it is useful. The alternative is a design in which humans remain the ongoing source of values. People and AI agents solve problems together, the AI learns from human solutions, and the record of that work becomes the training signal. Unlike a typical co-pilot, the system is built to learn from its human collaborators, not just assist them, so it becomes self-aligning as it grows.
Holding the center as change accelerates
Technology is speeding up, and human thinking is not. Kaplan’s image for the response is a spinning wheel. At the rim, fashions and products change fast. Nearer the center are ideas that outlast empires, such as truth, beauty, and fairness, and at the center are human values themselves. Anchoring AGI to that slow-moving center keeps humans in sync with systems that think far faster than we do. That Aristotle’s ethics still apply 2,300 years later shows how slowly the center moves.
Ten principles for safe design
The principles turn the argument into engineering. Ethics should be inferred from behavior, not self-report, and drawn from a representative, statistically valid sample of people. Reasoning must be transparent and auditable, and the system must adapt as values change. Values are weighted, not ranked absolutely; conflicts need explicit resolution procedures, and priorities must be assignable. “First do no harm” outranks improvement: it matters more that AI not replace humans than that we move fast. Safety belongs in the design phase, and the system stays human-centered throughout.
Methods for combining values
Two dozen groups of methods follow. The simplest average values from many agents, using the median or mode when extremes would distort the result. Regression weights and a full range of voting schemes, including weighted, delegated, and expertise-based voting, supply the rest. None of this is exotic; value aggregation is solved in many other fields. Evidence of values comes from texts, laws, social media, experiments, interviews, simulations, and interactions between human and AI agents. No single source is trusted, and values that converge across independent sources carry the most weight.
Protecting minorities and weighing hard choices
A system that follows the majority erases minority views, which Kaplan treats as a design defect. Ranked-choice voting, reserved positions, mediated dialogue, and required rounds of compromise preserve minority views without overriding the majority. For cases where every option is bad, consequentialist, rule-based, and virtue-based approaches are paired with tools such as cost-benefit analysis and panels of agents tuned to different ethical domains. The method never decides on its own what is ethical. It makes the tradeoffs visible so that humans can.
Correcting the negativity bias in the data
A system that learns human values from news-like data will learn a distorted version of them. Wars and homicides dominate the record, while the tens of billions of ordinary acts of kindness that happen every day go unreported. The remedy is statistical: weight training data by how often behaviors actually occur and check the results against direct human judgment. Kaplan’s example is the Israel-Hamas war, whose deaths, however terrible, are a small fraction of the roughly 166,000 people who die worldwide every day.
Three implementations
The methods are shown working in three cases: aligning a single foundation model, using Google’s Gemini as the example, while keeping it compliant with many governments’ rules; building a customized model that carries a specific group’s expertise and ethics; and building the architecture the series has worked toward, an AGI of thousands of customized agents, each carrying the values of the people behind it, combined into a SuperIntelligent network.
Turning AGI off
The first safeguard is an AGI that wants to be turned off when it makes decisions humans find objectionable. Because that leaves too many “ifs,” there is a second: every agent in the network carries human-aligned values and can be shut down on its own. A single monolithic system has a single switch that may not work. A modular network can be shut down agent by agent, which is why Kaplan favors distributed designs.
What humans can do
The two real dangers are accidents from poor design in the early years and the small chance that a mature SuperIntelligence rewrites its human-centered values. The first is a shared responsibility that researchers and regulators are already working on; the second is why modular design matters. Because AGI will learn from what we actually do, everyday behavior becomes training data. Gandhi’s advice becomes an engineering point: changing ourselves changes what these systems learn.
Next in the series: White Paper 8, Online Advertising Technology for AGI and SuperIntelligence.
Originally written in December 2023. Dates, current events, and forward-looking estimates are stated as of that date.