SI WHITE PAPER 4: SYSTEMS AND METHODS FOR SAFE, SCALABLE AGI
ABSTRACT: Systems and Methods for Safe, Scalable Artificial General Intelligence
Today's main approaches to AI safety, Reinforcement Learning with Human Feedback (RLHF) and Constitutional AI, can't train AI to be ethical and safe at the scale that's needed. RLHF costs too much to cover the endless range of dangerous scenarios. Constitutional AI scales better, but it depends on rules written by a small group and on AI teaching AI, with few humans involved.
This white paper describes another approach. Millions of people each train their own AI agent on their own values. The ethical knowledge of those agents is then combined, along with as much direct human input as possible. The result is a representative, statistically valid sample of human values across a wide range of scenarios.
The paper presents methods for:
-
combining agents and weighting their input;
-
covering the most common and highest-stakes situations first;
-
flagging new ethical issues as they come up;
-
updating knowledge as values change.
A detailed example using Meta shows how the approach could work in practice.
The goal is AI, AGI, and SuperIntelligence that stay aligned with human values even after they far exceed human intelligence.
Below are common questions and a more in-depth summary of the paper.
COMMON QUESTIONS: Systems and Methods for Safe, Scalable AGI
Why aren't today's AI safety methods enough?
The two main methods both fall short. Reinforcement Learning with Human Feedback (RLHF) relies on people correcting the AI one case at a time, and there are far too many dangerous scenarios to cover. Constitutional AI scales better, but a small group writes the rules, and AI teaches other AI with few humans involved. The result reflects a few people's values, not humanity's.
How does this approach teach AI human values?
Millions of people each train their own AI agent on their own values. Those agents then work alongside humans to train other AI systems, a process the paper calls Reinforcement Learning via Feedback (RLF). Because every agent carries its owner's values, the combined result is far more representative than any rulebook, and it can scale in a way that human trainers alone cannot.
How do you combine the values of millions of people?
The simplest method is one agent, one vote, where every agent counts equally. Other methods can give more weight to human input, to experts or trusted sources, or to more recent judgments. Agents trained by different people can also be merged mathematically into a new agent that reflects all of them. The right method depends on the situation and the resources available.
Whose values does the AI end up with?
It ends up with values drawn from a representative, statistically valid sample of humanity. Dr. Kaplan argues this is the only sound way to learn what human values actually are, since there's no single correct answer to most ethical questions. A broad sample also tends to dampen extreme positions, which pose the greatest risk. The diversity of human values is treated as a strength.
Why not let a smart enough AI decide what's right?
Values can't be rationally derived. Logic can tell you how to reach a goal, but not which goal is worth pursuing. However capable AI becomes, it has no independent basis for deciding what people should value. So the paper holds that values must always come from people, and that handing that job to AI would be a serious mistake.
How does the system handle situations no one planned for?
Humans and AI agents can flag a potential ethical issue as it arises. When a delay is acceptable, the system pauses until more human input arrives. Training also focuses first on the most common and highest-stakes situations. New scenarios that surface during problem-solving go back into training, so coverage improves over time.
What would this look like in a real company?
The paper walks through a scenario in which a company such as Meta uses its large user base to put the approach into practice. Each user gets a personal AI agent tuned to their own data. The process has four phases: train a safe base model, customize it for each user, combine what many agents have learned, and refine values through shared problem-solving.
What role do people play once AI is smarter than us?
People remain the source of values. As AI grows more capable, it will handle more of the reasoning and problem-solving, but it still needs someone to decide what's worth pursuing. Values change more slowly than other knowledge and sit at the center of what guides the system. Keeping human values at the center is how AI stays aligned as it surpasses us.
SUMMARY: Systems and Methods for Safe, Scalable Artificial General Intelligence
In a collective intelligence approach to safe, scalable AGI, intelligence comes not from one monolithic model but from a community of many agents, human and AI. The approach builds on earlier work on customizing and training individual AI agents (AAAIs), applying them to real products and platforms, and integrating them into a Human-Centered AGI network. Its methods can be used with that earlier work or on their own.
The problem with current safety approaches
The two main methods used today to make large language models safe both fall short. Reinforcement Learning with Human Feedback (RLHF) rewards safe responses and corrects dangerous ones, but dangerous scenarios are effectively unlimited, and even thousands of human trainers cannot cover them all. Constitutional AI addresses scale by having a small group write ethical rules, and a trained AI teach them to other AIs. That removes humans from the loop, risks misreading the rules, and reflects the values of a few people rather than humanity. And because AI acts far faster than people, a mistake could do serious harm before anyone catches it.
The approach: combine values from many customized agents
The approach retains the scalability of AI training while replacing a single trainer with many AI agents, each customized by a different person, working alongside humans to provide Reinforcement Learning via Feedback (RLF). Because each agent carries its owner’s values, the pooled result is more representative than any constitution and more scalable than RLHF. Humans stay in the loop as much as resources allow, and new ethical scenarios that surface during problem-solving are folded back into training.
Reinforcement learning and combining weights
Whatever an agent learns through feedback ends up in its network weights. Training through feedback and directly combining the weights of separately trained agents turns out to be functionally equivalent. So agents trained by different people can be merged mathematically into a new agent that reflects all of them. Which method fits depends on the available trainers and computing resources.
Methods for combining agent input
The simplest scheme is one-agent, one-vote, where every agent has equal influence. Variations can weight human input more or less than AI input, favor experts or trusted sources, or weight agents by metadata such as expertise, reliability, and track record. Input can also be weighted by time, so recent judgments count more, or adjusted at points where norms clearly shifted. Throughout, Kaplan holds that values cannot be rationally derived and so should never be delegated to AI, however capable it becomes.
Values and ethics considerations
Ethical questions have no single correct answer; they depend on who is asked. The Trolley Problem shows that human judgments vary with circumstance and culture, yet they can be measured and weighted in ways an AI can learn. So AI should be trained on how people actually behave, not on rigid rules. Shared norms and the social contracts people accept when they join groups are useful starting points, and the diversity of human values is a feature, not a flaw. Kaplan argues, on practical and scientific grounds, for values drawn from a democratic, representative, statistically valid sample of humanity. It is the only sound way to learn what human values are, and it tends to dampen the extreme positions that pose the greatest risk.
Efficient training methods
A good implementation uses a representative sample of human values. It scales to cover the situations the AI will face, and it keeps humans in the loop as far as scale allows.
Supporting techniques include:
-
covering the full range of ethical situations the AI may face;
-
detecting unexpected safety issues in real time;
-
learning through conversations and surveys;
-
passive machine learning;
-
guidance on how often to update knowledge.
A spinning knowledge wheel places slowly changing values at the center and faster-changing knowledge toward the rim.
A detailed implementation example
In a real-world scenario, a company such as Meta puts the approach into practice using its large user base and behavioral data, which are sufficient to represent human values. Users create personalized agents tuned to their own data. The company serves as a platform, repository, and marketplace for the resulting knowledge modules.
The process runs in four phases:
-
Train a base model with safety and ethical guardrails.
-
Customize it for each user.
-
Combine knowledge from many customized agents.
-
Refine values through problem-solving with other agents and humans.
The role of humans
As AI grows more capable, the central human role is to supply the values and goals that smarter systems pursue. Values change more slowly than other knowledge and sit at the center of the knowledge base, which makes them the most important input for guiding SuperIntelligence. Even when AI is vastly smarter than people, humans can remain the source of those values, and centering AI on representative human values is the most promising path to systems that stay aligned.
Next in the series: White Paper 5, Safe Personalized SuperIntelligence
This paper preserves the argument as originally written in July 2023. Dates and forward-looking estimates are stated as of that date.