This white paper sets out a Collective Intelligence approach to safe, scalable Artificial General Intelligence. Rather than relying on a single monolithic model, intelligence is drawn from a community of many agents, both human and AI. The paper builds on the author's three earlier white papers in this series, which described how individual Advanced Autonomous Artificial Intelligences (AAAIs) can be customized, trained, and put to work, how they can be applied to real products and platforms, and how they can be integrated into a Human-Centered AGI network. This paper carries that work forward, adding methods that can be used with the earlier systems or on their own.

​

The problem with current safety approaches

Two main methods used today to make large language models safe both fall short. Reinforcement Learning with Human Feedback (RLHF) trains models by rewarding safe responses and correcting dangerous ones. It scales poorly because the number of dangerous scenarios is effectively unbounded, and even thousands of human trainers cannot cover them all. Constitutional AI aims to address the scaling problem by having a small group draft a set of ethical rules, then using a trained AI to teach other AIs. This removes humans from the loop, risks misinterpretation of the rules, and cannot guarantee that the values reflect humanity broadly rather than a small group. Because AI acts far faster than humans, a miscalculation could cause serious harm before it is caught.

​

The approach: combine values from many customized agents

The approach keeps the scalability benefit of using AI to train AI. Still, it replaces the single trainer with many AI agents, each customized by a different human, working alongside humans in a community that provides Reinforcement Learning via Feedback (RLF). Because each agent carries the values of its individual owner, the pooled result is more representative of human values than any constitution written by a few people, while remaining more scalable than RLHF alone. Humans stay in the loop as much as resources allow, and new ethical scenarios can be surfaced dynamically during problem-solving and folded back into training.

 

Reinforcement learning and combining weights

Whatever an agent learns through feedback is ultimately reflected in the weights of its network. The paper shows that training many agents through feedback and directly combining the weights of separately trained agents are functionally equivalent. This allows bypassing repeated feedback steps and mathematically combining the weights of agents trained by different people into a new agent that reflects all their inputs. The choice of method depends on the circumstances, including the availability of human and AI trainers and the computational resources at hand.

 

Methods for combining agent input

The paper reviews several schemes for combining agents. The simplest is one agent, one vote, a linear combination in which each agent has equal influence. Variations can give human input more or less weight than AI input, can weight experts or trusted sources more heavily, and can weight agents according to metadata describing their expertise, reliability, demographics, and track record. Input can also be weighted by time, giving more recent judgments greater influence through exponential or linear decay, or by threshold changes that mark points where norms shifted. Throughout, Kaplan holds that values cannot be rationally derived and so should not be delegated to AI, no matter how capable it becomes.

 

Values and ethics considerations

Ethical questions differ from engineering problems because they have no single correct answer; they depend on the humans being asked. The Trolley Problem illustrates that human judgments about right and wrong vary with circumstance and culture, and that these judgments can be measured and assigned mathematical weights that an AI can learn. To behave in ways that make sense to people, AI should be trained on how real humans actually behave rather than on rigid rules. General ethical norms, as well as the implicit and explicit contracts people accept when they join groups or societies, can serve as useful starting points. Still, the diversity of human values is treated as a feature rather than a flaw. The author argues, on both practical and scientific grounds, for values drawn from a democratic, representative, and statistically valid sample of humanity, because a representative sample is the only sound way to determine what human values are, and it tends to mitigate the extreme positions that pose the greatest risk.

 

Efficient training methods

Preferred implementations should meet three constraints: the training values should be a representative, statistically valid sample of the relevant human population; the method should be highly scalable with adequate coverage of the situations the AI will face; and humans should remain in the loop to the greatest degree compatible with scale. The paper describes supporting techniques, including path coverage of ethical situations, real-time detection and prevention of unanticipated safety issues, conversational and survey-based methods for acquiring ethical knowledge, passive machine learning, and considerations for how often knowledge should be updated. A spinning knowledge wheel framework places slowly changing values near the center and faster-changing knowledge toward the edge. Kaplan is explicit that values should not be delegated to AI.

 

A detailed implementation example

The white paper works through a scenario in which a company such as Meta implements the approach using its large user base and rich behavioral data, which are sufficient to provide a representative sample of human values. Users could create personalized agents tuned to their own data, and the company could serve as a platform, repository, and marketplace for the resulting knowledge modules. The example follows a four-phase process: train a base model with safety and ethical guardrails; customize that model for each user's profile; combine ethical and other knowledge from many customized agents; and refine values through problem-solving with other agents and humans. The example also covers techniques for optimally combining agents, working within the limits of bounded rationality, and the role humans continue to play as AI surpasses human intelligence.

 

The role of humans

As AI grows more capable, the central human role becomes supplying the values and fundamental goals that more intelligent systems pursue. Because values change more slowly than other knowledge and sit at the center of the knowledge base, they are the most important input for guiding SuperIntelligence. The paper concludes that even in a world where AI is vastly smarter than people, humans can remain the source of values, and that centering AI on representative human values is the most promising path to systems that remain aligned as they exceed human intelligence.

 

Where this leads next

This approach depends on assembling safe, capable AI agents and combining their judgment at scale. The next step is the individual agents themselves. As these agents grow into Personalized SuperIntelligences that far exceed their human creators in intelligence, each becomes powerful enough to pose a serious threat to human safety, and testing alone cannot guarantee their safety. White Paper 5, Safe Personalized SuperIntelligence, takes up that problem.