1. Safety by Design vs. Testing
The first challenge is not unique to SI but is essential for developing any advanced software system. In the 1980s, IBM researchers were interested in the question of how to improve software quality. Improving quality also equates to improving safety for some applications since an error in some mission-critical applications directly endangers human lives. The areas investigated by the researchers covered every aspect of software development, including management innovations, design, coding, development, testing processes, and technology systems. [6]

The most significant result from IBM’s research can be summed up in the adage: “An ounce of prevention is worth a pound of cure.” For example, one IBM study found that each extra dollar spent reviewing and improving software quality at the design stage saved approximately $10,000 in expenses if a design error was missed, resulting in defective software products being shipped to customers. This 10,000:1 benefit emphasized the criticality of preventing mistakes early in the design stage rather than trying to detect and correct them in later testing stages of development or after the software had already shipped and was deployed.

Generally, the key to ensuring high-quality software is defect prevention at the design stage rather than defect detection at the testing stage. Unfortunately, when it comes to developing LLMs and many other advanced AI systems, the current approach relies heavily on detecting and correcting problems via testing rather than designing the systems to avoid problems in the first place.

​

2. Transparency and Understandability

Developers depend on testing because they do not fully understand how LLMs and advanced AI systems work. No one knows how LLMs represent knowledge in detail, so it is impossible to predict precisely how LLMs will respond to various prompts.

This lack of transparency means that current LLMs must be treated as black boxes.

Transparency challenges persist despite significant research efforts. [7] Many approaches amount to probing the LLM and determining from the probes how it might represent information and how it might act in various situations or prompts.

​

As LLMs scale from hundreds of billions of parameters to trillions of parameters, probing cannot keep pace. The result is that developers and users will lack confidence that enough probing has been done so that an advanced AI or SI will behave reliably in ways aligned with their interests.

​

3. Control

As AI agents are given increasing levels of autonomy, the problem of controlling them gets more complicated. [8] Companies and developers are caught in a dilemma. Limit the autonomy of one’s AI agents, and users will flock to a competitor's more powerful and autonomous agents. Enable greater autonomy and intelligence, and the risk of untrustworthy or dangerous behavior increases.

​

Developers typically give agents significant autonomy and then test to see if they behave well across various scenarios. There are automated approaches to the testing of AI and LLMs. [9] However, the fundamental black-box nature of LLMs typically leads to a reliance on testing rather than the more powerful approach of building control and safety mechanisms into the system’s design.

​

4. Alignment

The alignment problem can be stated as the problem of making “AI systems behave in line with human intentions and values.” [10] Multiple challenges exist in aligning SI with human values.

​

First is the issue of whose values the SI should be aligned with. Researchers have discussed this issue at length, bringing up related issues, including whether AI should be aligned with instructions, intentions, revealed preferences, ideal preferences, interests, or values. [11]

​

Second, even if we knew which (or whose) values to use, the transparency problem means relying on testing in alignment, which we have seen is impractical at scale.

​

Third, even if we could test in alignment today, if SI develops cognitive capabilities that far exceed those of humans, what reason do we have to think that such systems will remain aligned with human interests? SI will likely be far more intelligent than us. Hinton [12] has likened the difference in intelligence between SI and humans to the difference in intelligence between humans and frogs. He notes, “It didn’t work out too well for the frogs.”

​

Despite these challenges, the writings of Herbert Simon offer a ray of hope. In a book entitled Reason in Human Affairs, Simon [13] writes:

​

“Reason is wholly instrumental. It cannot tell us where to go. At best, it can tell us how to get there.”

If values cannot be rationally derived, then any intelligent system must get them from a source independent of logic and reason. Given a set of core values, SI will likely think faster and therefore be better than humans at determining how to achieve goals reflecting those values. But the values themselves must come from a non-logical source. There is no guarantee that SI will remain aligned with human values when its intelligence surpasses that of humans. Simon’s observation suggests that the ability to perform logical operations trillions faster than humans will not be the deciding factor. Further, designing SI to include humans as a necessary part of its operation, at least initially, can maximize the likelihood that SI chooses human values even if it reasons much faster and more comprehensively than humans.

​

5. Scalability

A significant challenge concerning safety and alignment is scalability. Any alignment or safety approach requiring humans in the loop must deal with limited human cognitive and computational capacity. In contrast, the cognitive abilities of SI are increasing over time. Even if human cognition can keep pace with SI cognition in the near term, humans will not be able to keep pace over the longer term. Therefore, any design that hopes to ensure that SI remains safe and aligned with humans must incorporate safety mechanisms that scale with the increasing intelligence of the SI system.

​

Existing approaches to the scalability problem, including constitutional AI, amount to variations on the theme that AI can be responsible for the safety and alignment of AI. But this seems like asking the fox to keep watch over the henhouse.

​

It will likely be necessary to integrate alignment and safety mechanisms into the system's architecture, which scales as the cognitive ability of SI scales.

​

6. Exponential Change

Underlying concerns about scalability and alignment are the general observation that computational power has increased exponentially or rapidly. While many researchers are familiar with Moore’s law, the observation that computing power per constant dollar roughly doubles every two years, it has now been replaced by Huang’s law, which applies to GPU-based computing power. Huang’s law suggests that computational power might take only about one year to double. [14] We can expect LLMs to be at least 1,000X more powerful in ten years, based on increased computational power alone. LLMs are typically mediocre, make errors (hallucinate), and are not nearly as good as top human experts in many fields. But with a 1,000X improvement, that will no longer be the case. In the future, humans will not be able to keep up with SI, just as the best human chess players no longer have any hope of keeping up with AI chess programs. However, just as a human chess player might task an AI chess program to explore specific openings or strategies, humans may be able to direct the cognition of SI if the SI accepts human direction.