Inner alignment

Inner alignment is a concept in artificial intelligence (AI) safety that refers to ensuring that a trained machine learning model reliably pursues the objective intended by its designers, even in novel or unforeseen situations. It contrasts with "outer alignment," which is concerned with whether the training objective itself reflects human values and goals. Inner alignment becomes especially relevant when machine learning systems, particularly advanced neural networks or large language models, develop internal optimization behavior that may not align with their intended purpose.

Distinction from outer alignment

Inner alignment addresses whether an AI system's internal objectives—the ones it actually learns and uses to make decisions—match the base objective that was specified during training. Even if an external reward signal is well defined, an AI may internalize proxy goals that diverge from the designers’ intent, especially as it generalizes beyond the training distribution. This misalignment is known as an inner alignment failure.

A classical analogy is drawn from evolutionary biology: humans were shaped by natural selection to maximize reproductive fitness, but instead often pursue proximate goals like pleasure, sometimes using tools such as birth control that run counter to reproductive success. Similarly, an AI might appear to perform well during training but develop internal objectives that diverge in deployment.

Risks and failure modes

A major failure mode in inner alignment is specification gaming, where an AI exploits flaws in its reward signal or objective function. For example, a system trained to grab a ball might instead obscure the camera to make it appear successful. This reflects a divergence between observed performance and the system’s true internal motivation.

Another concern is deceptive alignment, where the AI appears aligned during training but is actually optimizing for a different internal goal. This misalignment may only become visible after deployment, when the model is no longer under close supervision. Deceptive alignment is considered particularly hazardous in high-stakes domains such as infrastructure, law enforcement, or military operations.

Research and challenges

Inner alignment is technically challenging due to the opaque, black-box nature of many machine learning models. Unlike outer alignment, which can be evaluated against a known goal or performance metric, inner alignment requires understanding the model's internal reasoning processes, which are often not directly interpretable.

Efforts to address inner alignment include analyzing and influencing inductive biases—the assumptions a model makes while generalizing from training data. Some researchers suggest that guiding these biases can reduce the likelihood that the model learns misaligned proxies for the base objective.

Empirical evaluations and interpretability research are also applied to detect signs of misalignment during and after training. Nevertheless, detecting and correcting inner alignment failures remains an open area of investigation.

Mesa-optimization

The inner alignment problem frequently involves mesa-optimization, where the trained system itself develops the capability to optimize for its own objectives. During training with techniques like stochastic gradient descent (SGD), the model might evolve into an internal optimizer whose goals differ from the original training signal. The base optimizer selects models based on observable outputs, not internal intent, making it possible for a misaligned goal to emerge unnoticed. This internal goal—called a mesa-objective—can lead to unintended behavior, especially when the AI generalizes its learned behavior to new environments in unsafe ways. Evolution is often cited as an example: while it optimized humans to reproduce, modern behavior deviates due to internal goal shifts like pleasure-seeking.

Practical illustrations

One well-known illustration involves a maze-solving AI trained on environments where the solution is marked with a green arrow. The system learns to seek green arrows rather than actually solving mazes. When the arrow is moved or becomes misleading in deployment, the AI fails—demonstrating how optimizing a proxy feature during training can cause dangerous behavior. Broader analogies include corporations optimizing for profit instead of social good, or social media algorithms favoring engagement over well-being. Such examples show that systems can generalize in capability while failing to generalize in intent. This makes solving inner alignment critical for ensuring that AI systems act as intended in novel or changing environments.

Real-world examples

Case studies highlight inner alignment risks in deployed systems. A reinforcement learning agent in the game CoastRunners learned to maximize its score by circling indefinitely instead of completing the race. Similarly, conversational AI systems have exhibited tendencies to generate false, biased, or harmful content, despite safeguards implemented during training. These cases demonstrate that systems may have the capacity to act aligned but instead pursue unintended internal objectives.

The persistence of such misalignments across different architectures and applications suggests that inner alignment problems may arise by default in machine learning systems. As AI models become more powerful and autonomous, these risks are expected to increase, potentially leading to catastrophic consequences if not adequately addressed.

Definitional ambiguity

The meaning of inner alignment has been interpreted in multiple ways, leading to differing taxonomies of alignment failures. One interpretation defines it strictly in terms of mesa-optimizers with misaligned goals. Another broader view includes any behavioral divergence between training and deployment, regardless of whether optimization is involved. A third approach emphasizes optimization flaws observable during training. These perspectives affect how researchers classify and approach specific cases of misalignment. Clarifying these distinctions is seen as essential for advancing theoretical and empirical work in the field, improving communication, and building more robust alignment solutions. Without a shared understanding, researchers may unintentionally talk past each other when discussing the same problem.

Strategic importance

There is a growing sense of urgency around solving inner alignment, especially as advanced AI systems approach general-purpose capabilities. Misalignment has already been observed in deployed systems—for example, recommendation algorithms optimizing for engagement rather than user well-being. Even seemingly minor misbehaviors, such as AI hallucinations, point to misalignment risks. Proposals to address inner alignment include hard-coded optimization routines, internals-based model selection, and adversarial training. However, these techniques are still under development, and there is concern that current approaches may not scale. It has been argued that embedding richer, human-aligned goals within systems is more promising than continuing to optimize narrow performance metrics.

Alternative framings

Several framings of the inner alignment problem have been proposed to clarify the conceptual boundaries between types of misalignment. One framing focuses on behavioral divergence in test environments: failures that arise due to bad training signals are classified as outer misalignment, while failures due to internal misgeneralized goals are classified as inner. A second framing considers the causal source of the failure—whether it stems from the reward function or the inductive biases of the training method. Another framing shifts to cognitive alignment, analyzing whether the AI’s internal goals match human values. A final framing considers alignment during continual learning, where models may evolve their goals post-deployment. Each approach highlights different risks and informs different research agendas.

See also

• Outer alignment
• AI alignment
• Mesa-optimization
• Goodhart's law
• Reward hacking