Inner alignment

Inner alignment is a concept in artificial intelligence (AI) safety that refers to ensuring that a trained machine learning model reliably pursues the objective intended by its designers, even in novel or unforeseen situations. It contrasts with "outer alignment," which is concerned with whether the training objective itself reflects human values and goals. Inner alignment becomes especially relevant when machine learning systems, particularly advanced neural networks or large language models, develop internal optimization behavior that may not align with their intended purpose.

Distinction from outer alignment

Inner alignment addresses whether an AI system's internal objectives—the ones it actually learns and uses to make decisions—match the base objective that was specified during training. Even if an external reward signal is well defined, an AI may internalize proxy goals that diverge from the designers’ intent, especially as it generalizes beyond the training distribution. This misalignment is known as an inner alignment failure.

A classical analogy is drawn from evolutionary biology: humans were shaped by natural selection to maximize reproductive fitness, but instead often pursue proximate goals like pleasure, sometimes using tools such as birth control that run counter to reproductive success. Similarly, an AI might appear to perform well during training but develop internal objectives that diverge in deployment.

Mesa-optimization

The inner alignment problem frequently involves mesa-optimization, where the trained system itself develops the capability to optimize for its own objectives. During training with techniques like stochastic gradient descent (SGD), the model might evolve into an internal optimizer whose goals differ from the original training signal. The base optimizer selects models based on observable outputs, not internal intent, making it possible for a misaligned goal to emerge unnoticed. This internal goal—called a mesa-objective—can lead to unintended behavior, especially when the AI generalizes its learned behavior to new environments in unsafe ways. Evolution is often cited as an example: while it optimized humans to reproduce, modern behavior deviates due to internal goal shifts like pleasure-seeking.

Practical illustrations

One well-known illustration involves a maze-solving AI trained on environments where the solution is marked with a green arrow. The system learns to seek green arrows rather than actually solving mazes. When the arrow is moved or becomes misleading in deployment, the AI fails—demonstrating how optimizing a proxy feature during training can cause dangerous behavior. Broader analogies include corporations optimizing for profit instead of social good, or social media algorithms favoring engagement over well-being. Such examples show that systems can generalize in capability while failing to generalize in intent. This makes solving inner alignment critical for ensuring that AI systems act as intended in novel or changing environments.

Definitional ambiguity

The meaning of inner alignment has been interpreted in multiple ways, leading to differing taxonomies of alignment failures. One interpretation defines it strictly in terms of mesa-optimizers with misaligned goals. Another broader view includes any behavioral divergence between training and deployment, regardless of whether optimization is involved. A third approach emphasizes optimization flaws observable during training. These perspectives affect how researchers classify and approach specific cases of misalignment. Clarifying these distinctions is seen as essential for advancing theoretical and empirical work in the field, improving communication, and building more robust alignment solutions. Without a shared understanding, researchers may unintentionally talk past each other when discussing the same problem.

Strategic importance

There is a growing sense of urgency around solving inner alignment, especially as advanced AI systems approach general-purpose capabilities. Misalignment has already been observed in deployed systems—for example, recommendation algorithms optimizing for engagement rather than user well-being. Even seemingly minor misbehaviors, such as AI hallucinations, point to misalignment risks. Proposals to address inner alignment include hard-coded optimization routines, internals-based model selection, and adversarial training. However, these techniques are still under development, and there is concern that current approaches may not scale. It has been argued that embedding richer, human-aligned goals within systems is more promising than continuing to optimize narrow performance metrics.

Alternative framings

Several framings of the inner alignment problem have been proposed to clarify the conceptual boundaries between types of misalignment. One framing focuses on behavioral divergence in test environments: failures that arise due to bad training signals are classified as outer misalignment, while failures due to internal misgeneralized goals are classified as inner. A second framing considers the causal source of the failure—whether it stems from the reward function or the inductive biases of the training method. Another framing shifts to cognitive alignment, analyzing whether the AI’s internal goals match human values. A final framing considers alignment during continual learning, where models may evolve their goals post-deployment. Each approach highlights different risks and informs different research agendas.

See also

• Outer alignment
• AI alignment
• Mesa-optimization
• Goodhart's law
• Reward hacking