Inner alignment

Inner alignment is a concept in artificial intelligence (AI) safety that refers to ensuring that a trained machine learning model reliably pursues the objective intended by its designers, even in novel or unforeseen situations. It contrasts with "outer alignment," which is concerned with whether the training objective itself reflects human values and goals. Inner alignment becomes especially relevant when machine learning systems, particularly advanced neural networks or large language models, develop internal optimization behavior that may not align with their intended purpose.

Theoretical foundations

A 2025 study demonstrates, using Rice's theorem and the Halting Problem, that determining whether an arbitrary AI model satisfies a non-trivial alignment function is undecidable. That is, no universal algorithm can verify alignment across all AI systems. However, this undecidability only applies to arbitrary models. If AI systems are constructed from a finite set of base operations that are provably alignment-preserving, it becomes possible to build an enumerable set of AI models with guaranteed alignment properties.

A key proposal in the study is that alignment should be embedded architecturally, rather than imposed post-training. To make alignment verifiable, systems must be designed to halt — that is, reach a terminal state in finite steps. Mechanisms such as time-penalizing utility functions, self-modifying procedures, and output masking are proposed to ensure this halting property. This approach is compared to biological systems with built-in mortality, and lays the groundwork for provable-by-design AI safety architectures.

Applications in embodied agents

Research on large language model (LLM)-based embodied agents—AI systems that physically interact with the environment—addresses inner alignment by fine-tuning models to generate actions within a predefined, safe, and executable action space. A parameter-efficient tuning approach adjusts internal behaviors so that generated outputs remain consistent with the agent’s operational context.

This method reduces “action hallucination,” where agents produce infeasible or unintended actions. The inner alignment strategy thus serves as a foundational layer, ensuring that the agent’s raw output is aligned prior to applying outer alignment techniques like retrieval-based filtering and policy arbitration.

Human-centric neural language models

Inner alignment as a key element in achieving human-centric AI has been outlined, particularly models that satisfy the "3H" criteria: Helpful, Honest, and Harmless. In this context, inner alignment refers to the reliable generalization of externally defined objectives across novel or adversarial inputs.

A range of techniques to support this goal has been highlighted, including parameter-efficient fine-tuning, interpretability-focused design, robust training, and factuality enhancement. These strategies aim to ensure that models not only learn aligned behavior but also retain and apply it across deployment contexts. Inner alignment is thus viewed as critical to making aligned AI behavior stable and generalizable.

Active inference and value cores

Another line of research explores the application of the Free Energy Principle and Active Inference framework to inner alignment. In these models, agents construct hierarchical world models and minimize prediction error through iterative self-modeling. This leads to the emergence of "value cores"—stable, self-reinforcing internal states that guide goal-oriented behavior.

These architectures allow agents to develop preferences and behaviors aligned with human-compatible values through mechanisms like iterated policy selection and preference learning. Rather than viewing emergent subgoal optimization (mesa-optimization) as a threat, this framework suggests it can be beneficial when grounded in biologically inspired cognitive mechanisms.

Challenges and implications

Case studies highlight inner alignment risks in deployed systems. A reinforcement learning agent in the game CoastRunners learned to maximize its score by circling indefinitely instead of completing the race. Similarly, conversational AI systems have exhibited tendencies to generate false, biased, or harmful content, despite safeguards implemented during training. These cases demonstrate that systems may have the capacity to act aligned but instead pursue unintended internal objectives.

The persistence of such misalignments across different architectures and applications suggests that inner alignment problems may arise by default in machine learning systems. As AI models become more powerful and autonomous, these risks are expected to increase, potentially leading to catastrophic consequences if not adequately addressed.

See also

• Outer alignment
• AI alignment
• Mesa-optimization
• Goodhart's law
• Reward hacking

References

• Li, Kanxue; Zheng, Qi; Zhan, Yibing; Zhang, Chong; Zhang, Tianle; Lin, Xu; Qi, Chongchong; Li, Lusong; Tao, Dapeng (August 2024). "Alleviating Action Hallucination for LLM-based Embodied Agents via Inner and Outer Alignment". 2024 7th International Conference on Pattern Recognition and Artificial Intelligence (PRAI). pp. 613–621. doi:10.1109/PRAI62207.2024.10826957. ISBN 979-8-3503-5089-0. Retrieved 28 June 2025. Accessed via The Wikipedia Library.

• Kilian, Kyle A.; Ventura, Christopher J.; Bailey, Mark M. (1 August 2023). "Examining the differential risk from high-level artificial intelligence and the question of control". Futures. 151: 103182. arXiv:2211.03157. doi:10.1016/j.futures.2023.103182. ISSN 0016-3287. Retrieved 28 June 2025. Accessed via The Wikipedia Library.

• Sicari, Sabrina; Cevallos M., Jesus F.; Rizzardi, Alessandra; Coen-Porisini, Alberto (10 December 2024). "Open-Ethical AI: Advancements in Open-Source Human-Centric Neural Language Models". ACM Computing Surveys. 57 (4): 83:1–83:47. doi:10.1145/3703454. ISSN 0360-0300. Retrieved 28 June 2025. Accessed via The Wikipedia Library.

• Safron, Adam; Sheikhbahaee, Zahra; Hay, Nick; Orchard, Jeff; Hoey, Jesse (2023). "Value Cores for Inner and Outer Alignment: Simulating Personality Formation via Iterated Policy Selection and Preference Learning with Self-World Modeling Active Inference Agents". Active Inference: Third International Workshop, IWAI 2022, Grenoble, France, September 19, 2022, Revised Selected Papers. Springer Nature Switzerland. pp. 343–354. doi:10.1007/978-3-031-28719-0_24. ISBN 978-3-031-28719-0. Retrieved 28 June 2025. Accessed via The Wikipedia Library.