Mechanistic interpretability

Mechanistic interpretability (often shortened to "Mech Interp" or "MI") is a subfield of interpretability that seeks to reverse‑engineer neural networks, generally perceived as a black box, into human‑understandable components or "circuits", revealing the causal pathways by which models process information. The object of study generally includes but is not limited to vision models and Transformer-based large language models (LLMs).

History

Chris Olah is generally credited with coining the term 'Mechanistic interpretability' and spearheading its early development. In the 2018 paper The Building Blocks of Interpretability, Olah (then at Google Brain) and his colleagues combined existing interpretability techniques, including feature visualization, dimensionality reduction, and attribution with human-computer interface methods to explore features represented by the neurons in the vision model, Inception v1. In the March 2020 paper Zoom In: An Introduction to Circuits, Olah and the OpenAI Clarity team described "an approach inspired by neuroscience or cellular biology", hypothesizing that features, like individual cells, are the basis of computation for neural networks and connect to form circuits, which can be understood as "sub-graphs in a network". In this paper, the authors described their line of work as understanding the "mechanistic implementations of neurons in terms of their weights".

In 2021, Chris Olah co-founded the company Anthropic and established its Interpretability team, which publishes their results on the Transformer Circuits Thread. In December 2021, the team published A Mathematical Framework for Transformer Circuits, reverse-engineering a toy transformer with one and two attention layers. Notably, they discovered the complete algorithm of induction circuits, responsible for in-context learning of repeated token sequences. The team further elaborated this result in the March 2022 paper In-context Learning and Induction Heads.

Notable results in mechanistic interpretability from 2022 include the theory of superposition wherein a model represents more features than there are directions in its representation space; a mechanistic explanation for grokking, the phenomenon where test-set loss begins to decay only after a delay relative to training-set loss; and the introduction of sparse autoencoders, a sparse dictionary learning method to extract interpretable features from LLMs.

Mechanistic interpretability has garnered significant interest, talent, and funding in the AI safety community. In 2021, Open Philanthropy called for proposals that advanced "mechanistic understanding of neural networks" alongside other projects aimed to reduce risks from advanced AI systems. The interpretability topic prompt in the request for proposal was written by Chris Olah. The ML Alignment & Theory Scholars (MATS) program, a research seminar focused on AI alignment, has historically supported numerous projects in mechanistic interpretability. In its summer 2023 cohort, for example, 20% of the research projects were on mechanistic interpretability.

Many organizations and research groups work on mechanistic interpretability, often with the stated goal of improving AI safety. Max Tegmark runs the Tegmark AI Safety Group at MIT, which focuses on mechanistic interpretability. In February 2023, Neel Nanda started the mechanistic interpretability team at Google DeepMind. Apollo Research, an AI evals organization with a focus on interpretability research, was founded in May 2023. EleutherAI has published multiple papers on interpretability. Goodfire, an AI interpretability startup, was founded in 2024.

Mechanistic interpretability has greatly expanded its scope, practitioners, and attention in the ML community in recent years. In July 2024, the first ICML Mechanistic Interpretability Workshop was held, aiming to bring together "separate threads of work in industry and academia". In November 2024, Chris Olah discussed mechanistic interpretability on the Lex Fridman podcast as part of the Anthropic team.

Distinction between interpretability and mechanistic interpretability

The term mechanistic interpretability designates both a class of technical methods—explainability methods such as saliency maps are generally not considered mechanistic interpretability research—and a community. Mechanistic interpretability's early development was rooted in the AI safety community, though the term is increasingly adopted by academia. In “Mechanistic?”, Saphra and Wiegreffe identify four senses of “mechanistic interpretability”:

1. Narrow technical definition: A technical approach to understanding neural networks through their causal mechanisms.

2. Broad technical definition: Any research that describes the internals of a model, including its activations or weights.

3. Narrow cultural definition: Any research originating from the MI community.

4. Broad cultural definition: Any research in the field of AI—especially LM—interpretability.

As the scope and popular recognition of mechanistic interpretability increase, many have begun to recognize that other communities such as natural language processing researchers have pursued similar objectives in their work.

Key Concepts

Linear Representation

The latent space of a LLM is generally conceptualized as a vector space. There is a growing body of work around the linear representation hypothesis, which posits that high-level features correspond to approximately linear directions in a model’s activation space.

Superposition

Superposition is the phenomenon where many unrelated features are “packed’’ into the same subspace or even into single neurons, making a network highly over-complete yet still linearly decodable after nonlinear filtering. Recent formal analysis links the amount of polysemanticity to feature ‘‘capacity’’ and input sparsity, predicting when neurons become monosemantic or remain polysemantic.

Methods

Probing

Probing involves training a linear classifier on model activations to test whether a feature is linearly decodable at a given layer or subset of neurons. Generally, a linear probe is trained on a labelled dataset encoding the desired feature. While linear probes are popular amongst Mechanistic Interpretability researchers, its introduction dates back to 2016 and was widely used in the NLP community.

In 2023, Nanda et al. showed that world-model features such as in-context truth values emerge as linearly decodable directions early in training, strengthening the case for linear probes as faithful monitors of internal state.

Difference-in-Means

Difference-in-Means, or Diff-in-Means constructs a steering vector by subtracting the mean activation for one class of examples from the mean for another. Unlike learned probes, Diff-in-Means has no trainable parameters and often generalises better out-of-distribution. Diff-in-Means has been used to isolate model representation for refusal/compliance, true/false, and sentiment.

Steering

Steering adds or subtracts a direction (often obtained via probing, Diff-in-Means, or K-means) from the residual stream to causally change model behaviour.

Activation/Attribution Patching

Activation patching ablates model components or neurons to trace which part of the model is causally necessary for a behaviour. Automated Circuit Discovery (ACDC) prunes the computational graph by iteratively patch-testing edges, localising minimal sub-circuits without manual hand-holding.

Sparse Autoencoders (SAEs) and Dictionary Learning

SAEs learn a sparse, over-complete basis that reconstructs residual-stream activations; coefficients often activate monosemantically for human-interpretable concepts (e.g. “\newline”, “Latin-alphabet”). Scaling experiments recover thousands of such features in GPT-class models, hinting at a parts-list for larger systems.

Circuit Tracing and Automated Graph Discovery

Circuit tracing substitutes parts of the model, in particular the MLP block, with more interpretable components called "Transcoders". The goal is to recover explicit computational graphs. Like SAEs, circuit tracing uses sparse dictionary learning techniques to. Instead of reconstruction model activations like SAEs, however, Transcoders aims to predict the output of non-linear components given its input. The technique was introduced in the paper "Circuit Tracing: Revealing Computational Graphs in Language Models", published in April 2025 by Anthropic. Circuit tracing has been used to understand how a model plans the rhyme in a poem, perform medical diagnosis, and understand Chain of thought unfaithfulness.

Critique

Critiques of Mechanistic Interpretability can be roughly divided into two categories: critiques of the real-world impact of interpretability, and critiques of the mechanistic approach compared to other sources.

Critics have argued that current circuits-level analyses generalise poorly and risk giving false confidence about alignment, citing cherry-picking, small-scale demos, and absence of worst-case guarantees. In response to Dario Amodei's blogpost "The Urgency of Interpretability", which claimed that researchers are on track to achieve MI's goal of developing "the analogue of a highly precise and accurate MRI that would fully reveal the inner workings of an AI model", Neel Nanda, a researcher lead of the Google Deepmind Mechanistic Interpretability team, argued that Mechanistic Interpretability will never act as highly reliable monitors for safety-relevant features, such as deception. In particular, Nanda highlighted that only a small portion of the model seem to be interpretable and that current methods cannot guarantee the absence of particular features or circuits of interest.

Some have framed Mechanistic Interpretability as "bottom-up interpretability", where the emphasis is neuron-level circuits in contrast to "top-down interpretability", which focuses on emergent high-level concepts. They argue that LLMs are complex systems and high-level concepts cannot be simulated, predicted, and understood with low-level components. As one critic puts, "If you wanted to explain why England won World War II using particle physics, you would just be on the wrong track."