Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?

We would like to thank the reviewer for their comments and for appreciating our distinction between "what-then-where" and "where-then-what" approaches to mechanistic interpretability. Below, we address potential misunderstandings and clarify our contributions.

What is in the case of the MNIST NN the 'valid circuit'?

In the MNIST experiment, like all our experiments, there is no concept of one single "valid circuit." The field of mechanistic interpretability proposes various criteria to define what constitutes an accepted explanation. Our experiments do not require the notion of a "true" or "valid" circuit, but only test whether the criteria induce a unique accepted explanation. Specifically, we show that the existing criteria suffer from identifiability issues, they lead to multiple incompatible circuits that are all equally plausible according to the criteria. It is important to note that we are not testing specific interpretability methods; rather, we are testing the criteria underlying those methods. Furthermore, we are not proposing new metrics in this work. Instead, we provide counterexamples illustrating that existing criteria fail to yield a unique solution (a unique explanation). This raises foundational questions about the assumptions and objectives of mechanistic interpretability that we discuss at length in Section 5.

It seems challenging to generate insights that are relevant to interpretability as a total.

Our focus is on testing the interpretability criteria themselves. These criteria are meant to be applicable across contexts, including small neural networks. By focusing on small neural networks, we can enumerate all possible explanations exhaustively and demonstrate counterexamples that challenge current criteria. Such stress testing is intractable at scale, but our experiment with a larger NN on MNIST shows a lower bound on the number of circuits proving the problem remains at this scale. Our work is stress-testing the foundations of our field with respect to one property: identifiability. Our work seeks to contribute constructively to the field by questioning the underlying definitions of what is a valid explanation. A clearer understanding of what constitutes an explanation will ultimately guide the development of more reliable methods for generating them.

"Too little emphasis on existing literature that clearly touches on the underlying problem: disentanglement.

Our work does not target specific methods for finding explanations or critique particular instances of circuits previously identified using these methods (like IOI). Instead, we address a more foundational question: whether the criteria used to evaluate explanations reliably induce a unique solution (i.e., a unique explanation). Through extensive empirical evidence, we demonstrate that current criteria fail in this regard. While disentanglement-related literature provides valuable insights, our primary aim is to stress-test the criteria themselves, not to evaluate or propose methods based on disentanglement.

Finally, the source code used for our experiments will be released upon publication.

We hope this response clarifies our contributions and addresses any misunderstandings. Our work is intended as a constructive step toward more rigorous definitions and criteria for explanations, which we believe will ultimately strengthen the field of mechanistic interpretability. Please let us know if we can further clarify our contributions.

We greatly thank the reviewer for their extensive and insightful feedback, as well as for appreciating the usefulness and relevance of our work and the exhaustiveness of our experiments. This review has been highly instrumental in the revisions we made to the paper.

We first give a high-level response to the most important points raised:

On whether the conclusions about identifiability problems can be stated generally or only in particular for circuits as isolated objects: It may indeed be the case that identifiability issues of the "where-then-what" scenario stem from the isolation of circuits from the rest of the network. This would point to a problem with the definitions, in this work, we use the precise goal of circuit discovery that has been formulated in previous work. Also, we emphasize that in the what-then-where, the IIA requirement is a strong case of activation patching. It relies on exhaustively applying counterfactual interventions to parts of the network in context. Specifically, the computation flow goes through the entirety of the network's components when evaluating mappings. As a result, the incompatible explanations found in this strategy indicate that identifiability problems cannot simply be reduced to mischaracterization of the functioning of the network as a whole.

On the extension of the possible improvements mentioned in the abstract/introduction: We have largely rewritten Section 5 (discussion). Specifically, we now give in Section 5.2 several avenues of research through which identifiability issues may be resolved. Amongst these, we now discuss the inner interpretability framework mentioned in Vilas et al. (2024) as a promising path toward strengthening MI as an experimental science.

We now proceed to respond to individual feedback points:

On why a model's behavior should have a single, well-defined explanation, and whether MI assumes unicity: We have introduced a new Section (2.4), in which we now argue that the unicity of explanation of a phenomenon is a strong intuition deeply rooted in human reasoning. After slightly rephrasing the abstract and introduction, we have removed phrasing implying that unicity is an explicit requirement. We now frame the unicity of explanation as a property that we might intuitively expect but that is clearly violated in our experiments. We also argue by extracting quotes (documented in Appendix C) that previous MI research implicitly assumes a unique explanation.

Trivially, the circuit plus additional neurons to form the full network is another such circuit.

We exclude such a scenario: every time we find two circuits that perfectly compute the behavior of the full network but one is included in the other, we only keep the smallest one. Similarly, for the what-then-where approach, we also only keep the smallest mapping that we find. That's why we always report the number of "minimal mappings" and not the number of mappings.

On whether non-identifiability makes MI hopeless: We have clarified our positioning towards the need for identifiability. In Section 5, we question whether the lack of unicity poses a problem (5.1) and whether it is achievable in the first place (5.3). In summary, we think that, as a community, we could: (i) clarify the epistemic goals of an explanation (potentially accepting the lack of unicity), (ii) investigate less permissive criteria, and (iii) embrace broader frameworks like the inner interpretability framework.

On “the challenge lies in defining criteria that distinguish valid explanations from misleading ones”: We have removed this line from the manuscript, as it did not accurately reflect our positioning.

In addition, we have applied most of the suggested minor changes, including the following:

• Stronger justifications as to why a larger architecture could also lead to greater overparameterization have been given (l. 398-401).
• The “near impossibility of exhaustively searching all possible algorithms” has been rephrased as "intractability" (l. 49).
• The distinction between circuit-finding heuristics and approximation algorithms has been clarified (l. 53-57).
• We have edited several citations: removal of the Van Rooij citation, and addition of Vilas et al., 2023 on vision transformers in MI, Adolfi et al., 2024 on the inner interpretability framework, and Adolfi et al., 2024 on computational complexity.
• We have clarified the notation in Definition 4 (l. 197).
• The initial statement about computational complexity has been adapted and sourced (l. 204)

Finally, the source code used for our experiments will be released upon publication.

Please let us know if we can provide any additional information to clarify our work.

Dear Reviewer,

We would like to express our gratitude for the thoughtful and thorough review of our manuscript. The detailed feedback has been valuable in improving the overall quality of the paper. We also appreciate your suggestions regarding computational complexity and related references.

We are confident that the revisions we have made reflect your input and contribute to a clearer and more robust manuscript. If we have addressed your concerns, we would be grateful if you might consider revisiting your score.

Thank you once again for your constructive feedback.

We greatly thank the reviewer for their time and feedback, and for highlighting the relevance of our work and the discussion on the limits of its scope.

To address the lack of examples, we have added a new appendix B, which contains more examples of circuits, interpretations, algorithms, and mappings found. In addition, we have clarified the text in the article to emphasize the fact that the example circuits given in Figure 2 were also found as part of our experiments. We hope this will help readers visualize and assess the quality of our work.

We have clarified our position concerning identifiability in two ways; first, we now argue in section 2.4 that the need for identifiability stems from a strong intuition rooted in human psychology and that while not clearly stated, it is already an implicit assumption of MI. The paper now frames the unicity of explanation as a property that we might expect but do not find. Second, we have extensively rewritten the discussion (Section 5), in which we describe in which scenarios the lack of unicity may or may not matter. In this context, we now cite the recent debate about the interpretability illusion as an example of why explicitly stating the purpose of the explanation is relevant to the question of identifiability.

In addition, we would like to clarify that the what-then-where approach in our work does not rely on DAS. While DAS only explores a part of the search space via gradient descent and therefore produces an approximate solution, in our experiments we find all exact maximizers of IIA by exhaustive enumeration. Furthermore, we compute the true IIA value, while DAS typically only approaches it by randomly sampling inputs.

Finally, the source code used for our experiments will be released upon publication.

Please let us know if we can provide any additional information to clarify our work and thank you again for the feedback.

Dear Reviewer,

We sincerely thank you for your thoughtful and careful consideration of our work, as well as for maintaining the score. We appreciate the time spent reflecting on the details, and we're glad that our clarifications helped address the concerns about the reliance on DAS. Should any further questions arise, we are happy to provide additional insights.

Thank you again for your time and contribution.

We sincerely thank the reviewer for their time, insightful comments, and kind remarks about the clarity and interest of our work.

On the definition and measurement of “incompatibility”: We have expanded the definition of incompatibility in the manuscript (section 2.4). We broadly define fully incompatible explanations as pairs of explanations that share the same epistemological goal but lack overlap in location (within the neural network, the "where") and/or differ in internal states (e.g., algorithms, the "what"). We agree that compatibility can also be partial, but this is out of the scope of this work and we leave open formal definitions and measurements of compatibility for future research. Our study primarily focuses on at least partially incompatible explanations (i.e., no full overlap). The number of incompatible explanations that we find is particularly large and examples of fully incompatible explanations have been added in Appendix B.

On the distribution of the training set: We have added an experiment where the training procedure is repeated with varying training distributions. The results, in Appendix C.5, suggest that training biases do not significantly affect the conclusion.

On training error and overfitting: We have expanded our previous experiment on training dynamics, in which we analyzed scenarios where training was stopped at different loss cutoffs. The manuscript now includes additional figures to that extent (Appendix C.4). Our findings indicate that approaching perfect training error does not substantially alter the number of explanations found. Interestingly, many incompatible explanations can also be identified in randomly initialized (untrained) networks that just happen to implement the target logic gate.

Finally, the source code used for our experiments will be released upon publication.

Please let us know if we can provide any additional information to clarify our work.