Measuring Per-Unit Interpretability at Scale Without Humans

Dear Reviewer gaJB,
Thank you for your valuable feedback and for praising our paper as a “triumph of the genre” with ”outstanding quality” and finding it ”solve[s] a longstanding problem”. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you feel even more confident in our work.

Q: “The way the query images are selected is only mentioned in the appendix”
A: Thank you for your suggestion. We will make use of the increased page limit in the camera-ready version of the paper to move this information to the main text.

Q: “Why [are] weakest-activating samples [not] more random?”
A: We agree with you that it is not obvious why the least activating samples are also clustered, and all contain the same visual feature. However, one can think of these features as “counterparts”/”anti-features” to the features displayed by the maximally activating samples: A unit only has high activation if just the feature but not the antifeature is present, which potentially allows the network to learn more specific feature detectors.

Q: “Is there a way to quantify how superposition affects MIS?”
A: That’s a great question and suggestion! While we can’t think of a precise quantification right now, we can offer a thought experiment. Polysemantic units respond (strongly) to multiple concepts. If those different features elicit similar activation ranges, the query and reference images used to compute the MIS differ, resulting in harder tasks and lower MIS. We think that quantifying through human studies how well a drop in MIS is correlated with units being polysemantic will be an interesting follow-up for our work. Thank you for the suggestion!

Q: “why [might] one [...] expect "wider layers to be more interpretable"?”
A: Thanks for raising this question! In this plot, we compare the relation of a layer's relative width and its interpretability, i.e., we ask whether wider layers of a network are more interpretable than narrower ones of the same network. We chose this relative comparison exactly to circumvent your concern. We think our observation might be explained by the superposition hypothesis: Narrow layers do not have sufficient capacity to represent different concepts individually through single units and instead have to leverage polysemantic units. We will expand on this connection in the camera-ready version of our paper.

Q: “thoughts on training autoencoders to directly optimize for MIS and reconstruction loss?”
A: We think that using our proposed MIS to increase the interpretability of networks is very exciting — both for auto-encoders used to make large networks interpretable and for directly making networks more interpretable. While the current computation of the MIS can be used for non-gradient-based optimization (e.g., hyperparameter grid search), optimizing it scales inefficiently with parameter count. A challenge for future work will be to find a differentiable approximation of the MIS that can be optimized using gradient descent, circumventing the efficiency issue. We believe such an approximation can be defined when training with sufficiently large batch sizes, and hope to explore this further in follow-up work.

Q: “Do the red points in Fig. 3 represent the location of models tested by Zimmermann et al.?”
A: Yes, that is correct. We will update the caption to ensure future readers do not mistake them for the results of Zimmermann et al.

Q: “What models form the pareto frontier in Figure 4A?”
A: Thank you for this suggestion! We determined the Pareto frontier of Fig. 4A using the paretoset python package and will include the following table in the camera-ready version of our paper:

We found it interesting that although purely convolutional networks with high accuracy exist, all points on the Pareto frontier with high accuracy belong to transformer architectures.

Q: “In what sense is the word "define" used in L122”
A: We use the word define here in the same sense as one defines a statistical/machine learning model. Would you prefer the word “model” instead of “define” here?

Q: “Figure 14 [...] difference in y-axis scaling”
A: Thanks for the pointer, we will update this figure in the final version of the paper to use an aligned y-axis.

Q: Typos in L195 and caption of Fig. 17
A: We corrected these typos.

Dear Reviewer j42j,
Thank you for your detailed feedback. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you can confidently increase your score

Q: “find the underlying task to easy”, “does not account for polysementacity”
A: You are right that the underlying psychophysical task checks for a rather rudimentary level of understanding. Note, however, that understanding units at this level is necessary for any deeper understanding. This is important because experimental results show that human understanding of units at this level is rather brittle: Zimmermann et al. (2023) showed for two models that by performing the psychophysical task for slightly less extreme query images, human performance drops strongly (see Fig. 6/7 of their paper). We can reproduce this finding with our MIS for many more models (see Fig. 1 of the general response), showing that this task is not trivial to solve. Further, the task partially accounts for polysemanticity as the most extremely activating images of polysemantic units might not correspond to a single but various concepts, making the task even harder to solve. In conclusion, this means there is still ample room for improvement in future models.

Q: “application of interpretability score”, “(advantanges of) optimizing for MIS”
A: Thank you for raising this question. We see two types of practical applications for our MIS that go beyond analyzing and understanding neural networks: (1) Optimizing neural networks directly to become more interpretable via gradient descent. (2) Performing model selection based on interpretability. (2) Tuning hyperparameters of other interpretability tools to make them explain networks better. While having a differentiable version of the MIS is required for (1) and would surely benefit (2) & (3), note that a non-differentiable metric still provides valuable insights enabling the latter two directions (although potentially less efficiently). As an example, we performed experiments with sparse auto-encoders (L231ff & Sec. I), revisited and investigated inconclusive results of previous papers, and performed hyperparameter selections. We use the increased page limit of the camera-ready version to highlight these results more in the main text.

Q: “range of [MIS] values is tight”
A: On a per-unit level, we find that the MIS spans the entire theoretical value range (~0.5 to 1.0) (see Fig. 2B/C). When averaged per model, the effective range indeed becomes tighter. We argue, however, that this is not an issue of the metric but instead shows that models only trained for good downstream performance all learn similar representations (https://arxiv.org/abs/2405.07987) with mediocre interpretability. With an increased interest in interpretable models, we expect future models to achieve higher MIS values. By choosing less extremely activating samples as query images (see Fig. 1 of general response), we can also increase the task’s difficulty to have a meaningful signal also for more interpretable models.

Q: “[Incorporation] with other popular ideas in current interpretability literature?”
A: A particularly popular topic at the moment is SAEs. As described above, our MIS can be used to make model selection and hyperparameter tuning of SAEs more efficient, enabling large-scale sweeps. Further, it is conceivable to use the MIS as a guiding signal when finding interpretable circuits in a network: by excluding particularly uninterpretable units from the search, one might reduce the computational cost of finding circuits. Finally, as the MIS also works with explanations other than dataset examples (see Appx. E), it can be used with future explanation methods, too (e.g., MACO [1]).

Q: “Why do more accurate models have lower MIS?”
A: This is an important question. We hypothesize that this is related to the phenomenon of superposition: With limited capacity, one way for models to obtain higher downstream performance is to represent features in superposition/entangle them. However, at the same time, this makes units harder to interpret as they don’t correspond to individual features anymore, explaining the lower MIS. We politely disagree with your assessment that this result makes it difficult to sell “optimizing for MIS”. First, note that we see only a correlation, which does not mean that there has to be a tradeoff between performance and interpretability, as this would assume a causal relation between these two variables. Second, note that if accuracy and MIS were positively correlated, then there would be no need to optimize for MIS as one would get this for free. On the contrary, our results show clearly that one should not hope to automatically get very interpretable models by only optimizing for high accuracy. Instead, we need to optimize for both accuracy and interpretability/MIS.

Q: “[are] nodes with higher MIS are easier to name?”
A: Interesting question! To verify the correctness of our MIS, we used the human psychophysical data of [50]. This data also contains scores indicating how confident humans were when making their choices. Regarding your question, one might say that if humans find it easier to name units, they will have high confidence in solving the 2AFC task. Interestingly, we find a high linear/rank correlation of 80% between this confidence score and our MIS. Thus, we conclude that nodes with higher MIS tend to be easier to name.

Q: “correlation between the per-unit HIS and MIS scores for the new human study?”
A: Based on your suggestion, we computed the correlation between MIS and HIS for the new human experimental data. Specifically, we compute the correlation for all units used for creating Fig. 2C and again find a high correlation ($\rho_p = 0.85, \rho_s = 0.81$).

Dear Reviewer P5jG,
Thank you for your valuable feedback and for praising our paper as a “significant work” with ”interesting results”. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you can confidently increase your score

Q: “how do the authors envision their metric in a computer-human interaction (CHI) context?“
A: That’s a great question! We completely agree with you that, at its core, interpretability is a human-centered field, and, thus, humans need to be involved eventually. Therefore, we ensured that the proposed MIS explains human interpretability annotations well. We see numerous opportunities for the MIS to interact with/support humans: (1) The MIS might be used to increase a model’s interpretability to simplify the human’s job, e.g., through model selection, hyperparameter tuning or directly optimizing for it (where the latter would benefit from a differentiable approximation of the MIS). (2) The MIS can save time when interpreting models: Instead of attempting to understand incomprehensible units, we can start with particularly easy ones (based on their MIS) and allocate our time accordingly. (3) Such an MIS ordering can also be helpful when finding neural circuits in the network as excluding very uninterpretable units from the search reduces the combinatorial complexity and, thus, computational cost.

Q: “Do you expect HIS and MIS to have the same degree of correlation in, say, text classification tasks?”
A: This is a very interesting question! Assuming access to a sufficiently human-like perceptual similarity function, we expect the MIS to generalize to various data modalities. Given the tremendous progress in language modeling/embedding, we are optimistic the MIS will work on text data, too. Testing this hypothesis requires extensive human psychophysical experiments. We will include this exciting experiment in the outlook paragraph of our paper and hope it will inspire future work.

Q: “what about out-of-distribution (OOD) samples?”
A: Thanks for asking this question. Our experimental results indicate that OOD samples cause no problems. In Appx. E, we demonstrate that the MIS is still highly correlated with the HIS when using synthetic feature visualizations instead of dataset examples as reference images. These images pose a substantial distribution shift as they look very different from natural images. Therefore, we conclude that the MIS works correctly also for OOD samples.

Q: “reference for the claim "googlenet [..] is widely claimed to be more interpretable"?”
A: This statement refers to the fact that many interpretability papers focus on this network, producing more and more insights into how it operates. We see now that our statement was imprecise and will revise it accordingly.

Q: “why would interpretability decrease after the first training epoch?”
A: We have no definite answer to this question yet but hypothesize the following: This could be a sign of learning dynamics and the order in which features are learned. After initialization, the network can improve the fastest by learning very simple feature detectors (e.g., colors, simple geometric shapes), as those are weakly correlated with certain classes (e.g., blue colors increase the chance of seeing a fish). Those features are easy for humans to understand. Throughout the training, these feature detectors are replaced with more complex ones that are harder to decode. As suggested by reviewer gaJB, in later training epochs, the network might also tend to a state of stronger superposition to increase classification performance with the cost of decreased interpretability. In Fig. 3 of the general response, we show visual explanations of units with strong MIS drop between the second and last training epoch. We will use the increased page limit of the camera-ready version to discuss this hypothesis in the main text and include these visualizations in the appendix.

Q: “Why are wider layers more interpretable?”, “distributional study for the width”
A: Our data shows a moderate increase of the per-layer interpretability (i.e., per-unit scores averaged per layer) with increasing layer width. We see the same trend when instead of plotting the per-layer-average we plot the 5th or 95th percentile per-layer (see Fig. 2 of the general response). This suggests that the overall MIS distribution moves to higher values with increasing layer width and the effect is not dominated by few outliers.

Overall, we hope to have answered all of your questions. If you are satisfied with our responses, we would appreciate it if you would increase your score.

Dear Reviewer 8Jb5,
Thank you for reviewing our paper. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you feel confident in increasing your score.

Q: “I do not find any of the conclusions very exciting”
A: We see our contribution as part of a wider community effort to develop better techniques for understanding the inner workings of neural networks. As such, the MIS is better understood as improving our “microscope” for analyzing networks, rather than providing any specific conclusion. However, as you noted, our improved “microscope” already surfaces a range of interesting phenomena that have yet to be explained. It’s such unexplained phenomena that fuel and guide scientific progress, highlighting how relevant MIS is to the study of networks.

The improvement MIS provides is quite a step-change: compared to the manual approach of prior work (e.g. [50]), we can now test several orders of magnitude more units & models. This is why we could detect the anticorrelation between a model’s accuracy and interpretability, a relation that [50] failed to substantiate due to a lack of statistical power. It’s these kinds of analysis and phenomena that require automated interpretability metrics and which have been sought after for some time (e.g. see Rev. gaJB or [6]). Now, for the first time, such large-scale interpretability analysis is possible with our MIS.

Q: Relation to [50], “technical contribution”
A: Before our paper, there was no work that allowed large-scale quantification of per-unit interpretability in vision models. With respect to [50], please note: (1) The manual approach of [50] (and other prior works) does not scale at all. Our method is the first to scale their type of evaluation up and we succeed by finding a simple but clever way to automate it. We hope you can agree that, ultimately, a new method should be mainly evaluated by its potential impact. This impact is visible, e.g. by the fact that our study finds a clear anticorrelation whereas [50] could only produce inconclusive results due to multiple orders of magnitude fewer measurements. (2) We’d also like to emphasize that our technical contribution is far from obvious. It’s not clear why a global perceptual metric (DreamSim) could fit human responses in our 2AFC task. In particular, one might expect (e.g. [7]) that humans solve the task by searching for common local patterns in the reference images and then comparing the most common ones to those of the query images. Hence, the fact that a global perceptual metric fits human responses so well was quite surprising to us.

Q: “Differences to MAIA”
A: Thanks for asking this important question. Approaches such as MAIA and our MIS tackle completely different problems: MAIA is an automated explanation method, i.e. it tries to find explanations of what a unit does. But it does not allow quantifying how interpretable the unit is. On the contrary, our MIS is an interpretability metric that tells how interpretable a unit is given some explanations. Such an interpretability metric enables practitioners to increase the level of interpretability of a network, e.g., by model selection or hyperparameter tuning of the model or of an interpretability method.

Q: Relation b/w “diversity in exemplars” and legibility
A: Our MIS leverages an established 2AFC task used in multiple prior works [6, 49, 50]. We will integrate more information from these works on why this task measures how legible explanations are (i.e. how well they explain a unit). Their reasoning can be summarized as follows: The task requires humans to reason about positive (and negative) explanations, to identify the positive (or negative feature) shared by the explanations, and to recognize this feature in the correct query image. Please note that these features rarely correspond to completely unambiguous, clear-cut semantic classes, but identifying the common feature can be challenging. If the explanations are so diverse that humans fail to identify a shared feature, it shows that they cannot understand what the unit is firing for.

Q: “[Extension] to full distribution”
A: The MIS can very easily be extended to test more than just the extrema of the activation distribution: Instead of choosing the most extremely activating samples as query images, we can use less strongly activating ones and sample from other parts of the activation distribution. Please see Fig. 1 of the general response for a version of the paper’s Fig. 3 where we chose query images from the 98th/2nd percentile. As the human understanding (measured by HIS/MIS) even of the extrema is still limited and performance breaks down a lot when moving away from the extreme, we suggest using the MIS with images near the distribution’s tails (e.g. 98th or 99th percentile) to get a strong signal. Once models (let it be base models or SAEs) or explanations improve, it can be insightful to test larger parts of the distribution, too.

Q: “[What about] textual descriptions”
A: We are optimistic that the MIS can be generalized to textual descriptions too. With the recent progress in modeling language-vision models, it is conceivable to replace DreamSim with a language-vision encoder such that the MIS is based on the similarity between textual descriptions and query images. To keep our paper focused, and due to the lack of human interpretability annotations for such textual explanations, we decided to leave exploring this extension to future work. We will add this discussion to the final version of our paper.

Q: “SAE experiment performed on interpretable layers”
A: We chose a layer representing the median interpretability of GoogLeNet to neither make the experiment too hard nor too easy. We will re-run our experiments on a less interpretable model/layer and include it in the final version of our paper.

Thank you for your reply.

• if the conclusions in the paper are not the main contribution but rather the metric itself, please describe future potential use cases for it.

• Regarding the usage of DreamSim: because the aggregation over all image pairs is by taking an average, I believe there would not be much difference between the scheme you mentioned and the current implementation of MIS (DreamSim had shown to represent a global space for perceptual similarity that goes beyond pairwise comparisons.) Nevertheless- if the procedure you describe is indeed how humans perform the task, why not construct MIS accordingly?

• technically- I understand how MIS can be easily expanded to other "activation level" exemplars- but is it meaningful at these levels? The figure added is of 98% percentile, which is still very high. What about the percentages?

• will be great to see SAEs results during the discussion period if possible, thank you for the effort on this.

Dear reviewers,
Thank you again for your insightful feedback and engaging discussions throughout the review process! We are grateful that you recognize the potential and value of our work and now unanimously give this paper a positive score.

Based on your initial reviews and subsequent questions, we have made several revisions to enhance the clarity and impact of our paper (see previous post). In addition to the changes outlined in our initial responses, we have implemented the following for the camera-ready version:

• We conducted additional SAE experiments on less interpretable layers of another network (Rev. 8Jb5)
• We further emphasized the numerous potential practical applications of our MIS in both the introduction and discussion sections (Rev. 8Jb5)