TeLLMe what you see: Using LLMs to Explain Neurons in Vision Models

Dear Reviewer NqpP,

Thank you very much for taking the time to thoroughly review our work and for giving us the opportunity to clear up some of the questions.

Weakness 1)

Thank you very much for reading our work so thoroughly. All typos have been fixed.

Weakness 2)

In weakness 2 you raise an important point about the generalizability of our method. To go beyond the existing research in this area for vision models [1, 2], we ran additional experiments focused on exploring the extend of the generalizability. However, since the limited time we have the rebuttal period, we were only able to run one additional experiment for this. To maximize the insights, we are using the proposed method to explain neurons in a VGG method trained on CIFAR-100, but with the few-shot examples used in our main experiments (Section 4.1). This should show two things. Firstly, whether the method works well for models trained on other datasets besides ImageNet; secondly, whether it is possible to use few-shot examples from different models and datasets.

The Correlation score achieved by TeLLMe (ours) with the Weight-Label set-up and CLIP-Dissect when explaining neurons of the second last (2nd last) and last (last) hidden layers of a VGG model trained on CIFAR-100.

Weakness 3 & 4)

Following your request for additional validation of both the model and our assessment strategy, we have used CLIP-Dissect and the proposed method to label 6 additional neurons in an Alexnet model trained on ImageNet and conducted a survey with 20 anonymous, unpaid, volunteers. The participants were tasked to decide which explanation (un-labeled in random order) better fits the 25 images shown (which are the imagenet validation images with the highest activation for the neuron in question) for the 6 newly explained neurons, and the 4 we showed in Section 4 of our paper. Before delving into the results, it is worth pointing out that only assessing the explanations based on the top 25 images might to be entirely accurate, as many neurons show signs of significant polysemanticity [3].

These results will also be included in the camera-ready version of the paper.

Survey Results

As can be seen in the table above, for 100% of the neurons tested, TeLLMe's explanation was preferred. Overall neurons and volunteers, TellME's explanations received an average preference of 77.28%, compared to 22.72% for Clip-Dissect.

References

[1] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi:10.23915/distill.00024.001.https://distill.pub/2020/circuits/zoom-in.

[2] Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations, 2023.

[3] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022.

Dear Reviewer 1NcR,

Thank you very much for taking the time to thoroughly review our work and for giving us the opportunity to clear up some of the questions.

Weakness 1) As you correctly point out, when trying to understand neurons in vision models, we have a strong suite of visualization techniques at our disposal. However, our proposed method has two distinct advantages over manually inspecting the top activating images.

Firstly, manual inspection is not scalable. This means that if one wanted to remove a specific concept from the classifier (such as "cats"), with our method, it is possible to automatically explain all neurons in the net and simply remove those whose explanations contains the word "cat".

Secondly, as can be seen in Figure 8 and Figure 10, on average, the explanations created by the LLM perform better than the ones created by humans for the few-shot examples. However, for the latter, it is worth pointing out that we created the few-shot examples ourselves, so to draw strong conclusions w.r.t. the quality of human labeled and LLM labeled explanations, additional experiments with a better methodology for specifically testing this hypothesis would be needed.

Weakness 2) Cost is indeed an excellent point. Using the GPT models can be prohibitively expensive for practitioners and might not fully justify the performance increase over cheaper methods such as CLIP-Dissect. However, the quality of open-source models is rapidly catching up to their closed-source counterparts. Whilst our experiments showed that Llama-2-70B (Appendix C.2) is not capable enough to be used for our method, a number of models that have been released since then have been able to outperform GPT-3.5 on public benchmarks.

To test whether these models work for our method, we re-ran the main experiments (Table 1, 7, 8, 9) using the very recently released Goliath 120B model [1] and the weight-label method. As can be seen in the Table below the model actually outperforms GPT-3.5-turbo on 75% of the neurons we tested it on, and it does so by up to 30 percentage points. In the camera ready version of our work, we will adjust Appendix C.2 accordingly. Also, it is worth pointing out that this increase in performance is in-line with the observations in [2], where the authors found that the quality of the explanations increased with the capability of the LLM (in their case they compared GPT 2, 3, 3.5, 4).

Additional Results:

Neuron 150 (2nd last layer): GPT-3.5-turbo -2.17%; Goliath-120B 29.81%

Neuron 489 (2nd last layer): GPT-3.5-turbo 51.58%; Goliath-120B 42.30%

Neuron 150 (last layer): GPT-3.5-turbo 24.65%; Goliath-120B 27.51%

Neuron 489 (last layer): GPT-3.5-turbo 23.67%; Goliath-120B 31.07%

The Correlation score achieved by GPT-3.5-turbo (these are the ones reported in the original draft), and the open source alternative (Goliath-120B). Both LLMs are used in the Weight-Label framework.

References

[1] https://huggingface.co/alpindale/goliath-120b

[2] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023

Dear Reviewer KpfM,

Thank you very much for taking the time to thoroughly review our work and for giving us the opportunity to clear up some of the questions.

Weakness 1) This is a fair thing to point out. We were following the set-up used in [1], but after your review, re-ran some of the AlexNet (weight-label) experiments with the original, unscaled, weights. Unfortunately, GPT-3.5-turbo is not strong enough to reconstruct the terms it is supposed to simulate weights for (we had a similar problem when using Llama-2-70B (See Appendix C.2)), and thus makes it infeasible to use un-scaled weights until stronger foundation models are published.

Weakness 2) In addressing the concern that our approach heavily relies on GPT, it's important to clarify how the assessment process reflects not just the capabilities of the GPT model, but also the validity of the explanations generated. As detailed in our paper, we utilize GPT primarily as a tool for simulating neuron activations/weights in response to specific inputs and their corresponding explanations (Sections 3.1.2 and 3.2.2). This allows us to compare these simulations with actual neuron behavior, using various metrics to evaluate the explanations' quality (refer to Table 2 in Section 4.3).

It's crucial to note, however, that any inaccuracies in GPT's simulation of neuron activity might affect the simulated results, which in turn impacts the perceived quality of the explanations. However, these inaccuracies in simulation would degrade the scores reported, rather than improve them, and thus, the results shown in Tables 1 and 2 should be considered conservative lower bounds of the actual explanation quality. Additionally, to mitigate the risk of over-relying on GPT for evaluation, we include a manual inspection of the top 24 activating images for certain neurons, as illustrated in Figure 4. While this method is subjective and not scalable, it provides a valuable sanity check for our automated evaluation methods, ensuring that the explanations hold up under qualitative scrutiny.

Weakness 3) Thanks for giving us the opportunity to clarify, Table 4 shows the results of following different strategies for choosing Caption-Activation pairs for the few-shot prompts. If we understand correctly, you suggest to switch the actual weights of, for example, AlexNet, with random ones and testing whether GPT comes up with well-scoring explanations. To that end, we ran additional experiments with a randomly initialized AlexNet and the weight-label set-up. Quantitatively, as expected, we get an average correlation score of around 0 (which is caused by the network predicting a weight of 0 for most outputs; this seems makes sense as no patter was found so it relies on the few objects it was able to list in the explanation, rather than an umbrella term).

To gain some further insight, here is one of the explanations the model came up with "the main thing this neuron does is find various modes of transportation, such as aircrafts, vehicles, and submarines. It also identifies common objects and locations found in daily life, such as cash machines, ice cream, flowerpots, restaurants, and street signs. Additionally, it focuses on protective gear and equipment like crash helmets, shields, and rugby balls". Interestingly enough, the model tries to list more items (since there is no clear pattern), in an attempt to still achieve a good explanation for what is going in. This is a very interesting insight, and one we will include in the camera ready version of the paper. Thank you very much for your thoughtful suggestion.

Weakness 4) This is a great suggestion, and one we originally had in mind when writing this paper. However, as pointed out in Section 3.2.3, unfortunately using GPT to explain all neurons in the network (something that would be necessary in order to fully delete certain concepts) is prohibitively expensive, and strong open source models, are not yet capable of being a drop-in replacement (See Appendix C.2).

[1] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023

Dear Reviewer xVYG,

Thank you very much for taking the time to thoroughly review our work and for giving us the opportunity to clear up some of the questions.

Weakness 1) As you point out, using visualization techniques has lead to a good level of explainability in the field of computer vision. However, the main short-coming of this technique is that it isn't scalable. In many ways, the purpose of this work is to extend upon the visualization literature by "outsourcing'' the visualization->explanation pipeline from humans to LLMs, thus, for the first time, making it feasible to explain all neurons in a vision neural network.

Weakness 2) Thank you for the feedback. If you have specific suggestions on how we could improve the readability of our work, please don't hesitate to let us know, so that we can incorporate them.

Weakness 3) You are right in pointing out that a lot of information in ViTs comes from the attention mechanism. However, the purpose of this work is to explain neurons such that the creator is able to manually edit neural networks and better understand their decision making with respect to medium- and high-level features. To that end, explaining the attention mechanism is not a necessity ([1] follows the same logic for text-based transformers).

Question 1) The results presented in Table 2 (Section 4.3) aim to provide additional context to those in Table 1 (Section 4.1) by extending the measurements of success used from the correlation between the simulated activations/weights and the real ones to accuracy, MSE and MAE. Though, as pointed out in [1], the correlation score will provide most value, the average MAE (2.2) and accuracy (30%) suggest that the model is not only able to determine the trends in activations, but also, to a decent extend, the actual activation values (something that would not be reflected in the correlation score).

Question 2) Assessing the correctness of the GPT explanations is indeed a challenge, especially at scale. In the paper we offer two solutions to this. Firstly, in Sections 3.1.2 and 3.2.2, we introduce our strategy for using GPT as a simulator of the neuron activations/weights given a certain input and the generated explanation. We can then compare these simulated values to the actual activations/weights and assess the quality of the explanation using a number of different metrics (See Table 2 in Section 4.3).

However, though this is scalable, it is likely not a perfect estimation of the quality of the explanation, as any mistakes done by GPT when simulating the neuron activity, will degrade the accuracy of the simulated activations/weights, independently of the quality of the explanation. It is worth pointing out that this will only degrade the performance, and thus, the results presented in Tables 1 and 2 should be considered a lower-bound of the actual quality of the explanations. Secondly, in Figure 4, we manually inspect the top 24 activating images for an explained neuron. Though subjectively, the explanations seems very apt, this method is not scalable and, thus, we only use it as a sanity check for the automatic evaluation.

Question 3) This is an excellent question, and slightly beyond the scope of this work. Generally speaking, the purpose of this method is to find explanations for single neurons that capture the activation patters (especially w.r.t superpositions [2]). However, as you point out, in most cases, multiple neurons will have the same 'task' of recognizing object x. Thus, if a single neuron that triggers for a specific concept is removed, this does not necessarily change the network behaviour.

However, if one were to explain all neurons using our method, fully removing a concept from the network should be as easy as removing all neurons whoms explanation contains the object one wants to remove. Moreover, using the neuron explanations in combination with the absolute magnitude of activations of each neuron should give good insights into the 'thought' process of the network. I.e. given the network recognized an object as x, we can check what the explanation for the highest activating neurons are, and thus determine that the networks decided to classify the object as x, because it activated highly for features a, b and c.

References

[1] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023

[2] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022.