Describe-and-Dissect: Interpreting Neurons in Vision Networks with Language Models

Thank you for the thoughtful review and good suggestions! Below we discuss some concerns you had and how we have addressed them.

#1 Effect of each part of the pipeline

Thank you for the suggestion!

We have discussed the effect of Stable Diffusion’s synthetic images in the original draft, please see sec 4.3 and ablation study in Table 5, Fig 4, where we find that on average DnD with the full pipeline is rated 0.13 (on a scale of 5) higher than DnD without Best Concept Selection.

Following your suggestions, we have conducted additional ablation studies on attention cropping, image captioning, and GPT summarization and we describe the results in below #1(a), #1(b) and #1(c) respectively.

#1(a) Ablation study on “Attention Cropping”

We perform an ablation study on the effect of attention cropping to DnD performance in Appendix B.2. We qualitatively observe this change in Figure 11, noticing that the primary concepts for layer 3 neuron 475 and layer 2 neuron 467 are brought out more with attention cropping.

#1(b) Ablation study on “BLIP Image Captioning”

We also perform studies exploring the effect of BLIP image captioning on our pipeline. In Appendix B.3, we compare the performance of BLIP and BLIP-2 as image captioners. We found that BLIP and BLIP2 generally give similar results as shown in below Table R1, where we compare the cosine similarity scores between the neuron descriptions generated by BLIP and BLIP2 in a ResNet 50 model. For each layer in the model, we compute the mean CLIP cosine similarity between BLIP and BLIP-2 labels for 50 randomly chosen neurons. Similar conceptual ideas between both models are reflected in the high similarity scores.

It is also interesting that we found there are some cases where BLIP2 is too specific in its captions, causing it to miss the bigger concept and underperform compared to BLIP as seen in Figure 13 in Appendix B.3 in the revised draft. We can see that with neuron 248 in layer 1, the concept should clearly be something along the lines of “black” or “dark.” DnD w/ BLIP is able to capture this with its label “low-light scenes” while DnD w/ BLIP-2 gives a worse label of “monochrome timepieces.” We can see that BLIP-2’s captions are trying too hard to see things in these simple images, detecting the color white and objects where there are none.

Table R1: Mean Cosine Similarity Between BLIP and BLIP-2 Labels.

To further measure the effect of image captioning in our pipeline, we took your suggestion and experimented with replacing image captioning with a fixed concept set in Appendix B.4. We replace BLIP with CLIP, using CLIP to find text in the 20k concept set that are most similar to the highest activating images of a neuron. The text matched with each highly activating image serve as our image captions, which we then process through the rest of our pipeline. Our qualitative results in section B.4 with Figure 14 show that these short, often single-word captions aren’t descriptive enough for GPT-3.5-Turbo to find semantic similarities in. The more descriptive captions with BLIP yield better results with our pipeline. For quantitative results, we gather the labels produced by this pipeline for all neurons in ResNet-50’s FC layer and follow the experiment #2(b) to find the similarity between the labels and ground truths. We compare to our regular DnD pipeline and display the results in Table 9 (Table R2 below). Our original pipeline slightly outperforms DnD w/ CLIP using both CLIP cosine similarity and MPNet cosine similarity. Note that the final FC layer is likely where a fixed concept set would perform best, as its concise and common text elements are more likely to match up with well-defined class ground truths than the often abstract and polysemantic concepts of intermediate layer neurons. As such, we find the difference to be qualitatively larger on hidden layers and as such image captioning is essential for improving performance.

Table R2: Mean FC Layer Similarity of CLIP Captioning. Utilizing a fixed concept set (t = 20k) to caption activating images via CLIP Radford et al. (2021), we compute the mean cosine similarity across fully connected layers of RN50. Final labels that do not express concepts are given a similarity score of 0. With both CLIP cosine similarity and MPNet cosine similarity, we find the performance of DnD w/ CLIP Captioning is slightly worse than BLIP generative caption.

#2(d) Application: Finding Classifiers for New Concepts

Finally, we provide a novel application of neuron descriptions to find classifiers for unseen tasks in Appendix B.8, showing we can find relatively good classifiers for unseen tasks. To find a neuron to serve as a classifier, we found the neuron whose description was closest to the CIFAR class name in a text embedding space. Using DnD descriptions we found classifiers with average AUROC of 0.7375 on CIFAR10 classes, and 0.7606 on CIFAR100 classes. In comparison, using MILAN descriptions the average AUROC was only 0.5906 for CIFAR10 and 0.6514 for CIFAR100, likely due to their more generic descriptions. While this is mostly independent as an application of our method it also serves as another quantitative metric showcasing our descriptions are more useful than descriptions generated by alternative methods.

[1] Oikarinen & Weng, Clip-dissect: Automatic description of neuron representations in deep vision networks, 2023

[2] Bau et al., Network dissection: Quantifying interpretability of deep visual representations, 2017

[3] Kalibhat et al., Identifying interpretable subspaces in image representations, 2023

[4] Hernandez et al., Natural language descriptions of deep visual features, 2022

#3 Clarification regarding the role of concept sets in DnD and related works

One thing we wanted to clarify is that DnD does not utilize a sort of "augmented" concept set. In fact, our method is entirely concept set free. Instead, we generate image captions using BLIP and semantically combine these captions into neuron labels using GPT-3.5-Turbo. In regards to the respective concept sets used in the original publications, we utilize those discussed in their respective papers. For CLIP-Dissect, we employ the 20k concept set as that is the most expansive one discussed in its respective paper which gives it the highest performance possible on intermediate layers of networks. Network Dissection only utilizes the Broden labels which is reflected in our experiments, and MILAN, like DnD, is generative and thus does not employ a concept set.

#4 Summary In summary we have:

• #1 Performed further ablation studies for all components of our pipeline to explore the impact of each component
• #2 Added quantitative results through our experiment on the fully-connected layer of ResNet-50, as well as two other qualitative evaluations
• #3 Clarified the role concept sets play in our method and other baseline works.

We believe that we have addressed all your concerns. Please let us know if you still have any reservations and we would be happy to address them!

#6 Hyperparameters

Following your suggestion, we have gone through in the revised draft and defined what values we set certain hyperparameters to if we didn’t already in the original draft.

This paragraph is dedicated to addressing your question regarding the influence of certain hyperparameters and our reasoning behind setting them to the values we did.

• $\alpha$ was set to 4 as after some testing we found that value to be reasonable. Less than that and we could miss out on important concepts in an image that are causing the neuron to activate; more than that and our top activating images could be dominated by crops of one image, preventing us from taking the other highly activating images into account.
• $K$ was set to 10 as we found through testing that a value lower than this could exclude valuable information in other highly activating images and a value higher than this could introduce spurious concepts that confuse GPT.
• $N$ was set to 5 as we observed that GPT rarely produced over 5 distinct candidate concepts, so increasing $N$ beyond that would be superfluous. Decreasing $N$ below 5 could possibly eliminate accurate potential labels from being fed into Best Concept Selection.
• $Q$ was set to 10 to balance quality with runtime/computational cost, as while the quality of Best Concept Selection increases with $Q$, so does the runtime.
• $\eta$ was set to 0.5 as, intuitively, it makes sense for at least the majority of the space covered in our bounding boxes to be distinct; otherwise the attention crops would all be very similar. If $\eta$ was less than 0.5, we could find a problem of the crops being too small, preventing the image captioner from perceiving comprehensive concepts.

Finally, we clarified in the revised draft that, unless specified, the summary function g we use for our experiments is spatial mean. Spatial mean is used as the summary function in all experiments except for the MILANNOTATIONS experiment in section B.7 and Table 11. This experiment uses spatial max as the summary function since the “ground truth” MILANNOTATIONS were created on the highest activating images calculated with max pooling.

#7 Summary

In summary we have:

• #1 Added an application/use-case of DnD
• #2 Performed an experiment in which DnD provides multiple labels to address issues of polysemanticity and added a discussion of polysemanticity to our limitations
• #3 Developed a method for finding which neurons are uninterpretable and clarified our methodology regarding uninterpretable neurons.
• #4 Discussed crowdsourced evaluation choices and the difficulty of providing a fair evaluation between methods
• #5 Performed an experiment comparing our labels to MILANNOTATIONS, and shown why this is not a useful measure of description quality
• #6 Clarified our hyperparameters and what summary function g we used

We believe that we have addressed all your concerns. Please let us know if you still have any reservations and we would be happy to address them!

Thank you for the thoughtful review and good suggestions! Below we discuss some concerns you had and how we have addressed them.

#1 Motivation/application

Thank you for the suggestion!

While we were unable to reproduce the spurious correlation experiment of MILAN because they have not released the dataset used for the experiment, we came up with a new application to showcase the usefulness of our neuron descriptions.

To showcase a potential use case for neuron descriptions (and provide another way to quantitatively compare explanation methods), we experimented with using neuron descriptions to find a good classifier for a class missing from the training set.

Our setup is as follows: we explained all neurons in layer4 of ResNet-50(ImageNet) using different methods. We then wanted to find neurons in this layer that could serve as the best classifiers for an unseen class, specifically the classes in CIFAR-10 and CIFAR-100 datasets. Note there is some overlap between these and ImageNet classes, but CIFAR classes are typically much more broad. To find a neuron to serve as a classifier, we found the neuron whose description was closest to the CIFAR class name in a text embedding space (ensemble of CLIP ViT-B/16 text encoder and MPNet text encoders used in this paper). We then measured how well that neuron(its average activation) performs as a single class classifier on the CIFAR validation dataset, measured by area under ROC curve. For cases where multiple neurons shared the closest description, we averaged the performance of all neurons with that description. Results are shown below in Table R1:

Table R1: The average classification AUC on out of distribution dataset when using neurons with similar description as a classifier. We can see Describe-and-Dissect clearly outperforms MILAN, the only other generative description method.

We can see DnD performs quite well, reaching AUROC values around 0.75, while MILAN performs much worse. We believe this may be because MILAN descriptions are very generic (likely caused by noisy dataset, see Appendix B.7), which makes it hard to find a classifier for a specific class. We think this is a good measure of explanation quality, as different methods are dissecting the same network, and even if no neurons exist that can directly detect a class, a better method should find a closer approximation.

We have included this discussion in Appendix B.8 of the revised manuscript.

#2 Address polysemanticity

Thank you for the suggestions!

We have added the following discussion on polysemanticity under Appendix A.1: Limitations:

Polysemantic neurons: Existing works Olah et al. (2020); Mu & Andreas (2020); Scherlis et al. (2023) have shown that many neurons in common neural networks are polysemantic, i.e. represent several unrelated concepts or no clear concept at all. This is a challenge when attempting to provide simple text descriptions to individual neurons, and is a limitation of our approach, but can be some- what addressed via methods such as adjusting DnD to provide multiple descriptions per neuron as we have done in the Appendix B.9. Another way to address this is by providing an interpretability score, which can differentiate good descriptions from poor ones, which we have explored in the Appendix B.10. We also note that due to the generative nature of DnD, even its single labels can often encapsulate multiple concepts by using coordinating conjunctions and lengthier descriptions. However, polysemantic neurons still remain a problem to us and other existing methods such as Bau et al. (2017) Hernandez et al. (2022) Oikarinen & Weng (2023). One promising recent direction to alleviate polysemanticity is via sparse autoencoders as explored by Bricken et al. (2023)

To address this issue of polysemanticity, we have performed an experiment in Appendix B.9 in which we modify the DnD pipeline to provide multiple neuron labels when necessary, rather than just filtering for a single concept. The scoring function now chooses the top 3 candidate labels rather than just the best one. If the similarity computed by CLIP between two of these labels exceeded a certain threshold (for our purposes we used a threshold of 0.81), only the top label would be taken and the other would be discarded. Through qualitative analysis in section Figure 18, we show that this method is capable of capturing multiple concepts of a neuron. With neuron 508 from layer 2, DnD is able to capture not only that the neuron encodes for polka dot and metal textures, but also that the textures are primarily black and white. Similarly with neuron 511 from layer 3, DnD labels the neuron as not only primarily encoding for interior elements, but also finds that these elements are black and white.

#7 Clarification on discrepancy between Table 4 v.s. Table 2

Thank you for asking this question!

For experiment 4.2, we augment the DnD pipeline by adding Network Dissection, MILAN, and CLIP-dissect labels along with step 2's candidate concepts into the set of potential concepts fed into Best Concept Selection to increase diversity in potential candidate concepts. The purpose of this experiment is to find the highest performing scoring function, which we accomplished through Table 4 and deemed “TopK Squared + Image Products” to be the highest performing scoring function. By adding more diverse potential labels into the mix, we aimed to enhance the difference in quality between each scoring function’s selected label. Even then, we found that including descriptions from other methods into candidate concepts decreased average concept quality, as the scoring function may still occasionally select a worse label from the other methods, causing its average evaluation score to not be as high.

However, we can see that “TopK Squared + Image Products” in Table 4 and Describe-and-Dissect in Table 6 follow the same general pattern in score when traversing the intermediate layers of ResNet-50: the score starts high in layer 1, drops a bit in layer 2, and then gradually increases until layer 4. As such we can see that DnD is still generally performing the same in both experiments.

Additionally, these experiments were performed with two different groups of evaluators, so the score ranges are also naturally different. To exemplify this point, we note that the evaluators for Tables 4 and 5 are the same evaluation group, but different from the one in Table 2. We can see that the evaluation group from Tables 4 and 5 generally score all labels lower than the one from Table 2, as the former group gave an average rating of 3.97 to DnD’s labels while the latter group gave an average rating of 4.15. Though these scores are more similar, this difference alongside the aforementioned augmentation of the DnD pipeline for Table 4 explain the discrepancies between the scores for Table 2 and Table 4. As such we don’t intend to compare values between tables, only within the same table. We will make this point clear in the draft.

#8 Summary

In summary we have:

• #1 Added an analysis on using a different captioning model (BLIP2)
• #2 Added several other ablation studies to understand the importance of different parts of our pipeline
• #3 Provided 3 additional experiments not reliant on human evaluation
• #4 Clarified that we plan to release our code and data with the camera-ready version
• #5 Clarified attention cropping pipeline and its effects
• #6 Discussed the importance of using synthetic images, and scoring functions
• #7 Clarified the discrepancy in scores between Table 4 and Tables 2

Please feel free to let us know if you have any additional questions or comments, and we would be happy to discuss further!

#3(b) Additional Quantitative results: MILANNOTATIONS

We also experimented with using MILANNOTATIONS as another quantitative evaluation but found the annotations to be too noisy to provide a useful signal, to the point that a trivial baseline describing all neurons as “images” outperforms every explanation method. See Appendix B.7 of the revised manuscript for additional details.

#3(c) Application: Finding Classifiers for New Concepts

Finally we provide a novel application of neuron descriptions to find classifiers for unseen tasks in Appendix B.8, showing we can find relatively good classifiers for unseen tasks. To find a neuron to serve as a classifier, we found the neuron whose description was closest to the CIFAR class name in a text embedding space. Using DnD descriptions we found classifiers with average AUROC of 0.7375 on CIFAR10 classes, and 0.7606 on CIFAR100 classes. In comparison, using MILAN descriptions the average AUROC was only 0.5906 for CIFAR10 and 0.6514 for CIFAR100, likely due to their more generic descriptions. While this is mostly independent as an application of our method it also serves as another quantitative metric showcasing our descriptions are more useful than descriptions generated by alternative methods.

[1] Oikarinen & Weng, Clip-dissect: Automatic description of neuron representations in deep vision networks, 2023

[2] Bau et al., Network dissection: Quantifying interpretability of deep visual representations, 2017

[3] Kalibhat et al., Identifying interpretable subspaces in image representations, 2023

[4] Hernandez et al., Natural language descriptions of deep visual features, 2022

#4 Code/data release

Thank you for the question, we are committed to release the source code, data and human evaluation results in the camera-ready version for reproducibility of our result.

#5 Attention Cropping Details (Sec 3.1)

Following your suggestion, we provided more details on Sec 3.1, Sec 4.3 and Fig 4 below and in the revised draft.

Thanks for pointing this out, following your suggestion we have included a detailed description of attention cropping in Appendix B.1 of the revised draft, including Figure 10 to visualize the procedure.

We also follow your suggestion to do an ablation study on the effect of attention cropping to DnD performance in Appendix B.2. We qualitatively observe this change in Figure 11, noticing that the primary concepts for layer 3 neuron 475 and layer 2 neuron 467 are brought out more with attention cropping.

#6 Additional details on Concept Selection (Sec 4.3, Fig. 4)

To expand more on section 4.3, we elaborate on the importance of Best Concept Selection. Due to GPT-3.5-Turbo’s generative and variable nature, it can produce different outputs each time when given the same input. This means that if you simply generate one concept label from it for a neuron, it can give you different labels each time; obviously some of these labels will be better than others. To account for this fact and maximize performance, we employ Best Concept Selection. Generating images from the multiple labels GPT can produce for one neuron, we calculate which label activates the neuron the most and is thus most likely to represent what the neuron encodes. Generating new images helps us remove spurious concepts that can be present in the highly activating images of a neuron due to correlation, but do not by themselves cause the neuron to fire. An example of this is in Figure 4b, where DnD without Best Concept Selection gives “Monochromatic and patterned designs.” We can see that monochromatic is a spurious concept not required for high activation of neuron 927, and as such DnD with Best Concept Selection accounts for this and labels the neuron “abstract swirls.” This is far closer to the true label of the neuron and signifies why Best Concept Selection is essential for our pipeline to give its best performance.

Thank you for the positive feedback and comments! We would like to address your questions and concerns below.

#1 Impact of captioning model

Thank you for the suggestion! Following your suggestions, we have conducted additional experiments using BLIP2 as a captioning model in our pipeline in place of BLIP.

We found that BLIP and BLIP2 generally give similar results as shown in below Table R1, where we compare the cosine similarity scores between the neuron descriptions generated by BLIP and BLIP2 in a ResNet 50 model. For each layer in the model, we compute the mean CLIP cosine similarity between BLIP and BLIP-2 labels for 50 randomly chosen neurons. Similar conceptual ideas between both models are reflected in the high similarity scores.

It is also interesting that we found there are some cases where BLIP2 is too specific in its captions, causing it to miss the bigger concept and underperform compared to BLIP as seen in Figure 13 in Appendix B.3 in the revised draft. We can see that with neuron 248 in layer 1, the concept should clearly be something along the lines of “black” or “dark.” DnD w/ BLIP is able to capture this with its label “low-light scenes” while DnD w/ BLIP-2 gives a worse label of “monochrome timepieces”. We can see that BLIP-2’s captions are trying too hard to see things in these simple images, detecting the color white and objects where there are none.

Table R1: Mean Cosine Similarity Between BLIP and BLIP-2 Labels.

#2 Additional Ablation Studies

In addition to the effects of Image Captioning model studied above, we have conducted several additional ablation studies to study the importance of different parts in our pipeline, summarized below:

• Ablation: Attention Cropping. We added section B.2 in the Appendix where we evaluate the effect of removing attention crops from our pipeline.
• Ablation: Fixed Concept Set. In Appendix B.4 we evaluate how well our network performs if we use CLIP with a fixed concept set instead of a generative image captioning model, showcasing generative models that lead to much improved performance.
• Ablation: Effects of GPT Summarization. In Appendix B.5 we evaluated the performance of our method if we remove the GPT summarization step, showcasing it leads to noticeably worse concepts.

#3 Only human evaluation

Thank you for the suggestion. While we believe that in the end best evaluations of interpretability requires human in the loop, we have provided 3 additional quantitative experiments that are less human dependent.

#3(a) Additional Quantitative results: Final layer.

To provide additional quantitative results, we follow [1, e.g. sec 4.2] to evaluate the description accuracy of the final layer neuron. In [1], the authors proposed the idea as an alternative analysis to automatically quantify the quality of neuron descriptions because the “ground-truth” description is available for the final layers (i.e. the class name).

Here, we focus on comparing our method DnD to MILAN [4], as MILAN is the only other contemporary work that provides generative descriptions (see our table 1). Other works e.g. [1, 2] use fixed concept sets which give them an advantage when calculating the similarity between labels and ground truths. This is because the “ground truth” class labels or similar can be incorporated into the concept set. Therefore, it is expected that this evaluation method will be less favorable for generative methods like DnD and MILAN, and would be more fair to compare DnD and MILAN only.

We report our result of a ResNet-50 model trained on ImageNet below in Table R2 and also in Appendix Sec B.6 Table 10 of the revised draft. We use two embeddings (CLIP and MPNet) to determine similarity between the generated neuron descriptions and the ground truth class label. It can be seen that our DnD labels are closer to the ground truths than MILAN’s by a significant margin, indicating the effectiveness of our proposed method.

Table R2: Cosine similarity between predicted labels and ResNet-50’s “ground truths.” We can see that on average, DnD provides labels that are more similar to the ground truths than MILAN’s labels.

Thank you for the feedback! Please see our response below to address your concerns.

#1 Quantitative results

#1(a) Only has crowdsourced experiments and lacking quantitative results

Thank you for the suggestion! We believe that there might be some misunderstanding here. The human study results are usually considered as “quantitative” analysis in the prior work in this field, please see [1, table 5][2, table 3][3, table 1, 2, A.2]. For example, human evaluators give “quantitative” scores (e.g. ranging from strongly disagree to strongly agree as score 1 to 5) to evaluate the quality of neuron descriptions. Our original evaluation methods followed prior works in this field [1, 2, 3] by utilizing human evaluation for results on the intermediate layers of a network, and we reported the evaluation scores in Table 2, 3, 6, 7. Our results show that we are on average rated 0.94 better than the state-of-the-art models (which is a 37.56% increase), supporting the effectiveness of our proposed approach.

#1(b) Additional Quantitative results: Final layer.

To provide additional quantitative results, we follow [1, e.g. sec 4.2] to evaluate the description accuracy of the final layer neuron. In [1], the authors proposed the idea as an alternative analysis to automatically quantify the quality of neuron descriptions because the “ground-truth” description is available for the final layers (i.e. the class name).

Here, we focus on comparing our method DnD to MILAN [4], as MILAN is the only other contemporary work that provides generative descriptions (see our table 1). Other works e.g. [1, 2] use fixed concept sets which give them an advantage when calculating the similarity between labels and ground truths. This is because the “ground truth” class labels or similar can be incorporated into the concept set. Therefore, it is expected that this evaluation method will be less favorable for generative methods like DnD and MILAN, and would be more fair to compare DnD and MILAN only.

We report our result of a ResNet-50 model trained on ImageNet below in Table R1 and also in Appendix Sec B.6 Table 10 of the revised draft. We use two embeddings (CLIP and MPNet) to determine similarity between the generated neuron descriptions and the ground truth class label. It can be seen that our DnD labels are closer to the ground truths than MILAN’s by a significant margin, indicating the effectiveness of our proposed method.

Table R1: Cosine similarity between predicted labels and ResNet-50’s “ground truths.” We can see that on average, DnD provides labels that are more similar to the ground truths than MILAN’s labels.

#1(c) Additional Quantitative results: MILANNOTATIONS

We also experimented with using MILANNOTATIONS as another quantitative evaluation but found the annotations to be too noisy to provide a useful signal, to the point that a trivial baseline describing all neurons as “images” outperforms every explanation method. See Appendix B.7 of the revised manuscript for additional details.

#1(d) Application: Finding Classifiers for New Concepts

Finally we provide a novel application of neuron descriptions to find classifiers for unseen tasks in Appendix B.8, showing we can find relatively good classifiers for unseen tasks. To find a neuron to serve as a classifier, we found the neuron whose description was closest to the CIFAR class name in a text embedding space. Using DnD descriptions we found classifiers with average AUROC of 0.7375 on CIFAR10 classes, and 0.7606 on CIFAR100 classes. In comparison, using MILAN descriptions the average AUROC was only 0.5906 for CIFAR10 and 0.6514 for CIFAR100, likely due to their more generic descriptions. While this is mostly independent as an application of our method it also serves as another quantitative metric showcasing our descriptions are more useful than descriptions generated by alternative methods.

[1] Oikarinen & Weng, Clip-dissect: Automatic description of neuron representations in deep vision networks, 2023

[2] Bau et al., Network dissection: Quantifying interpretability of deep visual representations, 2017

[3] Kalibhat et al., Identifying interpretable subspaces in image representations, 2023

[4] Hernandez et al., Natural language descriptions of deep visual features, 2022