CoSy: Evaluating Textual Explanations of Neurons

We appreciate Reviewer 4 (R4) for their time taken and their great attention to detail shown in the review. We are honored by your positive feedback, and thankful that you consider our work as important.

[A1] Comparison to other evaluation scores: We thank the reviewer for asking these important questions. We extend the discussion of comparable methods in the final manuscript. While the results presented in the primary publication are comparable, with BERTScores around 0.5 for MILAN across architectures and good performance of CLIP-Dissect, we argue that due to the lack of standardized procedures across publications, we cannot rule out biases toward specific methods. Moreover, human-based evaluation methods often also include further biases. For example, many evaluations supply highly activating images [1], to make the neuron annotation by humans possible. These images may not accurately represent the neuron's overall behavior by only referencing the maximum tail of the distribution.

[A2] Broader range of explained models: We acknowledge the reviewer's suggestion to analyze a broader range of models. In response, we have now included ResNet50 pretrained on ImageNet in our analysis as suggested (see Table 2 in PDF). We can observe that INVERT achieves the highest AUC score, while CLIP-Dissect attains the highest MAD score, consistent with our previous benchmarking results on ResNet50 pretrained on Places365. Additionally, we plan to incorporate additional models, such as the suggested DINO, in our benchmarking table for a broader evaluation.

[A3] Additional textual explanation methods: We highly value the reviewer's comment and suggestion to include more explanation methods. Due to the limited rebuttal period, we were not able to complete the additional computations in time. Nonetheless, we aim to address this request going forward.

[A4] Clarification on AUC: Thank you for your remark on the AUC. We apologize for not describing it clearly enough in our initial submission. To clarify, we have included a plot in Figure 3 in the PDF that illustrates the AUC scores for a specific neuron (Neuron 358, ResNet18, Layer 4) across four different methods. The AUC score is the Area Under the Receiver Operating Characteristic (ROC) Curve. Figure 3 in PDF illustrates these ROC curves. In this context, the target is our binary label (0 for the control dataset and 1 for concept images), and the ''predictions'' are the neuron activations for each image. The AUC demonstrates how well the activations of the concept images discriminate against the control image activations. In our benchmarking table, we present the mean AUC scores across all selected neurons, providing a more comprehensive evaluation. We hope this explanation and the included visualizations help to clarify our use of AUC.

[A5] Computational Cost: We strongly agree with R4 that the computational cost can be a limiting factor. Thus, we tried to incorporate other open-source text-to-image models, which claim to improve upon computation time but found no significant improvements. In case of limited resources, the only option therefore is the reduction of inference steps to improve run-time. We note, however, that a smaller number of inference steps reduces image quality, which might affect the algorithmic performance.

[A6] CLIP Bias: We highly appreciate this question. The explanations from the CLIP-Dissect method are selected from a predefined list of concepts, therefore the potential bias towards CLIP-Dissect is minimal.

[A7] Consistent Terminology: We thank the reviewer for pointing out the inconsistency and will ensure consistent terminology in the final manuscript.

[1] Kalibhat, Neha, et al. ''Identifying interpretable subspaces in image representations.'' ICML, 2023.

We thank Reviewer 3 for the constructive and insightful remarks and appreciate the positive feedback regarding the presentation quality. Below, we address each of their points in detail.

[A1] Prompt Bias: We appreciate and strongly agree with the reviewer's insight regarding the dataset dependency of the prompt. We have addressed this in our manuscript by comparing the ImageNet dataset (object-focused) with the Places365 (scene-focused), leading us to use a more general prompt for Places365 models. For further investigations, we expanded our image similarity analysis from 10 to 50 classes, as suggested in [Q2]. We also compared results across both datasets on different similarity measures, as suggested in [W3]. For a detailed analysis, please refer to Figure 2 and Table 1 in our PDF. Our results show a significant difference based on prompt selection and dataset: close-ups work well for object-focused datasets, while more general prompts like ''photo of'' are better for scene datasets.

[A2] Prior Evaluation Methods:

Benchmarking existing metrics is challenging due to several reasons:

• Challenges in Human Evaluations: Human evaluations mostly fail to give a holistic description of a neuron, since they rely on describing highly activating images only [1]. Furthermore, human-based evaluations suffer from a lack of standardized experimental setups, e.g. varying protocols, tasks, and participant groups, introducing biases, inconsistencies, and are overall not scalable.

• Limited Scope of Ground-Truth Labels: Label-based evaluations are restricted to output neurons with predefined classes [2, 3]. Intermediate layers and open-vocabulary explanations are not covered, which limits the applicability of these metrics for a comprehensive evaluation across all neurons in a model. Furthermore, although output neurons are trained to detect specific concepts, their performance may not exactly match their intended function.

We will add these points to the paper. Thus, clarifying that our approach unifies the evaluation procedure for open-vocabulary explanations and future textual explanation methods for any neuron.

[A3] Limitations of Cosine Similarity as image similarity measure: We acknowledge the reviewer's concern regarding the limitations of using cosine similarity (CS). In response, we have incorporated two additional distance measures: Learned Perceptual Image Patch Similarity (LPIPS), which calculates perceptual similarity between two images and aligns well with human perception, using deep embeddings from a VGG model. Additionally, we included Euclidean Distance (ED) to capture the absolute differences in pixel values. These measures provide further context, allowing a broader assessment of visual similarity beyond CS. For detailed results and comparisons, please refer to our PDF, particularly Table 1, which analyzes image similarity with more concepts, as suggested in [Q2]. Importantly, our overall conclusions remain consistent even with these new metrics.

[A4] Dependency of a text-to-image generative model in evaluation framework: For our proposed approach, txt2img models are crucial, because most explanation methods rely on open-vocabulary descriptions for which images are not always available.

Theoretically, a way to mitigate reliance on txt2img models is to manually collect data points corresponding to the neuron label (i.e., images that, in the CoSy approach, are generated by the txt2img model), which however in practice would not be scalable. Furthermore, we ensure that the effects of the chosen txt2img model are limited by performing the sanity checks (see Section 4).

[Q1] Correlation of AUC and MAD: CoSy measures the difference between activation distributions in the control dataset and the images corresponding to the given explanations. The AUC, a nonparametric test, assesses whether a neuron ranks data points corresponding to the explanation systematically higher than random images. In contrast, MAD is a parametric test similar to the Student's t-test, using exact activations and being more susceptible to outliers. While practically high MAD scores often accompany high AUC scores, these metrics can sometimes disagree, thus complementing each other and providing a broader evaluation of the explanation's quality.

[Q2] Small Concept Size: We agree and expanded our experiment to include 50 ImageNet concepts, as shown in Figure 2 of the PDF, and applied the same methodology to 50 scene concepts from the Places365 dataset. The results for ImageNet remain consistent, with increased standard deviation but maintaining the general trend.

[Q3] Aggregation operation: The aggregation operation is only performed when the output of a specific neuron is multidimensional (non-scalar) and follows Equation 2 in the paper, corresponding to the average pooling operation. Negative activations do not impact the AUC or MAD metrics because these metrics assess distribution differences.

[Q4] Large errorbars in Figure 4: We acknowledge the concern about the large errors in Figure 4. These errors stem from the evaluation methods' performance. Contrary to the reviewer's claim, Figure 4 does not measure cosine similarity (CS); it measures AUC and MAD. The significant errors suggest that no single method is definitively superior, but the results still offer valuable insights for future research and improvements.

[1] Kalibhat, Neha, et al. ''Identifying interpretable subspaces in image representations.'' ICML, 2023.

[2] Bykov, Kirill, et al. ''Labeling Neural Representations with Inverse Recognition.'' NeurIPS, 2024.

[3] Oikarinen, Tuomas, et al. ''CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks.'' ICML, 2023.

We thank Reviewer 2 (R2) for the detailed comments and in-depth remarks. We are pleased that our work was found to be a valuable contribution. Below, we address all individual comments.

[A1] Text-to-image models: We agree that the effectiveness of CoSy depends up on the performance of the generative model, which indeed is a limitation, which we point out in the limitations section of the original manuscript. We recommend exercising caution when selecting specific text-to-image models. It is important to understand the performance characteristics and limitations of each model during the evaluation process.

[A2] Bias towards INVERT: We acknowledge the concern regarding the potential bias of the INVERT method toward achieving higher AUC scores in the CoSy evaluation. To give a fair and balanced assessment of explanation quality, we, therefore, introduced the MAD metric as an additional evaluation metric. Here, it is also important to note that CLIP-Dissect, another method included in our study, is biased toward CLIP models (also used in Stable Diffusion). By employing multiple evaluation metrics and methods, we strive to offer a comprehensive and unbiased evaluation of performance. For more details on the correlation of AUC and MAD, please refer to our answer [Q1] to R4.

[A3] Lower-level Concepts: We also thank R2 for putting further emphasis on the important issue of lower-level concepts. We share the reviewer's interest and plan to extend existing discussions in Section 5.2. The performance of generative models might be a fundamental problem for abstract concepts, as discussed in our limitations. To assess this concern and provide a basis for further discussion, we have also added an additional example of lower-layer concepts in the same style as Figure 5 (see attached PDF, Figure 1). We agree that the generation model faces an increasingly harder task to generate images corresponding to more abstract concepts, which in part contributes to the larger error bars for lower layers. However, we argue that the limitations of text-to-image are still not the fundamental issue even for lower layers as our example shows that explanation methods often fail to provide abstract concepts and present widely different concepts, such that we can still distinguish individual performances. We also performed further experiments regarding low-level concepts and discussed this topic in our response [A3] to R1.

We appreciate Reviewer 1 (R1) for their time taken and their great attention to detail shown in the review. We are honored by the positive feedback, that our work was found to be highly relevant and impactful. In the following, we address the reviewer's comments in detail.

[A1] Meta-Evaluation:

• Choice of wording: We strongly agree with R1 in their assessment of the term Meta-Evaluation. We apologize that this terminology was misleading. Due to the lack of ground-truth, we should not use the term Meta-Evaluation and therefore replaced it with Sanity Checks.

• Comparison to prior evaluation: Generally, prior evaluation methods can be divided into two major groups: evaluations based on human studies and evaluations based on assumed ''ground-truth'' labels. Due to the variety of evaluation approaches, a comprehensive comparison in terms of a meta-evaluation is not established. However, we discuss a qualitative comparison to previous methods in response to R4 [A1].

• Concept-bottleneck for lower-level representations: While we agree with the reviewer's concern regarding lower layer representations and share the interest in concept-bottleneck (CB) based meta-evaluation results, we initially refrained from such experiments. We believe the ground-truth nature of CB concepts and last-layer labels to be often similar. Specifically, mostly during training, intermediate concepts are learned in a supervised manner, often using a similar loss function as the output of the model [1]. Therefore we consider that there is little difference between CB neurons and output neurons.

[A2] Similarity of generated concepts to control data: We thank R1 for the detailed questions. We acknowledge that the control dataset may indeed contain images visually similar to those in the generated explanations. However, this is not a problem, since the control dataset primarily serves as a collection of random images that represent a diverse array of concepts. While the control dataset might include some images of “white objects”, the majority of images do not include “white objects”. Therefore, for an accurate explanation, the generated images all significantly activate certain neurons, whereas most control images will not. This distinction will be correctly reflected in the AUC and MAD metrics.

Regarding the performance of our methods with low-level neurons, we attribute the low AUC/MAD to the fact that these methods typically output semantically high-level concepts (see Figure 1 in the PDF). This may also be related to the inherent difficulty in describing low-level abstractions in natural language due to their complexity. We will include a broader discussion of this point in the manuscript.

[A3] Diversity of generated images or concept broadness: We thank reviewer 1 for bringing up the important consideration of the diversity in generated images per concept and the problem of semantically similar concepts. We will include this topic in more detail in the final manuscript. To this end, we will reference our results in Section 5, and Figures 1 (red objects and pomegranate) and 5 (last line), which show that even different concepts with high perceivable image similarity result in separable evaluation results. Moreover, we will discuss that, semantically similar concepts can warrant similar evaluation scores given similar meanings. Thus, differentiation between justified and failed evaluation for semantically similar concepts would require a ground-truth label and even then can be hard to argue. Regarding the diversity of generated images for a single concept, Appendix A.8 and Figure 9 show that the diversity of the generated images does not depend on the chosen prompt and concept, but rather on the temperature value (value of entropy) in the diffusion model. Thus, we see a high similarity of images generated for the same concept.

[Q1] Feature Directions: We appreciate the reviewer’s detailed question and thank them. Our evaluation procedure can be applied to extracted concepts or any scalar function within the model, such as linear or nonlinear combinations of neurons, for example, CRAFT. The choice for evaluating the neurons (i.e. canonical basis) and not concepts lies in the fact that the respective publications of most methods explain neurons specifically, with even more limited approaches such as FALCON [2] that can only be applied to certain ''explainable'' neurons. Thus we considered only the canonical basis, to establish a fair comparison. Nonetheless, we are very much interested in extending our work in this direction and plan to address latent feature directions in future work.

[Q2] Related Work: We thank the reviewer for pointing out these important works, which we will add to the related work section.

[Q3] Fitting a logistic classifier to the neuron activations: We appreciate the reviewer’s suggestion and the opportunity to address this point. However, we kindly request further clarification regarding the proposal. Specifically, could the reviewer provide more details on what is meant by applying a logistic classifier prior to computing the AUC?

[1] Koh, Pang Wei, et al. ''Concept bottleneck models.'' ICML, (2020).

[2] Kalibhat, Neha, et al. ''Identifying interpretable subspaces in image representations.'' ICML, 2023.