Transforming Visual Classifiers for Zero-Shot Text-Based Interpretability

Thank you very much for your time and effort in reviewing our paper, and for the thoughtful review and strong accept decision. We are delighted that you found our manuscript interesting and appreciated the importance and unique aspects of our work, such as the innovative, novel, and clever solution, the rigorous evaluations, clear writings, and detailed analysis. Your acknowledgment of the importance and strengths of our work is truly encouraging, and we are grateful for your supportive comments.

We thank the reviewer for his time in reviewing our paper. We clarify your concerns below.

Existing methods (e.g., Text-to-Concept [a]) have the same or lower number of parameters.

As mentioned in L70-L84 in the related work, while the parameters may be the same or less, Text-to-Concept is not faithful (largely biased towards CLIP), and not label-free (requires supervision from CLIP features as the ground-truth). Efficiency is not the sole objective of our work.

the idea of mapping image features to text features with MLP or linear layers has been reported in several existing studies [a,h]

Work [a] and its drawbacks are discussed clearly in the related work. We also mention how our method is different and how it tackles the problems of [a]. We refer you to L71-L84 (right). [h] is a work that uses image annotated captions as the training objective and therefore is very similar to DeVIL and the other mentioned works in Section 2 and suffers from the same limitations. We also mention how our method is different and how it tackles their problems. We refer you to L106 (left)-L61. The issues that you have raised are inherent in the literature and are thoroughly discussed in the related work. Our method is explicitly made to tackle the problems that these methods have.

The paper points out in L080 (right) that existing CLIP-based methods are biased against CLIP but lack discussion and rationale for this negative impact and the advantages of the proposed method.

We discuss this thoroughly in L21-29 (right).

Table 1 shows that the use of textual features degrades the performance on average, indicating that the transformation is not perfect and that some changes have occurred.

The performance drop is negligible (0.2 points drop on average across all models). In our view, this negligible drop does not degrade performance. Furthermore, the negligible loss is compensated by unlocking text-based interpretations to vision models, all in a zero-shot manner.

Since the image features and text features are not learned in a one-to-one correspondence, the paper cannot guarantee that the image features projected by MLP correspond semantically exactly to text features.

We explicitly align the distributions rather than the features. The loss we use guarantees that the distributions are aligned. Using a one-to-one feature-to-feature mapping as the reviewer suggests will ignore the relation of other classes to the image, and will ignore the classifier’s reasoning process. As a very simple example, consider an image of a dog, with a chair present in the background. The softmax distribution for this sample would assign a high probability to the dog class and a moderately high, yet non-negligible probability to the chair class. Therefore, using distribution alignment loss considers the presence of both these classes. It cares about the distribution of the image over all classes, which is also how the classifier reasons and makes decisions. That is why, using distribution alignment loss rather than feature alignment loss is a faithful, effective and valid way for our approach.

It has not been verified that the method generalizes to a variety of datasets and domains.

We did verify the zero-shot generalization of our MLPs trained solely on ImageNet class names, to the COCO dataset. We mention this in detail in L360-L366, where we describe how the two datasets differ in distribution. Furthermore, we showed zero-shot classification generalization experiments to other datasets (Places365) in the supplementary material (Table 5). We were also transparent and acknowledged in our manuscript that the classification generalization of our method to more fine-grained datasets is a limitation of our method.

Examining the intervention on the concept and the rate of recovery of the true concept would be helpful in evaluating the explanations. The claims regarding the quality and faithfulness of the explanations are not verified.

We thank the reviewer for the valuable comment. We agree that evaluating the intervention on concepts could offer additional insights into the interpretability and reliability of the explanations. However, evaluating with interventions on ImageNet poses challenges due to the lack of intervention annotations (the expected biases in ImageNet classes). We therefore chose the core, standard way of evaluating CBMs which is task accuracy. For decoding of visual features, the main issue is, that we don't know what the model uses for reasoning (no ground truth). There is currently a gap in the literature regarding a proper benchmark for such models. We have discussed the limitation of this evaluation in L355 (left) - L359 in the main manuscript. As there is no ground-truth of what the model uses for reasoning, using annotated captions is the best approximation we can have.

We thank you for your time and effort in reviewing our paper, and for the valuable feedback. We are glad that you liked our paper and thank you for all the positive points you reported for our work. Below, we address your concerns.

How does the faithfulness vary?

As a function of different text encoders, we showed in Table 6 of the appendix that the performance is not affected by different text encoders. The difference is at max 0.07 points for any text encoder we use. We also justified the reason for this in L703.

We also test the impact of the MLP on the faithfulness through the following ablations:

• Mean Ablation: For an image, we replace the input features to the MLP with a constant mean of the features calculated across the full ImageNet validation set.
• Random Features: For an image, we replace the input features to the MLP with random values sampled from a normal distribution with a mean and standard deviation equal to that of the features calculated across the full ImageNet validation set
• Random Weight Ablation: We randomize the weights of the MLP projection
• Shuffled Ablation: For an image, we replace the input features to the MLP with input features of another random image in the validation dataset

For all ablations, we calculate the ImageNet validation accuracy. We expect the accuracy percentage to drop in all ablations. This is clearly shown in the Table below. The accuracy drops to near zero in the ablations.

Faithfulness of the CBM transformation depends on the concepts

Our Zero-Shot CBM formulation is general and flexibly allows to use any concept set at test time directly (on-the-fly), including future improvements on reference concept sets. Indeed, if a (very) poor concept set is used, the approach is not able to faithfully reconstruct the original distribution. We assume that for the CBM the provided concept set is sufficiently expressive, which is the case even for most automatically generated concept sets such as from a LLM or the top 20k most common words. In this work, we chose the 20K most common words in English as our concept test for the sole reason of establishing a fair comparison with other works. We will address the case of poor choices for concept sets in our extended limitation discussion.

I find the qualitative examples a little bit left to the human interpretation.

While we acknowledge that our qualitative example in Figure 5 may appear open to human interpretation and subject to confounding factors (e.g., background cues like greenery), this observation directly reflects the inherent ambiguity of the concept set. As mentioned in the previous concern, our zero-shot CBM formulation (Eq. 2) is independent of the concept set used (we do not train on any concept set), allowing for diverse input concept sets that can be used at test time directly (on-the-fly) that can yield more precise, attribute-specific interpretations; we refer you to C1 of Reviewer 4kqE for further clarification and qualitative examples on different concept sets attached in an anonymous link.

Table 3 simply shows to me that ConvNext together with LLM, and the authors construct, generate captions that correlate to the ground truth captions better than the CLIP-based models. However, it does not tell me anything regarding whether a human-user would understand how the visual model itself "reasons" about an image to achieve a prediction.

The table indeed shows that ConvNext (and some other models ) with our adaptation performs better than the widely-used CLIP-based models. We agree that a user-centric benchmark would be very valuable for evaluation, which however does not exist yet. The main issue is, that we don't know what the model uses for reasoning (no ground truth), so even if a human user study would be conducted, the user could only judge whether the generated text is sensible with respect to the input and corresponding predicted output. This hence does not evaluate whether the textually-decoded visual features correspond to the network's inner reasoning. There is currently a gap in the literature regarding a proper benchmark for such models. We have a corresponding discussion of the limitation of this evaluation in L355 (left) - L359 in the main manuscript. As there is no ground-truth of what the model uses for reasoning, using annotated captions is the best approximation we can have.

We thank you for your time and effort in reviewing our paper and for the valuable feedback. We thank you also for acknowledging the strengths of our paper. In what follows, we address your concerns and remarks.

[C1] In the examples, the “concepts” often appear more like class labels rather than granular attributes, raising the question of how meaningful these concepts are in practice.... the paper’s examples sometimes resemble re-labeled classes rather than a set of attributes.….How does one ensure that the discovered “concepts” are not merely providing similar or re-labeled classes?

This issue is due to the concept set we used, it is not an issue of our formulation. Unlike existing CBM works that train on a specific concept set, our zero-shot CBMs allow us to use any concept set at test time directly (on the fly); this is mentioned in L197 (left). Our method is completely independent of the concept set, and we do not train our core method (Section 3) nor our CBMs (Section 4.1) on a concept set (as mentioned in L127 (right) and detailed in Section F of the appendix). The chosen concept set is merely used at test time in the applications, and our method is flexible to any concept set as it merely involves encoding the concept set with the text encoder. In the example in Figure 5, we used the 20K most common words in English as our concept set. The 20K concept set includes both class-like names (e.g., duck, bird, pigeon) as well as granular attributes, and the reason for using it in our work as the main concept set is to establish a fair comparison with other works that also used it [R1].

In Figure 5, as mentioned in the Figure caption, while the last three examples use the 20K concept set, the first example uses LF-ImageNet—an attribute-specific concept set extracted from ImageNet classes using a large language model. We include examples from different sets precisely to highlight the flexibility of our approach to different concept sets. When using the 20K-word set, class-like names often appear closer to the image in the embedding space than fine-grained attributes, which explains why they dominate the top detected concepts. In Figure 6, we again use the LF-ImageNet set, which focuses on fine-grained attributes (e.g., powerful jaws, harpoon, gills, large triangular fins, etc.).

To address your concern even better, we show in an anonymous link the top-detected concepts for the second images (goose) shown in Figure 5, using exactly the same setting but with various different concept sets. We used the following concept sets:

• Creating our own concept set specific to attributes from ImageNet by prompting GPT4o-mini with the prompt: “Which physical features and attributes make a {class name} different from others of the same type?”, where {class name} is a class name from ImageNet.
• The concept set of LF-ImageNet
• Creating our own concept set specific to attributes from ImageNet by prompting GPT4o and Llama LLMs with the prompt: “What are useful visual features for distinguishing a {class name} in a photo?”, where {class name} is a class name from ImageNet.

The anonymous link is here: https://github.com/user-attachments/assets/cd8e0ba0-062b-4d44-a541-2fda87f870a2

As it can be shown, when using a concept set of fine-grained attributes, this issue no longer exists. Again, we merely used the 20K as our main concept set in order to establish a fair comparison with other works.

[C2] How does the paper ensure that these text-based “concepts” fulfill the CBM philosophy of providing a transparent, concept-level explanation for the final class decision?

Indeed, the concepts should provide a transparent explanation for the final classification. Our designed approach follows this principle, where the classification layer is a simple linear layer from the concepts to the output (classes):

$pred = w_1 c_1 + w_2 c_2 + w_3 c_3 + \dots + w_Z c_Z$, where $c$ is the concept, $w$ is the weight connecting the concept to the prediction class, and $Z$ is the total number of concepts. As this is a linear layer, we can interpret the classification decision by looking at the concepts with the highest weights connecting to the prediction. However, by merely looking at the weights, we only get a global explanation (since the weights are constant and do not change as a function of the input). That is why, in order to make the explanation local to the prediction, we have to multiply those weights with their concept activation scores obtained by feeding the image (as mentioned in L320-left). Therefore, we dissect the prediction into its input elements: $pred = [w_1 * c_1 , w_2 * c_2 , w_3 * c_3 , \dots , w_Z * c_Z]$ and then obtain the top-k elements. The numbers shown next to the concepts in Figure 5 are calculated by multiplying the weight of the concept by its concept activation score ($w_x * c_x$).

[R1] DN-CBM: Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery, ECCV 2024