Interpreting the Second-Order Effects of Neurons in CLIP

We thank the reviewer for their question.

“What is the relation between exploring what examples the neurons activate on vs. exploring their contribution in the output space, what are the advantages/limitations of one over the other, does one show complementary aspects that the other cannot observe?”

There are non-trivial relations between the highly activated images of a neuron (its “input space”) and its output space. Here, for example, we rely on the assumption that if a neuron’s second effect contributes toward some direction (e.g. the “dog” text representation), it probably means that the neuron is activated on images that have the concept in them. While it is mostly the case and allows us to attack the model, we found some instances where a neuron has relatively high activation patterns on images of a concept, but since the neuron is followed by a GeLU (which can produce negative values for positive inputs), its second-order contribution direction is negative. Nevertheless, in most cases, there is a clear positive correspondence between the concepts in highly activated images and the neurons’ output space. Moreover, in most cases, there is a clear correspondence between the activation patterns and the output space (e.g., a dog segment will be highly activated for a neuron that has a high contribution towards the “dog” text representation). This property allows us to mine for the neurons with a high contribution toward a text representation and use their activation maps for zero-shot segmentation (see Figure 7 and Table 4).

We thank the reviewer for the valuable comments and questions.

“Sparse coding to find textual descriptions of neurons may be very computationally expensive”

Empirically, we didn’t find the sparse coding technique being too expensive. We used an off-the-shelf sklearn library (CPU-based solution) and did not try to optimize this part. Applying this sparse coding technique to all the neurons we analyzed (one layer in the model) with a dictionary of ~30000 entries took a few hours (please see Appendix A.8 for more details).

“Not considering nonlinearities in second-order effects (Eqn 5)”

Indeed, for simplicity, we do not mention the nonlinearities (that come from the layer normalizations) in the main text. Nevertheless, in all of our experiments, we include this part in the computation of second-order effects. We included the extended derivations that we use to compute the second-order effects with layer normalization in Appendix A.6.

“Are there any gradient-based approaches to finding textual descriptions of each neuron's second-order effect?”

To the best of our knowledge, there are no gradient-based approaches for finding textual descriptions for neurons, mostly due to the difficulty of back-propagating into a discrete space.

“Were any additional text-to-image models evaluated other than DeepFloyd IF, if so was there similar success to images generated with DeepFloyd IF?”

We specifically used DeepFloyd IF as our model, because it does not use CLIP model in its conditioning. Nevertheless, the same approach can be applied to other generative models as well. Moreover, we suspect that with better models, that follow the prompt better, our adversarial results will improve, and we plan to analyze it in the future.

We thank the reviewer for the valuable questions.

“What happens if you apply the same method to the text encoder in CLIP?”

Although the same technique can be applied to the text encoder, we did not do that. More specifically, the text encoder is also a transformer, so we can compute the second-order effects similarly, and then apply sparse decomposition into a dictionary of text/image representations. We focused here on the vision encoder for exploring the polysemantic behavior of vision neurons with text and to demonstrate that the second order is useful for finding adversarial images. We plan to explore the text encoder in future work.

“Have the authors tried it on other variants of CLIP?”

We evaluated our model on different model sizes and found similar trends (please see Figures 9 and 12 in the supplementary material).

“Do the findings of this paper apply to other domains like medical imaging (MedCLIP)?”

We did not apply it to medical imaging but we think that the same findings can be applied there as well, as the architecture of the model is similar.

We thank the reviewer for the valuable comments and questions.

“Can the authors provide a discussion on TextSPAN, highlighting the differences in the decomposition and analysis of the direct effects of the neurons?”

[1] explores the direct effect of different components in CLIP. As shown in [1], the direct effects of the MLPs are very low – mean-ablating the direct effects of all the neurons in all the layers results in a negligible drop in performance (please see Table 1 in [1]). Moreover, the main observation is that most of the direct effect comes from the late attention layers. Therefore, in our analysis, we look at the information flow from neurons through such attention layers, directly to the output. A different way of looking at it is that in our analysis we decompose the contributions of attention layers into further flows from previous layers through them, and focus on the flows from individual neurons.

“The focus on a single dataset, i.e., ImageNet, is a bit restrictive in terms of analyzing the behavior of the proposed approach”

Indeed, we mostly focus on ImageNet classification as our downstream evaluation metric, to analyze how much our ablations change the final representations, similarly to [1]. Nevertheless, as shown in Figure 10, performing similar analyses on other datasets (e.g., ImageNet-R) shows similar trends.

“What happens when trying to generate adversarial examples in a more complicated dataset?”

Our approach for adversarial attacks does not rely on the images of the datasets. We only use the class names from the datasets to showcase that we can generate an image that will be classified as one class while having objects from the other class. We specifically used classes that look very different (random CIFAR class pairs - e.g., horse and automobile) to produce those examples. We provide additional examples for other classes in Figure 15.

“Did the authors reproduce the results for all the methods in Table 4?”

Yes, we reproduced all the baselines and re-evaluated the different approaches. The difference in numbers from TextSpan is due to a different choice of a model – we focus on the model that was pre-trained by OpenAI on their data, while TextSpan focuses on a model that was trained by OpenCLIP on LAION.

“Qualitatively, what are the differences when choosing a different sparse set size m?”

We found that different m’s provide relatively similar results when looking at the top coefficients. Nevertheless, larger m’s allow better reconstruction of the second-order of a neuron. That suggests that neurons are polysemantic and correspond to multiple texts, but many of them have only a few significant and coherent text directions for which they are responsible.

“What is the motivation behind the consideration of a binary classification task instead of a classical setting? What classifier is used and how is it trained?”

We aim to produce an adversarial attack on CLIP. Therefore, we use CLIP as the zero-shot classifier, by comparing the output image representation to the text representations of the two class descriptions (e.g., “An image of a dog”/”An image of a cat”).

Typically, adversarial attacks are images that cause a classifier to label them as one class while they are perceived by humans as another class. This is the reason for choosing pairs of classes. As there is no training or any other usage of the dataset except for the class names, we decided to choose class pairs that are very different and CLIP can usually easily distinguish between them (e.g., ship and truck). We show additional examples from other pairs of classes (not only from CIFAR10) in Figure 15.

“The authors mention that the generated adversarial images lie on the manifold of generated images differently from non-semantic adversarial attacks. Can the authors elaborate on this claim?”

Classical (non-semantic) adversarial attacks modify images directly by adding pixel noise to the images, which causes the image to be classified incorrectly. Differently from this approach, our images are generated from a generative model that is trained to model the natural image distribution (for example – a diffusion model). We do not tamper with the generative model or the image pixels to make it adversarial.