Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP

Thank you for your detailed comments on our paper, we appreciate the feedback. We are glad that you find the direction of our work important and interesting. However, we disagree with your characterization of our work on multiple points. We answer your concerns below:

[W1] In my opinion, the authors are just interpreting CLIP features ...

We disagree with this comment. It is true that by mapping to CLIP space and interpreting using the CLIP text encoder, we are "constrained" to using concepts that CLIP has learnt well. This does not mean, however, that this method of interpretation is ungrounded. Our results from the experiments on image-based image retrieval and zero-shot spurious correlation are evidence against this. In both these experiments, the mapping functions $f_i$ are only used for selecting the components to be used or ablated based on some specific property, and not used in performing the task itself (like image retrieval and inference on data with spurious correlations). If it were true that the property encoded in $f_i(c_i)$ is different from $c_i$, we would not observe property relevant images being retrieved or any significant improvement in worst group accuracy.

The authors put no restriction on preserving the semantics of c_i. ..

We discuss this in Section 4 Theorem 1 and the following paragraph. We show that an orthogonal map preserves the relative importance of features, since it is an isometry. We encourage $f_i$ to be orthogonal during training.

the regularization component in Theorem 1 has no effect in preserving the component role ....

The cosine distance (the first term in the loss function) measures how well $\sum f_i(c_i)$ match z_CLIP. The orthogonality regularizer (the second term) encourages the mapping to be an isometry so as to not distort the relative features (like you mentioned). Since these are two distinct objectives, the regularization component is not expected to have any effect on the cosine similarity - in fact it is a bit surprising that the cosine distance marginally decreases. The relevant metric for faithfulness of the mapping is the match rate which measures how well the TextSpan descriptions of a component (using the ImageNet trained linear head as the "text bank") match the corresponding TextSpan description of the same component after CLIP mapping and using the CLIP text embeddings of ImageNet classes as text bank. We observe a significant improvement in the match rate when the regularizer is introduced (in Table 1).

However, you indeed have a point that the improvements are not very large. This is because any randomly sampled high dimensional matrix is approximately orthogonal already, as for any two independent zero-mean Gaussian/uniform random variables $x, y$, $E[xy] = 0$ . In our experiments, training this matrix via gradient descent does not significantly change this property. However, since the orthogonality property is crucial to preserve the faithfulness of the mapping, we explicitly add it in the loss term. With this, the norm $\|| f_i f_i^T - I \||_F$ decreases by around 50% and match rate increases by around 0.03.

[W2] The authors did not show the validity of RepDecompose algorithm. ... It is shown that the match rate is extremely low (0.18), which disproves the validity of the algorithm.

[W5] The pseudo-algorithm in Page 4 is not clear and there are a lot of unexplained steps.

We address W2 and W5 together. RepDecompose is a generalization of the manual process by which Gandelsman et al computed component contributions. We have added a step-by-step explanation of the execution of RepDecompose on an attention-mlp block in the global rebuttal. This is still a simplified graph for illustrative purposes, the actual computational graph has many more nodes and is somewhat more complicated. We provide the code we use in the supplementary, the RepDecompose algorithm can be found in the file helpers/linear_decompose.py.

The match rate in Table 1 (as mentioned previously) is only meant to validate the faithfulness of the mapping functions $f_i$, and it is intended as a metric for CompAlign. It does not validate the RepDecompose algorithm in itself. The contributions returned by RepDecompose match those computed by Gandelsman et al exactly in the case of vanilla ViTs.

We acknowledge that the pseudocode in Algorithm 1 is unclear. While the intuition is very simple ( traverse through linear nodes by applying the distributive property and stop at non-linear nodes), the actual implementation is lengthy and requires extensive handling of edge cases which might have obscured the intuition. We will present the pseudocode at a higher level of abstraction in the camera ready.

[W3]

Our experiments on text-based image retrieval using components is only meant to show that (a) identifying certain image properties can be localised to certain individual components, and (b) our scoring function provides a useful signal for identifying these components. We do not claim that the individual components can retrieve images better than the overall representation or the CLIP representation. We show this quantitatively in Table 2 and 4. Please let us know If there are any concerns with the experimental setup for this section.

[W4] ... quantitative experiments on visualizing token contributions ...

Thank you for your suggestion. We have now added results of our experiments on image segmentation with competitive zero-shot segmentation baselines like GradCam and Chefer et al's method in the global rebuttal. Some of the methods are not equipped to work with certain models, in which case the cells have been left blank.

We hope that we were able to sufficiently address your concerns in the rebuttal. Please let us know about any remaining issues or concerns, and do consider increasing your score if you are satisfied with our answers.

I thank the authors for addressing my concerns. However, I still think W1 is not addressed. I specifically asked for experimental evidence (other than the match rate) for W1 in the rebuttal, which the authors did not provide. The match rate is extremely low: 0.185, and improves over the unfaithful baseline by only 0.03. Finally, having one metric to evaluate a whole new direction and the faithfulness of the method, is not convincing to me. The authors could formulate 1-2 experiments to prove faithfulness and that the semantics of the original model, is still preserved. I am also happy to provide suggestions on how to evaluate this, should the authors want to.

I am raising my rating by one score, but I cannot give any form of accept for this paper, as this paper does not really satisfy it's objective as written in the title, and the authors have not addressed W1. In fact, the paper makes the problem worse because it leads to interpretation of a different model (e.g., it interprets DINO as CLIP). Although all reviewers suggest accept, i think i have a valid point and I stand by it.

However, I will ensure to have fruitful discussions with the area chair and other reviewers after the rebuttal, to ensure that my decision is fair.

In the meantime, if the authors would like to discuss on this matter, I would be happy to do so.

Thank you for reading our rebuttal and updating your score. We are eager to address your remaining concerns with W1. But as stated in our rebuttal, we already offer strong experimental evidence in the form of zero-shot spurious correlation mitigation and image-based image retrieval. A quick recap:

1. We use RepDecompose to extract the direct contributions of these components $c_i$ such that the final representation $z = \sum c_i$
2. We use the trained linear maps $f_i$ from CompAlign to map each component $c_i$ to $f_i(c_i)$, and then using our scoring function, identify some salient properties encoded by these components
3. We then either select or ablate the top-k component contributions $c_i$ (and not $f_i(c_i)$) depending on the task, and perform image-based image retrieval or spurious correlation mitigation using these $c_i$.

This shows that regardless of the "low match rate", our method as a whole is successful at identifying roles performed by the components so that they can be used to perform these tasks. Please not that only the $c_i$ are being used for these tasks. If the $f_i$ maps were not faithful to the "true" interpretation of the model components, we could not perform well on these tasks. We would like to understand the reason why you don't believe in these experiments.

Also a note on match rate - we introduce this metric so that we can cleanly identify the delta between using the ImageNet pretrained heads vs CLIP text embeddings for interpretations. However, this metric has some flaws:

1. The match rate shown in the table is the exact match rate, and the approximate matches are not included in the metric.
2. CLIP text embeddings of the ImageNet classes are significantly different from the embeddings from the ImageNet trained fully conncected layer and may in fact represent different concepts. For example, CLIP's concept of a "library" may be different from the library images in ImageNet.
3. Some contributions may not have any clear interpretation, and in this case CompAlign is unfairly penalized since we take the ImageNet derived descriptions as proxy for ground truth.

However, we observed that any relative improvements in this metric did correspond strongly to the quality of the visualizations. Our purpose in providing the metrics was to show the relative improvement of CompAlign as compared to the "one map only" baseline that can be found in prior literature. Prior work such as Moayeri et al [1] have already used a single linear map to interpret the final layer of vision models using CLIP. Note that we improve on this baseline by around 0.1, which is significant.

We are happy to add any additional experiments that you suggest, if it can help in clarifying this point further.

[1] Moayeri, M., Rezaei, K., Sanjabi, M. & Feizi, S.. (2023). Text-To-Concept (and Back) via Cross-Model Alignment. Proceedings of the 40th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research 202:25037-25060 Available from https://proceedings.mlr.press/v202/moayeri23a.html.

Unfortunately there was an accidental omission in the reader list, which may have caused you to not see our response. We have edited the reader list now.

Thank you for the authors response. I would also like to apologize for the late reply.

we already offer strong experimental evidence

These are not indicative of faithfulness. Let's take the image-based image retrieval as an example. These experiments are based on selecting $k$ components, mapping them, and adding the results up to form a visual feature representation in CLIP space. The authors mention in the supplementary material: "the number of components k used for the image-based image retrieval experiment was tuned on an image-by-image basis", and they do not mention what $k$ was used for each image. $k$ could be tuned for each image seperately such that the results are good. If $k$ is good enough, it would be enough to approximate the visual feature representation in CLIP space which at the end, is almost equivalent to using CLIP to encode the image. We don't even know what the individual mappers are learning, and now we come to the problem of "interpreting an interpretation technique". There could be many cases one out of an endless possibility of cases. There is no experiment showing what these mapper components are really doing and how faithful they are, which is what I asked for in my first review. Once this visual feature representation is achieved by the mapper output, it is already in CLIP space, and it is reasonable (and not surprising at all) to expect a retrieved image which matches the reference image.

Throughout the rebuttal, it appears that the authors are avoiding conducting such experiments and instead are attempting to justify their work using existing experiments in the paper (and especially the match rate). In my previous response, I also gave the opportunity to the authors to select any other experiments to prove faithfulness. Since they did not, I could suggest two experiments (I didn’t give this a deep thought, these are from the top of my head), taking DINO as an example:

• Take the image samples that were classified differently by DINO (using the trained linear head) and CLIP (using zero-shot classification). Let’s denote those samples by Y. Avoid all samples that were classified correctly by both DINO AND CLIP. Now, what is the accuracy of Y for DINO before and after? If the mappers really preserve the semantics of DINO, then the accuracy of Y for DINO should not change much.
• The authors could use some interpretation technique such as Grad-CAM or Rollout or any other technique. This highlights relevant features. The authors could do this for the DINO model before and after the mapping. By then applying the Insertion and Deletion evaluation metrics for example (I assume the authors know them well if they work on interpretability), the Area-Under-Curve (AUC) should remain roughly the same.

Again, I kindly ask the authors to avoid justifying their answers with already existing experiments or metrics in the paper. I am aware of the authors paper well, and have read it three times already, two times during the initial review, and one time during the rebuttal. My comments are meant to direct what I find missing and important in the authors work.

These experiments are based on selecting components, mapping them, and adding the results up to form a visual feature representation in CLIP space.

This is exactly the misconception that we wanted to address. The image-based image retrieval does not involve adding the mapped component contributions in CLIP space, but instead adds the component contributions prior to the mapping, which are still in DINO space (or any other model space). The maps and the mapped components are only used in the interpretation process, and not during the validation of the interpretation. Thus, no matter what component contributions we add, they would always be in the representation space of the original model and not the CLIP space. The visual feature representation which is obtained by summing up the contributions is also in representation space of the original model.

The same also holds true for the spurious correlation mitigation experiments. This is precisely why we specifically pointed out these two experiments as evidence, as these do not involve the mapping during CompAlign.

The authors mention in the supplementary material: "the number of components k used for the image-based image retrieval experiment was tuned on an image-by-image basis", and they do not mention what was used for each image.

We mention in the next sentence that "$k$ is approximately around 9 for larger models like Swin or MaxVit, and around 3 for the rest." To be more precise, what we mean is we tune $k$ within a range of $9 \pm 4$ for SWIN/MaxVit and $3 \pm 2$ for the rest of the models. The main reason for this variation in $k$ is that we break down SWIN/MaxVit into many more components as compared to the rest, which means that each contribution vector for SWIN/MaxViT is less informative. We then select the top $k$ contributions strictly according to the scoring function. This is however beside the point because as we explained no matter what $k$ we choose, we can never recover the CLIP representation using the component contributions because they are not in the CLIP space in the first place. In addition, we also present spurious correlation mitigation experiments in which we do not tune $k$ at all.

Further experiments

We will think more on the lines of the experiments you suggested and hope to add some results if time permits. However, we note that as currently formulated, the experiments do not cleanly disentangle the use of the mapping functions for interpretation and application. Ideally, there should be an "interpretation" stage where we use the maps to get some information about the models, and a "validation" stage where we use this information to manipulate the model somehow, without involving the maps or the mapped contributions in any form. Therefore, we still believe that the existing experiments are much more convincing.

Thank you for your favorable review. We are happy to hear that you found our work clear and coherent, with comprehensive evaluation. We answer your questions below:

Ambiguity in Introduction

Thank you for the feedback. We will take care to expand and rephrase the introduction to remove this ambiguity. We have also added a step-by-step explanation of the execution of RepDecompose on one attention-mlp block in the global rebuttal. The reason why we can add multiple contributions together is because RepDecompose automatically applies the transformations to the output of each component ($z_i$ in the figure) so that the contributions $c_i$ are in the same vector space as $z$.

Please let us know if there are any other points in the introduction that were unclear or ambiguous.

Generalizability

It is true that for our algorithm to work, all non-linear components must be short-circuited via a residual/skip connection in the model architecture. Fortunately, this is almost always the case in modern neural networks, even in modern CNN architectures like ConvNext. This is because the skip connections are added to ensure that any "dead neurons" in the non linearities does not affect gradient flow. This means that it is now a safe assumption that non-linearities can be "skipped" using residual connections. It is also true that only the linear parts of the model can be decomposed, but modern ViTs are overwhelmingly linear in nature with very few non-linearities in practice. Therefore, we expect our method to generalize well across multiple modern architectures.

The reduction of c_{i,t} to c_{i} through direct summation relies on a strong underlying assumption

The "underlying assumption" being referred to here is not very clear to us. Since the attention is linear in the OV circuit, it can be further decomposed as a sum over contributions from tokens. This holds as long as there are no non linearities in the OV circuit. Please do clarify further if we misunderstood you.

Limited Practical Applications

This is a relevant point, thank you for bring this up. In our paper, we demonstrate a practical application for mitigating spurious correlations by ablating the output of a specific set of attention heads, identified using our method. You can refer to Section 6.4, where we show improvements in both worst-case and average group accuracy (see Table 3) for the Waterbirds dataset. This result illustrates that manipulating or removing specific components, as guided by our framework, can enhance model robustness. We have also added results on zero-shot segmentation comparing it with other competitive baselines such as GradCAM and Chefer et al's method. Our method outperforms the baselines whenever they are applicable for a given model.

Our interpretability framework is primarily validated through downstream applications in text-based and image-based image retrieval, as demonstrated in Sections 6.1 and 6.2. Our findings however suggest that practical tasks like image-conditioned image retrieval based on specific features (e.g., patterns or locations) can potentially be performed in a zero-shot manner using any vision encoder, rather than requiring a specialized model. While achieving state-of-the-art results in this task is not the primary focus of our work, we believe our results lay the groundwork for unlocking various capabilities in vision encoders, all without the need for additional training.

Structural and Writing Issues

We thank the reviewer for pointing out these issues. We will correct them in the final version of the paper.

Have you conducted experiments to show how manipulating or removing specific components based on your RepDecompose and CompAlign observations affects model performance on specific tasks?

Our experiments on mitigating spurious correlations show that removing specific components helps in mitigating spurious correlations. We expect that our framework can also be applied to do model unlearning in an efficient manner by finetuning or removing specific components, however we leave this for future work.

While image retrieval and token visualization provide insights into model behavior, have you considered more advanced downstream tasks that could demonstrate the practical benefits of your framework?

How might your framework be extended to not only understand but also guide the improvement of ViT models? Are there plans for future work in this direction?

Thank you for raising this important question. Beyond model understanding, in our paper we demonstrated the potential applications of our framework in spurious correlation mitigation, image/text-based image retrieval, and zero-shot segmentation. We believe our framework can also be applied to address issues related to harmful biases or copyrighted image information within the network. By using probe text descriptions, it is possible to identify model components that encode such biases or infringement information, which can then be ablated to develop safer vision models in a post-hoc way. However, we note that these applications, while impactful, are beyond the scope of the current paper and represent promising directions for future research.

Thank the authors for your response. The rebuttal clears most my previous concerns. Hence, I have decided to keep the original positive rating.

Thank you for the favorable review. We answer your questions below:

Lack of qualitative or quantitative comparison with the related works, such as the previous image representation decomposition method mentioned in Sec.3.

We would like to point out that our work can be viewed as an extension or generalization of the work from Gandelsman et al, which was specific to CLIP models. Our method, on the other hand, works on models with various architectures trained with different methods.

We now include a comparison of our method with competitive baselines like Gradcam and Chefer et al's method in the global rebuttal. Note that some of the baselines are not equipped to work on architectures like SWIN or MaxVit, in which case we leave it blank.

lack of a quantitative evaluation of the text or image-based image retrieval performance.

We have provided a quantitative evaluation of the text based image retrieval performance in Table 2 and 4 in the paper. In the paper, this is meant as a validation of our overall method as we compare the ability of various components to retrieve images wrt a certain property as measured by the CLIP score ranking.

Could the flowchart of REPDECOMPOSE and COMPALIGN also illustrated by a more intuitive figure? The algorithm in Sec3.1 and Sec4 and the corresponding descriptions are not very clear.

We have added a step-by-step explanation of the execution of RepDecompose on one attention-mlp block in the global rebuttal, and we will add this in the camera-ready version should the paper be accepted. We will also revisit the CompAlign section and elaborate the algorithm further. In essence, the purpose of CompAlign is to map (or "align") the contributions to the shared CLIP space via an approximately orthogonal map.

How to explain that with some models, the top-3 images retrieved by the most significant components for a specific property are unrelated to that property?

It is possible that in some cases, we do not find any component which predominantly encode the property of interest. In these cases, retrieving images based on these components may also return images which are similar in dimensions other than the specified property.

We hope that your concerns with the paper are addressed with this, if there are still some remaining issue please do let us know. We request you to consider increasing the score if you are satisfied with our answers.

Thanks for the response. It addressed my concerns. I decide to keep my rating.

Thank you for your favorable review. We are glad that you found our work novel, clear and convincing. We address your questions below:

A brief explanation of the TextSpan algorithm can be useful to understand the paper without the need for reading Gandelsman et al.

We agree and apologize for the oversight. We will add this in the appendix in the final version of the paper.

It is not clear that the presented scoring function is more useful than using TextSpan - a qualitative and quantitative comparison between the two approaches can provide a more convincing case for the introduction of this approach.

Thank you for this point. We do not claim that our scoring function is superior to TextSpan - the motivation and use cases of the two are somewhat different. TextSpan is intended to automatically discover the roles of various components by mining text descriptions from a text bank. The text descriptions thus obtained are sometimes coherent and point to a well-defined role for the component, but at other times all over the place, making the attribution of a role to the component difficult (see the appendix of Gandelsman et al). Thus, TextSpan is a more unsupervised method which tries to understand the model's components on its own terms.

Our scoring function, on the other hand, is useful for identifying components which have a role in representing some externally specified property. Since the property is specified manually, it is unlikely that there will be a specific component which corresponds to this property. This is the reason we may need to select multiple components corresponding to a particular property. Thus, we need a way to sort and select components which encode a given property, which is where our scoring function comes into picture.

Leaving the motivation aside, TextSpan and our scoring function are closely related in a technical sense. Both use the variation of the component contributions with respect to a subspace determined by the text encoder. We also use TextSpan to validate CompAlign's mapping functions in Table 1.

While the presented approach uses CLIP, it doesn't use the main advantage of this model ... As the scoring algorithm can use these datasets instead of the small set of concepts, more text descriptions might introduce a new understanding of the different components.

This is an interesting idea. We do not know if ImageNet trained/ self supervised models can learn to encode complex object relations (as present in "the dog chases after the cat") in the absence of text supervision. In fact, even CLIP models suffer from limited understanding of compositionality and are prone to treat the text prompt as a bag of words. However, we would have to defer this for future work owing to lack of space.

Low match rate - it's not clear what is the reason behind the low match rate. One possibility is that CLIP is not expressive enough. It will be useful to show how does the match rate behaves for different CLIP models.

There may be multiple reasons for this:

1. The match rate shown in the table is the exact match rate, and the approximate matches are not included in the metric.
2. CLIP text embeddings of the ImageNet classes are significantly different from the embeddings from the ImageNet trained fully conncected layer.

Note that even with the low match rate, we are able to determine components which perform a specific role well enough to do zero shot spurious correlation mitigation and image-based image retrieval.

We tried two CLIP models, ViT-B-16 and ViT-L-14, as the source of $z_{\text{CLIP}}$ for the experiments and we found that the bigger model (ViT-L-14) yielded better cosine similarity and match rate. We would need to conduct more experiments before we can definitively conclude this.

It is not clear why the representations should be mapped to CLIP image features first, as opposed to directly mapping them to a sparse linear combination of words in CLIP (i.e. via sparse autoencoder). A comparison to this approach can be very useful.

The primary reason for this is ease of training, as in this manner, our method can be used on any image dataset without accompanying text descriptions. It also uses a simple cosine distance loss rather than a more complicated loss like contrastive loss. Directly mapping the $c_i$ to the text encoder space also has issues due to the well-known "modality gap" between CLIP image and text representations.

In some experiments that didn't make it into the paper, we trained the maps $f_i$ using a contrastive loss on the MS-COCO dataset. The loss decreased slowly, and the resulting $f_i$ were unsatisfactory in terms of match rate and cosine similarity. One potential reason for this is that the supervision offered by the contrastive loss over text descriptions is much weaker than the cosine similarity with CLIP image features.

Thank you for your positive review. We are encouraged that you found our work novel with comprehensive experiments. We reply to your questions below:

The paper does not include comparisons with any baseline methods. Although previous methods may not extend to arbitrary Vision Transformer models, a comparison with specific models would be beneficial.

Thank you for your suggestion. We have now added results of our experiments on image segmentation with competitive zeroshot segmentation baselines like GradCam and Chefer et al's method in the global rebuttal. Chefer et al's methods are not equipped to work with SWIN and MaxVit, in which case the cells have been left blank.

The paper lacks an analysis of failure cases, leaving it unclear how well the proposed method can be applied in real-world scenarios.

Some limitations are discussed in the Limitations section in the appendix. When considering the failure cases for real world applications such as spurious correlation mitigation, the most significant is the case when both the spurious features and core features happen to be represented in the same component(s). In this case, it might be necessary to decompose the component contribution a second time into higher order components. However, as we mention in the Limitation section, we do not investigate this possibility currently.

Could you explain why ImageNet pretrained models encode useful features redundantly?

We tentatively offer the following hypothesis. ImageNet pretrained models (and its components) presumably learn many features that are most useful for datasets in-distribution with ImageNet. Since we conduct our experiments on the ImageNet validation split, it is likely that these models have many more features useful for ImageNet classification compared to other models such as CLIP or DINOv2. It seems that for non ImageNet pretrained models, much of these useful features are concentrated in the last layers. Thus, when performing layer ablation, it so happens that the accuracy drops more quickly when compared to ImageNet pretrained models.

How did you accumulate the contributions from the lower layers into a single contribution ?

Referring to the figure and the explanation in the global rebuttal, we see the operation of RepDecompose on a single attention mlp block of a vanilla transformer. Here, $z_5$ contains the contributions from all the previous layers. RepDecompose appropriately applies the suitable transforms (in this case, only the LayerNorm) and obtains the contribution of $z_5$ to the final representation $z$. In this case, the contributions from the lower layers are automatically accumulated into $z_5$

Why does component ordering tend to have a higher correlation than feature ordering, particularly for SWIN and MaxVit, as shown in Table 2?

There are some components whose direct contribution is not useful for encoding any feature at all, although they may have useful indirect effects. When ordering the components according to their ability to retrieve a specific feature, these components are predictably at the bottom. However, for any component, there is no feature which is predictably the most (or least) dominant. Thus, the component ordering problem is inherently a somewhat easier problem, and our proposed scoring function is better at ordering components as compared to features.