Interpreting CLIP's Image Representation via Text-Based Decomposition

We thank the reviewer for the valuable comments.

I am thinking about the task from a human-AI taming point of view. How hard is it for the human to identify the role of each head? Is it possible to introduce some humans during inference time to prune/select which head to use for final prediction?

In our experiments, we manually annotated the heads by examining the output text descriptions of our algorithm. For many of the heads, this task is simple (requires finding a commonality between 60 text descriptions). A possible approach for automating it is to query an LLM to summarize the role of the head based on its text descriptions (e.g. “What is common between all these image descriptions?”). Moreover, for pruning the heads for a specific task (e.g. bird classification), we can provide the discovered head roles to an LLM, and query it about each spurious ques that each role can provide for the given task (e.g. “Can geo-location be a spurious cue for bird image classification?”). This model-driven strategy would probably be more efficient than introducing humans during inference time, which might be too costly.

“Note everything has a direct role” - Does it mean there exists complicated features that we need to leave it as a black box, or we can ignore them as redundant feature points?

By “not everything has a direct role”, we are referring to the fact that our analysis studies the direct effects of individual components on the output representation rather than indirect effects.

In more detail, the direct effects are the contributions of each layer to the residual stream (which is later projected into the output space). The indirect effects are the contributions of a layer that are processed by subsequent layers. Most of the early layers, for example, have meaningful indirect effects - they contribute to the inputs of later layers. To analyze these features, one should examine these contributions and their downstream effects. As a preliminary example of this, we unrolled the direct effect of a “counting head”, and found a second-order effect from an MLP layer that fires mostly when digits appear in the image, which could produce spurious cues if the presented digit usually corresponds to the number of objects in the images (e.g. a pack of 6 tomatoes with the text “6 tomatoes” appearing on the pack). Removing the second-order effect of this MLP on this head may result in more accurate counting abilities.

We thank the reviewer for the valuable comments.

“the zero-shot accuracy of the embeddings from only late attention layers is very competitive with the original performance. Will this also hold true where the representations are used for downstream tasks that require adaption?”

To verify that the effect of the early attention layers and MLPs is negligible even after adaptation, we followed the suggested experiment and applied linear probing on ViT-B-14 for CIFAR100 classification. The test set accuracy of linear probing is 84.3. When we mean-ablate all the MLPs and the early attention layer up to the last 4 layers we get an accuracy of 84.2%. This suggests that even after adaptation, the late attention layers still have most of the effect on the output.

“How do the zero-shot results compare against methods like MaskCLIP”

MaskCLIP uses a different approach, that works only on CLIP models that have attention pooling before the projection layer. Differently from these models, other CLIPs (and specifically - all the OpenCLIP models) use the class token directly as the input to the final projection, without using attention pooling. We note that it is possible to compare our model to maskCLIP by modifying the decomposition to include attention pooling in it, but we think it is out of the scope of this paper.

We thank the reviewer for the valuable comments.

"can the authors comment on how indirect effects can be leveraged to understand the internal representations?"

Examining the indirect effect can result in a finer understanding of the internal computation graph in these models. For example, we can analyze the second-order effects going through a specific layer/head (e.g. what does MLP 2 contribute to attention head 5 in layer 30). This way, one can remove more fine-grained spurious correlations.

As a preliminary example of this, we unrolled the direct effect of a “counting head”, and found a second-order effect from an MLP layer that fires mostly when digits appear in the image, which could produce spurious cues if the presented digit usually corresponds to the number of objects in the images (e.g. a pack of 6 tomatoes with the text “6 tomatoes” appearing on the pack). Removing the second-order effect of this MLP on this head may result in more accurate counting abilities.

“Did the authors analyse the OpenAI CLIP variants using this framework? The OpenCLIP and OpenAI variants are trained on different pre-training corpus, so a good ablation is to understand if these properties are somehow dependent on the pre-training data distribution.”

We provide here an additional comparison between OpenCLIP-ViT-L14 and OpenAI’s CLIP-ViT-L14. First, our observation that the last few attention layers have most of the direct effects on the final representation still holds: keeping the last 5 attentions, and ignoring the direct effects of other layers, result in only a small decrease from 75.5% to 73.2% in accuracy on ImageNet. Second, we evaluate OpenAI's CLIP-ViT-L-14 for zero-shot image segmentation:

As shown here, the localization properties of OpenAI-CLIP are comparable to the results of OpenCLIP.

“While this (initial descriptions pool) set is a good starting point, can the authors comment on how this set can be extended to make it more diverse?”

We repeat our experiments with a larger and more diverse image description corpus and present the results in section A.4 in the revised version. We query ChatGPT for class-specific descriptions and create an additional pool of 28767 unique sentences (8 times larger than our previous corpus). This results in higher accuracy with fewer basis vectors per head (smaller $m$) compared to the other pools that contain 3498 vectors, as shown in Figure 7.

Aside from this, and more importantly for future work, our method can be made adaptive, by applying it iteratively and using a data-driven refinement step to update the pool. Specifically, at each iteration, we can apply our algorithm to retrieve a basis set and the corresponding descriptions (which indicates roughly what concepts the model is using), query an LLM to generate “more like these” descriptions (to retrieve more fine-grained descriptions), and add the generated new examples to the pool before re-computing the basis set again. This approach may result in more fine-grained descriptions that capture better the output space of each head. We plan to explore this idea further in future work.

Clarification about the image set in Sec. 4.1

We apply our algorithm on ImageNet validation set (50000 images), as it is the largest and most diverse dataset we could use given our compute budget.

We thank the reviewer for the valuable comments.

“No ablation studies on the impact of the … basis size (m)”

We believe that Figure 3 already implicitly contains the desired ablation. Specifically, in Figure 3 we simultaneously replace each head’s direct contribution with its projection to the $m$ text representations found by our algorithm. We consider a variety of output sizes $m \in \\{10, 20, 30, 40, 50, 60\\}$. We evaluate the reconstruction by comparing the downstream ImageNet classification accuracy. As shown in the figure and discussed in section 4.2 the accuracy improves with larger $m$ values. We found that 60 descriptions ($m=60$) were sufficient to reach an accuracy that is close to the original model accuracy, but with fewer descriptions, the accuracy drops.

“No ablation studies on the impact of the pool size (M)”

To vary the pool size $M$, we repeat our experiments with a larger and more diverse image description corpus and present the results in section A.4 in the revised version. We query ChatGPT for class-specific descriptions and create an additional pool of $M=28767$ unique sentences (8 times larger than our previous corpus). This results in higher accuracy with fewer basis vectors per head (smaller $m$) compared to the other pools that contain $M=3498$ vectors, as shown in Figure 7 in the revised version.

“all layers but the last 4 attention layers has only a small effect on CLIP’s zero-shot classification accuracy” maybe just because the early layers’ feature are not semantically distinctive? But they should be still important to extract low-level features.

In our analysis, we consider only the direct effects and ignore the indirect effects (which would include extracting low-level features that get used in later stages of processing). Our claim is only that the direct effect of the early layers on the output is negligible (Figure 2). In contrast, we expect the indirect effects to be large, since the early layers contribute to the inputs of later layers.

“Is it possible to achieve the “dual” aspect of the text encoder of the CLIP”

While we think that decomposing the text encoder is out-of-scope for this paper, we believe that such a dual analysis is possible, since the text encoder has a similar transformer architecture. Moreover, decomposing both the text encoder representation and the image encoder representation will allow us to examine what each (patch, word) pair contributes to the overall similarity score. We plan to explore it in future work.