PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts

Thanks for your support! We have incorporated all the suggestions from all reviewers into the revised version. Below, we address your questions.

Q1: For some of the experiments I couldn’t find if they employed ClassAttr or PureAttr.

Thank you for highlighting the need for clarity regarding the use of ClassAttr or PureAttr in our experiments. In all the experiments, we use ClassAttr if not specified (noted in Appendix E.4). To make it more clear, we have now included a specific note in Section 7 of our revised paper.

Q2: In Algo 1, I assume it should be “Set of classes Y” rather than “class Y”, and that the sum is over y \in Y.

Thanks for the question. In our paper, we use $Y$ to denote the random variable and $y$ to denote its realization. While, in practice, it represents a set of classes, we opted for the term "class Y" in the algorithm for mathematical conciseness. Your point about the summation is correct. It indeed encompasses all possible realizations.

Thanks again for your time and effort in reviewing our paper! We are happy to have more discussions for any further questions.

Thanks for your constructive suggestions. They are very helpful in enhancing our paper's quality. We have incorporated all of them to our revised paper. Below, we address your questions in detail.

Weakness 1: If I understand correctly, Figure 3 gives only the score (x100) of the correct class in different scenarios. Is this completely informative? I think it can be easily misleading. What if the model provides relatively higher scores to some of the wrong classes as well? Can the authors analyse the score distribution of the wrong classes? Reporting the mean of CLIP_score@topK might be a good start to understanding the false positives.

We appreciate your insightful comment. To address this, we've included Figure 5 (referenced alongside Figure 3) to scrutinize the increased CLIP score for both the ground-truth and wrong classes. This analysis is detailed in Appendix E.1.

In Figure 3, we compared {no contextual attribute, correct contextual attribute, wrong contextual attribute, random strings} and found that incorporating correct contextual attribute improves the CLIP score of the ground-truth class the most, indicating CLIP understands those attributes. However, as you rightly pointed out, it does not necessarily equate to increased accuracy since other classes could also have improved CLIP scores. That's why we test the accuracy directly in Section 5.1 and Table 2.

We agree that comparing the improvement for the correct class and the wrong classes is a good idea to complete the picture. Therefore, for each class $y$, we calculate the improvement brought by contextual attributes with

$\Delta_y \triangleq \mathtt{CLIP}(y, z^*; x) - \mathtt{CLIP}_1(y; x)$.

In Figure 5, we contrast these improvements, comparing $\Delta_{y^*}$ for the correct class against $\Delta_{y_{wrong}}$, with the latter being the mean increase across the Top-K incorrect classes. The results confirm that the scores for the correct class see a more substantial increase when the contextual attributes are accurately described. This is because the accurate description of the class and the attribute best aligns with the corresponding image. Figure 3 and 5 together validate that the CLIP model understands the contextual attributes, and describing correct class and attributes yields higher similarity scores as described in Equation (4). Such findings also explain the increased accuracy in Table 2 when incorporating the correct contextual attributes.

Weakness 2: I appreciate the visualisation study provided by GradCAM as a qualitative analysis but I am not confident of the calculation of “core” versus “spurious” features [1].

We appreciate your concern and acknowledge the insights from the concurrent work [1]. The observation of the registers is really interesting and intriguing. Although our visualization, which is based on Grad-CAM, differs from the attention maps tested in their paper, there could also be a register problem in our case. This is an interesting open question. Additionally, whether visualizations like Grad-CAM truly show how the model works remains an open question. Athough these open questions exist, the visualization still provides us with valuable insights, allowing us to better understand the model. We have tried our best to isolate the potential influence of the registers. In Figure 4, we demonstrate that the reduction in reliance on spurious features is notably observed only when the correct context is introduced. This effect is distinctly absent when incorrect contexts or random tokens are used, suggesting a more nuanced interaction than merely an increase in token count. Additionally, we have added a quantitative evaluation in Table 3 (new results). These results align with our visualizations in Figure 4, indicating that the correct contextual information indeed directs the model's focus towards core features, an effect that diminishes with the use of random strings.

While we acknowledge the limitations of current visualization techniques, we believe our multi-faceted approach, combining both qualitative and quantitative analyses, provides a comprehensive understanding of how contextual attributes influence model behavior.

[1] Darcet, Timothée, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. "Vision Transformers Need Registers." arXiv preprint arXiv:2309.16588 (2023).

Since the authors did not provide responses, I would like to keep the original rating.

Thank you for the constructive feedback. Your suggestions are very helpful. We have incorporated all of them in our revised paper. Here, we address your questions and concerns in detail.

Weakness 1: Collecting the contextual attributes requires either pre-knowledge for the test image or a large dataset containing captions, which hinders the generalization ability of the proposed method in the real world. For instance, contextual attributes for the CelebA dataset are manually defined, e.g., gender, age, etc. To collect the contextual attributes for the remote sensing dataset EuroSAT, the authors first retrieve similar images and captions from a large image+text dataset LAION-400M, then ask GPT-4 to summarize the contextual attributes. What if we do not have external datasets to provide captions?

Thanks for bringing this up. It's true that our method, like other zero-shot classification approaches, requires some prior knowledge of the data. However, the type and amount of prior knowledge needed in our case are relatively minimal and practical to obtain. We only need basic, class-independent contextual attributes like background or illumination for natural images and gender for facial images, which are generally known or easily inferred. This simplicity contrasts with other methods [1-2], which require more detailed, class-specific descriptions for every class.

One advantage of our method lies in its flexibility with contextual attributes. We only need a set of possible contextual attributes since CLIP itself can first infer the most relevant contextual attribute from this set. Such flexibility makes our approach adaptable to a wide range of datasets which was demonstrated in our paper across 13 different datasets.

For situations where contextual attributes are not readily available, we propose an alternative: using the LAION caption dataset and GPT-4 to automatically identify relevant attributes. This approach is grounded in the fact that OpenCLIP is trained on LAION-2B [3], and CLIP was trained on similar large-scale caption datasets. Since CLIP acquires the knowledge of contextual attributes from the training data, LAION is a good source to find contextual attributes automatically for almost all datasets.

[1] Menon, Sachit, and Carl Vondrick. "Visual classification via description from large language models." ICLR 2023.

[2] Pratt, Sarah, et al. "What does a platypus look like? generating customized prompts for zero-shot image classification." CVPR 2023.

[3] https://laion.ai/blog/large-openclip/

Weakness 2: The qualitative results in Figure 4 indicate that introducing the contextual attributes reduces reliance on the spurious features and the model focuses more on the core features. It would be fairer to provide a quantitative evaluation, e.g., counting the percentage of model attention on the core features versus on spurious feature on all test set in ImageNet, and compare the ratio of different models.

Thank you for your valuable suggestion. We've now added Table 3 in our paper, presenting quantitative results. This table, along with a detailed discussion in Section 5.1 and Appendix E.3, addresses this point.

To calculate the percentage of model attention on the core versus spurious feature, we need the segmentation of them. We do not have such segmentation for the ImageNet dataset. Therefore, our evaluation is conducted on the Waterbirds dataset where we have the segmentation of the core feature (e.g., bird) and spurious feature (e.g., background).

The results in Table 3 highlight that when the correct context (here, background) is specified, CLIP shifts its focus more toward the core feature for classification. This effect is not observed when using incorrect or random context. Thus, both Figure 4 and Table 3 collectively demonstrate that introducing contextual attributes effectively reduces the model's reliance on spurious features and enhances its focus on core features.

Weakness 4 & Question 1: The results in Table 7 are not consistent among different backbones. It is hard to get any conclusion on which method is better. In Table 7, why lower gap between the Avg and Worst is better?

In Table 7 (Table 8 in our revised paper), our goal is to demonstrate how our method enhances group robustness by incorporating contextual attributes regardless of the backbones. Group robustness measures how consistently the model performs across different groups within a dataset. For example, in the Waterbirds dataset, there are four distinct groups: {waterbird on water, landbird on land, waterbird on land, landbird on water}. The first two are considered majority groups as they are more prevalent in the training data, while the latter two are minority groups. Typically, due to training set imbalances, models tend to perform better in majority groups. However, we need models that perform well in all groups. To this end, following the convention in this field [1], we assess model accuracy on each of the four groups, calculating both the average accuracy (Avg) and the worst group accuracy (Worst). The gap (Gap) is the difference between these two metrics. A higher gap indicates a bias towards certain groups, implying less consistency and robustness. Conversely, a lower gap signifies a more balanced and robust model performance across all groups. Therefore, the lower gap is better.

We would like to clarify that the purpose of Table 7 isn't to compare different backbones. Instead, we aim to show that our method is effective across various backbones. By comparing our method (with $\mathcal{Z}$) against the baseline (without $\mathcal{Z}$) for each backbone, we demonstrate that our approach consistently reduces the gap between the average and worst group accuracies, thereby achieving better group robustness.

[1] Sagawa, Shiori, et al. "Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization." ICLR 2020.

Thanks again for all your time and effort in reviewing our paper! Could you please consider increasing the score if you are satisfied with our answers? We are happy to have more discussions for any further questions.

Thank you for your support and all your questions. Here, we address them in detail.

Weakness 1: The authors assert at least twice that PerceptionCLIP mirrors human perception. However, I'm not entirely convinced. Authors gave the preliminary that: “humans first infer contextual attributes (e.g., background and orientation) which help separate the foreground object from the background, and then classify the object based on this information.” Yet, there's no evidence indicating that PerceptionCLIP actively separate foreground from background during classification, or that such separation is utilized the model. It's possible that PerceptionCLIP utilizes background attributes differently.

Thanks for your feedback! Let me clarify how PerceptionCLIP emulates human perception. Our approach, outlined in Algorithm 1, involves two main steps inspired by the human perceptual process. Firstly, CLIP infers contextual attributes, which is akin to how we humans first discern context, like background or orientation, to understand a scene. Secondly, this contextual information is used for classification, much like how we classify objects based on the context we've perceived.

It's important to note that in our model, separating foreground from background is just one example of using contextual attributes, not the entire story. The key aspect of human perception that we're trying to capture is the ability to use context effectively. By integrating these contextual attributes into our model's prediction process, we've seen a notable boost in zero-shot classification performance. PerceptionCLIP isn't about mimicking the exact human process of separating foreground and background, but rather about leveraging context in a way that's inspired by human perception.

Weakness 2: The authors refer to background information as spurious features (e.g. Figure 1). To my knowledge, it is not completely correct. Though they can sometimes overlap, they are not the same. Background information is a broader concept, while spurious features specifically refer to misleading patterns that a model might incorrectly learn as being important. In addition, the GradCAM in Figure 1 primarily emphasizes the foreground, consistent with [a], without highlighting any reduced reliance on spurious features. It's more accurate to state that it offers enhanced focus on foreground objects.

Thank you for pointing out the distinction between spurious features and background information. You're right in noting that spurious features are generally misleading patterns that a model might erroneously learn as significant. In our context, we use the term "spurious feature" to also encompass parts of the image with a recognizable concept, like the background, as referenced in several works including [1].

In Figure 1, our aim was to demonstrate the model's shift from relying on spurious (or background) features to focusing more on core (or foreground) objects. The model's reliance ratio changes from 31:69 (background vs foreground) in CLIP to 23:77 in PerceptionCLIP, suggesting that our method indeed reduces dependency on what we've termed as spurious features. This aligns with your observation about the model offering an enhanced focus on foreground objects. We appreciate your feedback and will consider clarifying this distinction in our revision.

[1] Moayeri, Mazda, et al. "Spuriosity Rankings: Sorting Data to Measure and Mitigate Biases." NeurIPS 2023.