ERICT: Enhancing Robustness by Identifying Concept Tokens in Zero-Shot Vision Language Models

Q1: The paper observes that “tokens whose semantics align more closely with the prompt tend to have lower similarity scores,” but provides only limited examples to support this claim. Given that this observation appears counterintuitive, stronger empirical validation is necessary.

A1: Thanks for the reviewer’s suggestion. We have provided additional image results (https://anonymous.4open.science/r/rebuttal-6618/more-image.pdf) from different datasets to further support this point. In each row, the leftmost image is the original image, followed by the score heatmaps and score maps for three different concepts in the image. These results further validate our findings.

Q2: Based on this observation, the method should ideally select the top-k lowest similarity token embeddings. However, it seems to do the opposite, which requires further explanation.

A2: We thank the reviewer for pointing out this error. In Equation (6), $S_i$ should actually be $S_{i}^{'}$, with $S^{'} = Sort(-1 \times S)$. In the implementation, consistent with the finding in line 178, we use sparsification to select tokens with lower scores to obtain the final visual representation. We will correct this error in the revised version. Additionally, we would like to re-explain the top-k strategy, which is used in ERICT-C to select potential class prompts for aggregation as auxiliary embeddings. In this process, we sort the inference results from vanilla CLIP and select the class prompts that are most similar to the image.

Q3: The selection of auxiliary prompts appears to require domain knowledge, and ERICT-C is only evaluated using the top-3 version. Both aspects require a more in-depth investigation.

A3: ERICT uses the superclass of the task category as prior knowledge to compute the auxiliary embedding. This prior does not require domain experts and is not complex. It is also used in the baseline method, PerceptionCLIP.

ERICT-C, on the other hand, does not rely on any prior knowledge. It computes embeddings for each class prompt in the classification task and aggregates the top-k embeddings with the highest similarity to obtain the auxiliary embedding. To demonstrate the effectiveness of the top-k strategy, we presented results with top-3 in the paper on multi-category datasets such as ImageNet-R. Additionally, to further illustrate this, we have provided more results for different values of k.

Q4 Compared with existing works, this work focuses on zero-shot learning without the help of extra LLMs and labeled data. However, expert knowledge to design the auxiliary prompt is needed, which might not be trivial for all tasks. The alternative method, ERICT-C, is not well evaluated.

A4: ERICT does require the introduction of certain priors to use auxiliary prompts, but these priors are similar to the simple assumption of "bird in photo" in the waterbirds dataset, which is also used in the widely discussed work PerceptionCLIP in this field. At the same time, ERICT-C does not rely on any priors, addressing this limitation and demonstrating superior performance. We have also provided additional experiments (e.g., Tables 1, 2, 3, 6 in our manuscript and the table in Q5) to validate the effectiveness of our method.

Q5: Compared with baseline method, ROBOSHOT, the method missed experiments on other datasets, e.g., PACS, VLCS and CXR14.

A5: We sincerely clarify that our paper primarily focuses on the issue of spurious correlations, while PACS, VLCS, and CXR14 are domain generalization (DG) benchmarks, not spurious correlation problems. Following the reviewer's suggestion and the experimental setup of ROBOSHOT, we conducted a comprehensive comparison on PACS, VLCS, and CXR14. The experimental results show that our method still demonstrates strong robustness on these DG benchmarks.

Q1: Section 5 lacks a detailed discussion on disentangling invariant and spurious factors, requiring further exploration.

A1: Consistent with previous works[1-2], our theoretical analysis adopts the classic data assumption, which disentangles spurious datasets into invariant and spurious parts. In this paper, our method identifies concept tokens to effectively extract both invariant and spurious information.

[1] Robust Learning with Progressive Data Expansion Against Spurious Correlation.NeurIPS 2023.

[2] A Sober Look at the Robustness of CLIPs to Spurious Features. NeurIPS 2024.

Q2: The experiments on different prompts were not presented.

A2: We thank the reviewer for pointing out this issue. The experiments in our paper used the commonly adopted prompt template "a photo of a {}". Additionally, we have conducted extensive experiments on CelebA to evaluate the performance of ERICT under different prompt templates, where Prompt 80 represents the average of 80 templates designed by OpenAI for CLIP.

Q3: The authors introduce a threshold l in Eq (5), but don't analyze its impact or the effect of different threshold methods.

A3: We further clarify the threshold l. In this paper, l is dynamically calculated based on the sparsification process, considering that the number of invariant information tokens varies across different images. Each image in the task has a different threshold depending on the variation in spurious information (see Equations (4) and (5)). The key parameter influencing the sparsification process is the temperature coefficient, which we have already analyzed in the ablation study section (lines 385-420). To further demonstrate the effectiveness of this process, we provide comparative experiments on different threshold settings using the CelebA.

Q4: The discussion about the auxiliary prompt. Other worthy prompt forms.

A4: We sincerely apologize for not clearly expressing this point in the submitted paper. What we actually want to convey is that the auxiliary embedding is very important as it guides model inference. Specifically, ERICT utilizes superclass information, while ERICT-C employs a Top-K strategy to aggregate class prompt embeddings. Since they obtain auxiliary embeddings in different ways, their performance varies across different tasks. We do not want to emphasize that the "superclass of the task category is better than the ground-truth category", but to point out the limitations of the auxiliary prompt (which requires certain priors), and then introduce ERICT-C. We will clarify this issue in the final version.

Q5: The caption of Figure 2.

A5: Thanks for the reviewer's suggestion. Figure 2 mainly includes two steps: In Step 1, we construct an auxiliary embedding $z_t^a$ to identify tokens containing invariant information, obtaining a token-level mask. In Step 2, we apply this token-level mask within the attention mechanism of the vision encoder, making tokens containing spurious information "invisible" to the [CLS] token.

Q6: The authors should clarify how the binary mask mitigates spurious correlations, particularly its impact on feature selection, information filtering, and decision-making.

A6: Thanks for the interesting question raised by the reviewer. According to our findings (line 178), the non-CLS tokens in the visual encoder capture local information. The original visual representation is obtained through the attention interaction mechanism between the CLS token and other tokens. By using a binary mask, we "shield" the tokens that focus on spurious information from the CLS token, so that the final representation no longer attends to spurious-related information (as demonstrated clearly in our T-SNE visualization). This prevents the model from making decisions based on spurious information.

Q7: What are the underlying reasons for spurious correlation in VLMs?

A7: Thanks for your interesting question. In our view, although pre-trained VLMs use large-scale image-text pairs, they still fundamentally follow a supervised learning paradigm, which cannot fully avoid the influence of fundamental learning issues, a point supported by related work [3]. Unfortunately, real-world data are full of statistical biases.

[3] Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning. ICML 2023

Thank you for your positive feedback on our work. We sincerely appreciate your recognition of our contributions and the significance of the research problems we address. Your support further strengthens our confidence in the proposed approach. If you have any additional questions or suggestions, we would be happy to discuss them.

Q1: However, the distinction between ERICT and ERICT-C needs clarification

A1: The key difference between ERICT and ERICT-C lies in the way they obtain the auxiliary embeddings. As shown in Step 1 of Figure 2, ERICT uses auxiliary prompts (e.g., "bird in photo") as input to the text encoder to obtain auxiliary embeddings. In contrast, ERICT-C constructs prompts using class names (e.g., "a photo of a landbird") as input to the text encoder. Then, ERICT-C ranks the embeddings output by the class names based on similarity and aggregates the top-K most similar embeddings to obtain the auxiliary embedding, where K is a hyperparameter less than or equal to the maximum number of categories in the dataset.

Q2: The underperformance of ERICT-C in some scenarios requires further investigation.

A2: ERICT and ERICT-C each have their strengths and weaknesses under different settings, primarily due to their use of two different strategies (as shown in Q1) for obtaining auxiliary embeddings. The difference in how these embeddings are obtained leads to varying performance in the task.

Q3: The choice of the top-3 strategy in ERICT-C lacks justification and the limitations mentioned in the text are not adequately addressed in subsequent sections.

A3: We apologize for any confusion caused to the reviewer. We have clarified the Top-K strategy. In the experiments, the value of k should be less than or equal to the number of categories in the dataset. For example, for the binary classification task on the Waterbirds dataset, K=2. Similarly, we have provided experiments with different values of K on three distorted versions of the Imagenet dataset, as shown below.

The limitation mentioned in the paper mainly refers to the suboptimality of the auxiliary prompt. In fact, we discuss this limitation in the "Impact of the auxiliary prompt" section of the ablation study (lines 400-412). How to obtain an auxiliary embedding with the best guiding significance is indeed a meaningful topic for further exploration. However, the main contribution of this paper lies in enhancing the model's robustness during the inference phase without relying on training, the assistance of LLMs, or group labels. We hope to address this limitation for future work.

Q4: The paper could benefit from experiments on more data such as MetaShift, and Living-17 datasets, as well as referring to more debiasing algorithms.

A4: As suggested by the reviewer, we add extensive experiments comparing the MetaShift and Living-17 datasets. The experimental results are shown below. These experiments demonstrate the robustness of our method across different datasets. Please note that the code for "Debiased Fine-Tuning for Vision-Language Models by Prompt Regularization" mentioned has not been open-sourced. The experiments for CLIPood and additional baseline results are presented in Q5.

Q5: The paper does not compare with ERM, AFR and CFR algorithms.

A5: We thank the reviewer for pointing out this issue. We have added detailed experiments comparing our method with extensive baseline methods. It is worth noting that although these baseline methods are training-based, our approach is training-free. However, in certain settings, our method outperforms these training-based methods, further demonstrating the robustness of our approach.