Cross-modal Mitigation of Spurious Correlation for Prompt-tuning in VLMs with Causally Motivated Logic Alignment

Thank you very much for your valuable comments and feedback, which significantly contributes to improving the quality of this paper. Detailed responses to your concerns and questions are listed below.

W1: Causal Representation Learning baselines adapting from other tasks can largely consolidate the motivation of this paper.

Answer: We adapt two representative causal representation learning methods from invariant learning: IRM [1] and IB-IRM [2]. They mitigate spurious correlations by ensuring the invariance of the conditional probability of the label $Y$ given the causal representation across varied training environments. The evaluation is conducted using ResNet-50 as the backbone model. The experimental results on four datasets are list as follows:

[1] Arjovsky, Martin, et al. "Invariant risk minimization." arXiv preprint arXiv:1907.02893 (2019).

[2] Ahuja, Kartik, et al. "Invariance principle meets information bottleneck for out-of-distribution generalization." Advances in Neural Information Processing Systems 34 (2021): 3438-3450.

Analysis: We can find that LogicAl-PT outperforms the typical causally invariant representation learning methods. The underlying reason stems from the advantage of 'sufficient and neccesary' causal representation over traditional causal representation, which also forms the motivation for proposing LogicAl-PT for prompt tuning of VLMs. Prevalent causal representation learning methods primarily aim to mitigate non-causal spurious correlations. In contrast, the concept of 'sufficiency and necessity' goes further by excluding not only non-causal spurious correlations but also causal relationships that are 'sufficient but not necessary' or 'necessary but not sufficient'. We provide specific examples to clarify these types of relationships and explain why only 'sufficient and necessary' relations remain stable across diverse data distributions in Figure 4 on page 14 in Appendix A.

 

W2: Ablation study of the proposed is not enough. What is the effect of different $\alpha$, $\beta$?

Answer: We add experiments to evaluate the effects of two significant hyper-parameters in the proposed objective (i.e., $\alpha$ and $\beta$) on model performance. Since the results on other datasets present the similar tendency as on ImageNet-1K, we herein focus on ImageNet-1K, with ResNet-50 as backbone model. When evaluating the effect of $\alpha$, we fix $\beta=1.0$ . When evaluating the effect of $\alpha$, we fix $\alpha=20.0$. The experimental results are shown in the following two tables:

Analysis: When $\alpha=0.0$, model is tuned with only textual logic alignment; when $\beta=0.0$, models is tuned with only cross-modal logic alignment. We can find the performance of LogicAl-PT is more sensitive to the selection of $\alpha$ than the selection of $\beta$. To effectively mitigate spurious correlations in VLMs, careful tuning of $\alpha$ is essential. Regarding $\beta$, a small value is safer in practice, as a large $\beta$ may compromise the discriminative capability of the extracted features.

Thank you very much for your valuable comments and feedback, which significantly contributes to improving the quality of this paper. Detailed responses to your concerns and questions are listed below.

W1: Confused symbols: $\Phi$ denotes the filter in Cross-modal logic alignment but visual representation space in Textual logic alignment.

Answer: Thanks for your suggestions. To distinguish the filter from the representation space, we use a different symbol ($\mathbf{h}$) to denote the filter in the revised paper. Meanwhile, $\Phi_t$ and $\Phi_v$ continue to represent the textual and visual representation spaces, respectively. Accordingly, the corresponding symbols in Figure 1 (on page 5) have also been updated.

 

W2: Timid evaluation section.

Answer: To comprehensively validate the effectiveness of the proposed method, we have expanded the evaluation section (Section 5) by adding the following contents into the main text of the revised paper:

1) Two state-of-the-art prompt tuning methods as baselines (PromptSRC [1] and DePT [2]), in Section 5.2 (on page 8);

2) Experiments on another backbone model (i.e., ViT-B/32), in Section 5.2 (on page 8);

3) Visualization experiments to evaluate whether the proposed LogicAl-PT effectively mitigates cross-modal spurious correlations and enhances logical alignment between visual representation and text label, in Section 5.3 (on page 8-9);

4) Visual explanation to illustrate the necessity of the proposed textual logic alignment, in Section 5.4 (on page 9);

5) Ablation study on the sensitivity of hyper-parameters, in Section 5.4 (on page 10).

[1] Khattak, Muhammad Uzair, et al. "Self-regulating prompts: Foundational model adaptation without forgetting." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Zhang, Ji, et al. "Dept: Decoupled prompt tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

 

W3: The motivation is not clear since the authors employ PNS without explanation; It would be helpful to include a better explanation of what the "spurious correlation in vision-language models" is exactly.

Answer: To better clarify the motivation for using logic alignment to integrate cross-modal mitigation of spurious correlations and cross-modal feature alignment, we summarize four specific examples in Figure 4 (on page 14 of Appendix A) to address the following questions:

1) What the "spurious correlation in vision-language models" is exactly.

2) Apart from mitigation of spurious correlations, why corss-modal logic alignment (i.e., sufficiency and necessity) is also necessary for enhancing out-of-distribution generalization performance in vision-language models?

 

W4: Too much content from the existing papers and it could be removed by referring these papers.

Answer: Very good suggestion. We have removed the redundant parts in Section 3.2 (PNS) and Section 3.3 (PNS modeling). Additional details about PNS have been moved to Appendix C for further reference.

 

W5: Section 4.1: The NSC feature shown in Figure 1 is not explained by the authors themselves how to make use and take advantage of this.

Answer: Sorry, we missed providing the explanation for the NSC feature in Figure 1. "NSC" represents "necessary and sufficient cause''. Specifically, the NSC features in textual and visual modalities are given by $f([Q,CLASS])$ and $h(g(X))$, respectively. The interventions in textual and visual modalities are given by $f([\bar{Q},CLASS])$ and $\bar{h}(g(X))$, respectively.

At the training stage, "NSC" features are optimized by adjusting the learnable prompt $Q$ and filter $h$ using the proposed objective (10), as stated on line 290, page 6.

At the inference stage, predictions are made using the cosine similarity between textual and visual "NSC" features.

Advantage: By optimizing the textual and visual "NSC" features through the proposed objective (10), the optimal textual "NSC" features become logically aligned with both text labels and the optimal visual "NSC" features. Leveraging these features for prediction effectively eliminates spurious correlations and improves out-of-distribution generalization performance.

W6: Compared to CoOPood, it seems that the proposed method exploits PNS terms instead of mutual information to align cross-modal representations. I am wondering how effective the PNS term is in cross-modal mitigation of spurious correlation.

Answer:

1) Clarification on CoOPood: In CoOPood, the conditional mutual information is specifically employed to disentangle visual invariant features from visual spurious features, rather than facilitating cross-modal feature alignment. In contrast, CoOPood employs the assumption that the spurious correlation between visual spurious features and text label follows approximately uniform probability distributions to achieve cross-modal alignment. Therefore, the mitigation of spurious correlations and cross-modal alignment in CoOPood cannot be ensured if the assumption is not met. In comparison, our method does not rely on any assumptions about spurious correlations.

2) Effectiveness/Usefulness of the PNS term: To verify that the tuned models developed by our method LogicAl-PT exploit the necessary and sufficient features rather than spurious features, we sample some data instances to generate visual explanations for the selected model using Grad-CAM [3]. The commonly used Grad-CAM can produce a localization map which highlights the important regions in the input image that a deep learning model depends on for predicting the label.

The visualization results are displayed in Figure 2 (on page 9), and the detailed analysis on the visualization results is provided on line 410-431, page 8. In summary, visualization results demonstrate the proposed LogicAl-PT can effectively exploit the 'sufficient and necessary' features and mitigate the unstable spurious features, including non-causal spurious features, 'sufficient but not necessary' features and 'necessary but not sufficient' features. This explains why LogicAl-PT achieves superior out-of-distribution generalization performance, delivering more consistent results across diverse data distributions compared to its competitors.

[3] Selvaraju, Ramprasaath R., et al. "Grad-cam: Visual explanations from deep networks via gradient-based localization." Proceedings of the IEEE international conference on computer vision. 2017.

 

W7: Usefulness of the textual logic alignment: Why was it necessary to do textual logic alignment?

Answer:

1) Qualitative Analysis: Since the textual representations (corresponding to variable $\Phi_t$) are the class-wise mapping from the text labels, the sufficiency of variable $Y$ for variable $\Phi_t$ (i.e., $Y\Rightarrow \Phi_t$) is naturally guaranteed while the reverse $Y\Leftarrow \Phi_t$ is not ensured. In other words, textual representations ($\Phi_t$) must be necessary causes for variable $Y$, but they don't have to be sufficient causes for variable $Y$. Therefore, textual logic alignment is proposed to enhance the sufficiency of text representations ($\Phi_t$) for label $Y$. Accordingly, when cross-modal logic alignment (i.e., $\Phi_t \Leftrightarrow \Phi_v$) is achieved, combining textual logic alignment can mitigate the visual features that are not sufficient for variable $Y$.

2) Experimental Validation: To investigate the actual role that textual logic alignment serves, we visualize the features which is utilized by the model tuned without textual logic alignment (w/o TLA), i.e., $\beta=0$. In particular, when we set $\beta=0$, $\alpha$ is tuned to its optimal value, i.e., the cross-modal logic alignment ($\Phi_t \Leftrightarrow \Phi_v$) is enhanced. The visualization results are displayed in Figure 3 on page 10. From the visualization results, we find that adding textual logic alignment mitigates the visual features which are not sufficient for predicting $Y$. Therefore, the above qualitative analysis is validated by the visualization results.

Detailed visualization results and analysis are provided on line 483-513, page 10.

Thank you very much for your valuable feedback and suggestions, which significantly contributes to improving the quality of this paper. Detailed responses to your concerns and questions are listed below.

W1: Ablation studies about the hyper-parameter chosen should be provided..

Answer: We add experiments to evaluate the effects of two significant hyper-parameters in the proposed objective (i.e., $\alpha$ and $\beta$) on model performance. Since the results on other datasets present the similar tendency as on ImageNet-1K, we herein focus on ImageNet-1K, with ResNet-50 as backbone model. When evaluating the effect of $\alpha$, we fix $\beta=1.0$ . When evaluating the effect of $\alpha$, we fix $\alpha=20.0$. The experimental results are shown in the following two tables:

Analysis: When $\alpha=0.0$, model is tuned with only textual logic alignment; when $\beta=0.0$, models is tuned with only cross-modal logic alignment. We can find the performance of LogicAl-PT is more sensitive to the selection of $\alpha$ than the selection of $\beta$. To effectively mitigate spurious correlations in VLMs, careful tuning of $\alpha$ is essential. Regarding $\beta$, a small value is safer in practice, as a large $\beta$ may compromise the discriminative capability of the extracted features.

 

W2: Visual results about the improvement on spurious correlations should be provided.

Answer: We add visualization experiment in the updated paper.

Setup: For the purpose of verifying that the tuned models developed by our method LogicAl-PT exploit the necessary and sufficient features rather than spurious features, we sample some data instances to generate visual explanations for the selected model using Grad-CAM [1]. The commonly used Grad-CAM can produce a localization map which highlights the important regions in the input image that a deep learning model depends on for predicting the label.

Results: The detailed visualization results are displayed in Figure 2 (on page 9). The pivotal features employed by various prompt tuning methods are highlighted in red. The visualization results reveal that the proposed LogicAl-PT demonstrates three notable advantages over existing prompt-tuning methods: 1) LogicAl-PT can effectively eliminate the non-causal spurious features that are associated with the label (i.e., 'background' in WaterBird dataset and 'baby' in ImageNet-1K dataset). 2) LogicAl-PT can mitigate the 'sufficient but not necessary' features that demonstrate inconsistent presence across different data instances. For example, the shape of feet is a 'sufficient but not necessary' feature for classifying the picture of a bird as 'waterbird' or 'landbird' because its feet can retract or remain hidden when the bird is lying down or in flight. 3) LogicAl-PT can mitigate the 'necessary but not sufficient' features which can impact the classification performance when the distribution of these 'necessary but not sufficient' features varies. For example, the wings of birds are 'necessary but not sufficient' features for distinguishing 'waterbird' from 'landbird'.

Analysis: In summary, visualization results demonstrate the proposed LogicAl-PT can effectively exploit the 'sufficient and necessary' features and mitigate the unstable features, including non-causal spurious features, 'sufficient but not necessary' features and 'necessary but not sufficient' features. This explains why LogicAl-PT achieves superior out-of-distribution generalization performance, delivering more consistent results across diverse data distributions compared to its competitors.

[1] Selvaraju, Ramprasaath R., et al. "Grad-cam: Visual explanations from deep networks via gradient-based localization." Proceedings of the IEEE international conference on computer vision. 2017.

Thank you very much for your valuable feedback and suggestions, which significantly contributes to improving the quality of this paper. Detailed responses to your concerns and questions are listed below.

W1: The proposed method shows superior performance compared with other benchmarks. What is the computational efficiency compare to simpler methods?

Answer: Thank you for your appreciation of our work. Regarding computational efficiency, we analyze and compare the computational overhead of our method with several existing methods. The results are presented in the following table.

Analysis: We can see the the overall parameters and Floating Point Operations (FLOPS) of our LogicAl-PT are only 2.1% and 0.02% higher than those of CLIP and CoOp, respectively. Compared with the improvement in out-of-distribution generalization performance, our method LogicAl-PT exhibits considerable computational efficiency in terms of the number of parameters and FLOPS.