Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition

We sincerely appreciate your valuable comments and recognition of our work. Below are our responses to your concerns.

W1: The optimization efficiency of the proposed method

In Table 5 of the main paper, we demonstrate the optimization efficiency of our method and the effectiveness of the selective optimization trick. As shown, with the same number of epochs, applying selective optimization allows SAS to train on ImageNet with only approximately 10 additional minutes compared to the baselines. Meanwhile, our method does not introduce any extra computations during testing.

Here, we present the optimization time costs for SAS and selective optimization across other datasets. The time is determined as the runtime of the training script, which is based on the implementation of CoOp [A].

As shown in the table above, for most datasets, the integration of SAS only increases the training time by approximately 3 to 5 minutes, while selective optimization further reduces this time to a negligible amount. In fact, the selective optimization trick is proposed to address large-scale datasets, such as ImageNet, which contains 1000 categories. For regular datasets (~100 categories), the time consumption of SAS is fully acceptable. In the revised paper, we have provided a detailed analysis of the optimization efficiency of our method in Section B.14 of the Appendix (highlighted in blue).

W2: More visualization examples of the proposed method

In Section 5 of the main paper, we visualize the saliency map of example images with and without SAS. For the chocolate cake in (a), SAS succesfully mitigates VLMs' reliance on typical spurious attributes, such as plates and utensils. In (b), as mentioned by the reviewer, for the personal laptop, SAS significantly reduces attention to likely its most common acommpanying object, i.e., the mouse. In (c), for street sign recognition, SAS removes the model's bias towards the road, which is particularly beneficial for autonomous driving. It is worth mentioning that the saliency maps here are used for a qualitative understanding of the method, as they are not perfect indicators of localization.

However, we appreciate the reviewer's suggestion that more visualization examples are expected to demonstrate the effectiveness of SAS. We have revised the paper and included six additional examples in Section B.12 of the Appendix (highlighted in blue). As shown in Figure 9, SAS consistently mitigates the impact of spurious cues across various categories. For instance, for the tree frog, SAS reduces the VLM's dependence on tree branches, while for the airliner, the model no longer focuses on the sky or clouds, which are typicall spuriously correlated with airliners. Similarly, for the polar bear, where the snow-covered ground is a common spurious attribute, SAS effectively alleviates the model's bias towards it.

Reference

[A] Learning to Prompt for Vision-Language Models, IJCV2022

[B] Conditional Prompt Learning for Vision-Language Models, CVPR2022

[C] Self-regulating Prompts: Foundational Model Adaptation without Forgetting, ICCV2023

W3: Performance gains from the proposed method or data augmentation

Our proposed method, SAS enhances model performance by teaching it to distinguish between main objects and spurious attributes through the construction of pseudo categories that feature the latter, i.e., negative samples as mentioned by the reviewer. Therefore, the performance gains should stem from increased model robustness to spurious cues rather than from the vanilla data augmentation. In Section 4.2 of the main paper, we conduct a simple ablation study to verify the contribution of spurious attributes on our method by adjusting $\gamma$.

Here, as suggested by the reviewer, we further conduct an ablation study to compare the performance improvements brought by extra data and our proposed method to prove this point. Specifically, along with our proposed method, we design two baselines. In the first baseline, we involve additional data directly from the original dataset featuring the main objects, extending the training data from 16 shots to 32 shots (32-shot main). In the second baseline, we consider additional data generated by pseudo categories that are the same as the main categories, i.e., positive samples as mentioned by the reviewer (16-shot main + 16-shot positive). In contrast to these baselines, our approach creates pseudo categories based on spurious attributes (16-shot + 16-shot negative). For fairness, we ensure that the amount of training data is identical between the two baselines and our approach. We consider three typical methods for comparison, including CoCoOp [B], PromptSRC [C], and MaPLe [D], and evaluate them on the base-to-new generalization task as illustrated in the main paper. All results are averaged across 11 datasets.

As shown in the table above, generating additional data using spurious attributes, i.e., negative samples, significantly outperforms the vanilla data augmentation, i.e., positive samples (76.36% vs 73.53%). Moreover, our proposed method even surpasses the performance of the 32-shot main (76.36% vs 74.34%). It is important to note that this comparison is not entirely fair to our method, as the latter relies on more labeled data from the original training set. This further suggests that the performance gains are primarily driven by the model's enhanced robustness to spurious attributes, rather than merely the augmented data. We have revised the paper and included this ablation experiment in Section B.11 of the Appendix (highlighted in blue).

Q1: More visualization examples

Please refer to W2 above.

Q2: Ablation study on performnace gains

Please refer to W3 above.

Reference

[D] MaPLe: Multi-modal Prompt Learning, CVPR2023

Dear reviewer rex7,

We sincerely appreciate your insightful comments and recognition of our work. We believe your feedback has been very helpful in enhancing the completeness and soundness of the paper. We have carefully addressed each concern and question with detailed responses and revisions. As the discussion deadline nears, we would like to confirm that if our revisions fully address your concerns. Please feel free to share any further questions or suggestions.

Best,

Authors

We greatly appreciate your insightful comments and your recognition of our work. Here are our responses to your concerns.

W1: The comparison fairness to selected baselines

In Section 4 of the main paper, the baselines we use include prompt tuning, e.g., CoCoOp [A] and KgCoOp [B], attribute-based methods, e.g., ArGue [C] and CPL [D], as well as adapters [E], all of which are few-shot approaches that require a few labeled training data. Our proposed SAS, as a plug-and-play method, is consistent with them and uses the same training data. The only zero-shot baseline we compare against is zero-shot CLIP, as illustrated in Figure 3 of the main paper, which serves to indicate the vanilla performance of CLIP.

W2: Evaluation of the proposed method on two new approaches

We thank the reviewer for the valuable suggestion. Here, as mentioned by the reviewer, we evaluate our method on the two recently proposed works. Specifically, for MMA [F], we train the newly introduced adapters in the deep layers that bridge the text and image representations, following their setting and implementation. For DMN [G], we optimize its memory projection functions and incorporate both the static and dynamic memory networks, which is the strongest variant according to their paper. We select the base-to-new generalization task, as illustrated in Section 4 of our main paper, and record the new category accurarcy, which directly reflects the generalization performance.

As shown in the table above, SAS consistently improves performance on both methods, demonstrating its complementarity to the suggested approaches. In the revised paper, we have cited the mentioned works and included this experiment in Section B.16 of the Appendix.

Reference

[A] Conditional Prompt Learning for Vision-Language Models, CVPR2022

[B] Visual-Language Prompt Tuning with Knowledge-guided Context Optimization, CVPR2023

[C] ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models, CVPR2024

[D] Concept-Guided Prompt Learning for Generalization in Vision-Language Models, AAAI2024

[E] CLIP-Adapter: Better Vision-Language Models with Feature Adapters, ArXiv

[F] MMA: Multi-Modal Adapter for Vision-Language Models, CVPR2024

[G] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models, CVPR2024

Dear reviewer Yv7Q,

We would like to once again thank you for your insightful feedback and recognition of our work. We believe your comments have been instrumental in improving the comprehensiveness and clarity of our paper. We have presented responses and revisions addressing each listed concern and question. As the discussion deadline approaches, we would like to know if our revisions effectively address your concerns. Please let us know if you have any further questions or suggestions.

Best,

Authors

Dear reviewer Yv7Q, ​

​We sincerely appreciate your response and recognition of our work. ​

​Thank you once again for your time on reviewing this paper. ​

​Best regards, ​

​Authors

We are grateful for your valuable comments and your acknowledgment of our work. Below are our responses to your concerns.

W1: The computational efficiency of the proposed method

As suggested by the reviewer, here we provide a detailed overview of the computation and time costs of the proposed method.

• The cost of training. Since the construction of pseudo categories augments the original data, the training process may need more computation to converge. However, we want to emphasize that our method introduces additional computations only during training, without adding any extra computations during testing. Furthermore, to alleviate the training computation, in Section 4 of the main paper, we introduce selective optimization trick. Instead of optimizing every main category, we focus on a small subset of categories that seriously suffer from spurious correlations. As demonstrated in Table 5 of the main paper, this approach significantly reduces training time while preserving the majority of the accuracy gains.

• The cost of diffusion generation. Here, we provide the estimated inference time required to construct pseudo categories through Stable Diffusion for each dataset. Please refer to Section A.4 of the Appendix for the detailed settings and prompts of the diffusion.

Note that due to time constraints, the inference time provided above is an approximation based on the time required for each generation. As shown, the total inference time is proportional to the size of the dataset, particularly the number of categories involved. For most datasets, the inference time is under half an hour, and the entire inference process can be completed within half a day. It is important to note that this is a one-time operation, and no additional inference is needed during subsequent training.

• The cost of GPT prompting. In our method, a key step is identifying the spurious attributes within each category, which we accomplish by prompting MLLMs, i.e., GPT. As suggested by the reviewer, we provide the time cost of this process along with a thorough analysis. Specifically, to enhance efficiency, we employ batch inference as implemented in [A], where multiple queries can be processed concurrently, which significantly reduces the inference time for GPT.

As shown in the table above, the GPT inference time for most datasets is under 10 minutes. The complete inference process takes approximately three hours, which is also a one-time operation that does not need to be repeated thereafter. It is worth noting that upon obtaining the responses, we need to perform post-processing such as filtering and selection to determine valid attributes, as detailed in Section A.3 of the Appendix, which may require additional time.

We have included the above statistics and analysis regarding computation and time costs in Section B.14 of the revised paper. (highlighted in blue)

Reference

[A] Visual Classification via Description from Large Language Models, ICLR2023

We would like to sincerely express our gratitude once again for your response and positive feedback on our work. Below, we have provided further responses to your question and concern.

Q: Is SAS deployed only in training?

The short answer is yes. The essence of SAS lies in incorporating an auxiliary objective during training, enabling the model to distinguish between main objects and spurious attributes, thus enhancing its robustness to spurious features. Unlike adapters or prompt tuning, SAS does not introduce additional learnable parameters. Therefore, during testing, the model remains consistent with the original, without adding computational overhead.

W: The training overhead by incorporating GPT and SD

We understand your concerns. Here, we further elaborate and clarify the efficiency of model training and the inference of GPT or SD.

Regarding model training, in Section B.14 of the revised paper, we have included the training time statistics of SAS on additional datasets beyond ImageNet as shown in Table 14. For your convenience, we have directly displayed the table below.

As shown in the table above, for most datasets, the integration of SAS only increases the training time by approximately 3 to 5 minutes, while selective optimization further reduces this time to a negligible amount. In fact, the selective optimization trick is proposed to address large-scale datasets, such as ImageNet, which contains 1000 categories. For regular datasets (~100 categories), the time consumption of SAS is fully acceptable.

Regarding inference of GPT and SD, for the former, we would like to emphasize that, compared to conventional works that identify visual attributes via manual labeling [J, K] as introduced in Section 2, directly obtaining spurious attributes via GPT is a much more efficient and cost-effective method. Moreover, following the implementation of [A], we have limited the prompting time for most datasets to within 10 minutes.

For the latter, since we employ standard diffusion hyperparameters following [L], we believe that by exploring more settings, such as reducing the number of diffusion steps or deploying different schedulers, we may achieve a better trade-off between accuracy and efficiency. Due to time constraints, we leave this as future work. Nonetheless, we want to clarify that, for most datasets, the SD inference time can still be kept under half an hour. It is worth mentioning that throughout the experiment, we use a single NVIDIA 4090 GPU as described in Section 4, and we expect that utilizing more or more powerful GPUs would further reduce the required time.

Thank you again for your recognition of our work. We believe your comments have greatly improved the comprehensiveness and soundness of the paper. We hope that our further responses may address your remaining concerns.

Reference

[J] How to discover spurious features in deep learning?, ICLR2021

[K] Leveraging sparse linear layers for debuggable deep networks, ICML2021

[L] SuS-X: Training-Free Name-Only Transfer of Vision-Language Models, ICCV2023

To better address the reviewer's efficiency concern, here we update the response with a simple ablation study on the efficiency of diffusion inference. Specifically, we vary the number of diffusion steps, which is the key hyperparameter determining the inference time cost. Intuitively, fewer steps are more efficient yet yield lower image quality, while more steps ensure image fidelity but require more computation. We select CoCoOp as the baseline and record the new category accuracy on base-to-new generalization. Due to time constraints, here we provide the results for four representative datasets.

As shown in the table above, by default, we use 100 steps throughout the paper as described in Section A.4, which requires an average of 31 minutes to generate images per dataset. Here we try fewer steps, such as 50, and observe that the time required for diffusion nearly halves (31min -> 16min) with minimal degradation in performance (67.09 -> 66.97). However, while the number of steps is further reduced to 25, there is a dramatic performance drop (66.97 -> 66.05), possibly due to the decline in image quality. This suggests we may safely adjust the number of steps from 100 to 50, which halves the required time with minimal accuracy loss, significantly improving the efficiency of SAS.

This simple experiment highlights SAS's significant potential for improved efficiency. With advancements in diffusion sampling strategies, we believe that the computational overhead introduced by diffusion will soon no longer be a concern. We have added this ablation study in Section B.17 of the revised paper. Further attempts with different settings, such as various schedulers or checkpoints, are left as future work.

Please check the review carefully. Reference [A-C] is provided at the end of Weaknesses 5 . Reference [D] is provided at the end of Weaknesses 6 . Reference [E] is provided at the end of Weaknesses 7 .

Thank you for the insightful comments. The reference section for [A]-[E] is not displayed in the review. Could you update that part?

We would like to thank you once again for your constructive comments and questions. Following are our responses to your concerns.

W1: Performance gains from extra data or the proposed method

SAS improves model generalization by training it to differentiate between main objects and spurious attributes through the creation of pseudo categories that feature the latter. Therefore, the performance gains should result from enhanced model robustness to spurious concepts instead of the vanilla constructed data. In Section 4.2 of our main paper, we perform an ablation study to confirm the contribution of spurious attributes on our method by adjusting $\gamma$.

Here, as suggested by the reviewer, we further compare the performance gains brought by extra data and our proposed method to validate this point. Specifically, in addition to the proposed method, we design two baselines. In the first baseline, we consider additional data directly from the original dataset featuring the main objects, where we extend the training data from 16 shots to 32 shots (32-shot main). In the second baseline, we involve additional data generated by pseudo categories, where instead of featuring spurious attributes, these pseudo categories are the same as the main categories, i.e., vanilla constructed data (16-shot main + 16-shot pseudo main). In contrast to the first two baselines, our approach creates pseudo categories based on spurious attributes (16-shot main + 16-shot pseudo spurious). For fairness, we ensure that the amount of training data is identical between the two baselines and our approach. We select three typical methods for comparison, including CoCoOp [A], MaPLe [B], and PromptSRC [C], and evaluate them on the base-to-new generalization task. All results are averaged across 11 datasets, as illustrated in the main paper.

As shown in the table above, generating additional data using spurious attributes significantly outperforms vanilla constructed data for main categories (76.36% vs 73.53%). Furthermore, our proposed method even exceeds the performance of the 32-shot main (76.36% vs 74.34%). It is important to note that this comparison is not entirely fair for our method, as the latter relies on more labeled data from the original training set. This further suggests that the performance gains are primarily driven by the model's enhanced robustness to spurious attributes, rather than merely the increased training data. We have revised the paper and included this ablation experiment in Section B.11 of the Appendix (highlighted in blue).

Reference

[A] Conditional Prompt Learning for Vision-Language Models, CVPR2022

[B] MaPLe: Multi-modal Prompt Learning, CVPR2023

[C] Self-regulating Prompts: Foundational Model Adaptation without Forgetting, ICCV2023

Dear reviewer bxnj,

We sincerely appreciate the effort you dedicated to reviewing our paper. Your feedback has greatly enhanced the solidity and clarity of our work. We have provided thorough responses and revisions based on your concerns and suggestions. As the discussion end date approaches, we are eager to know if our revisions have addressed your comments, and we welcome any further questions or suggestions.

Best,

Authors