Leveraging One-To-Many Relationships in Multimodal Adversarial Defense for Robust Image-Text Retrieval

Thank you for taking the time to review our paper and your insights on it. We address the concerns raised below.

[W1, Q3] Limited evaluation datasets.

We humbly disagree. For the sake of coherence with the related work in adversarial robustness, we used the standard datasets Flickr30k and MSCOCO, as in Zhang et al.[A] and Lu et al.[B]. To the best of our knowledge, these datasets are considered big and varied enough to validate adversarial attack and defense methods. If you have any specific recommendation for additional datasets that could enhance the evaluation or provide broader insights, we would greatly appreciate your suggestions.

[W2] Other vision-language models

We humbly disagree, as using CLIP alone is the standard in the related work on adversarial robustness. While we acknowledge that experiments on more models would enhance our claims, our primary focus in this work was to conduct a thorough study on the effectiveness of diverse augmentation techniques in enhancing adversarial robustness in image retrieval tasks (ITR). To achieve this, we dedicated more computational resources to exploring data variations rather than model differences. This approach is consistent with the methodology of the related work TeCoA [C], which also focuses exclusively on CLIP-B/32 for detailed analysis. Following Zhang et al.[A] and Lu et al. [B], who studied adversarial attacks for ITR, we selected the CLIP-B/16 model, which is larger than CLIP-B/32. Although we are aware of the existence of other VL models such as ALBEF or BLIP, CLIP is still the most used backbone in vision-language research. Moreover, since their image-text matching is fundamentally similar, we believe that including more backbones would not lead to more additional findings other than a boost on the base performance.

[W3] Typo

Thank you so much for pointing this out. We have fixed it.

[W4, Q1] Clear framework diagram and visual results

Thank you for your constructive suggestion. We will include them in the camera ready.

[Q2] Comparison with other multimodal adversarial training

Since we are the pioneers in proposing multimodal adversarial training for ITR, there is no existing baseline method for direct comparison. Existing adversarial training methods for CLIP, such as TeCoA [C] (included in our experiments) and FARE [D], focus on defending against image-only attacks. Our method, MAT, is specifically designed for image-text multimodal attacks.

References:

[A] Zhang, Jiaming, Qi Yi, and Jitao Sang. "Towards adversarial attack on vision-language pre-training models." ACMMM 2022.

[B] Lu, Dong, et al. "Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models." CVPR 2023.

[C] Mao, Chengzhi, et al. "Understanding zero-shot adversarial robustness for large-scale models." ICLR 2023.

[D] Schlarmann, Christian, et al. "Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models." ICML 2024

Thank you for taking the time to review our paper and your insights on it. We address the concerns raised below.

[W1] Using only CLIP is a limitation

We humbly disagree, as using CLIP alone is the standard in the related work on adversarial robustness. While we acknowledge that experiments on more models would enhance our claims, our primary focus in this work was to conduct a thorough study on the effectiveness of diverse augmentation techniques in enhancing adversarial robustness in image retrieval tasks (ITR). To achieve this, we dedicated more computational resources to exploring data variations rather than model differences. This approach is consistent with the methodology of the related work TeCoA [C], which also focuses exclusively on CLIP-B/32 for detailed analysis. Following Zhang et al.[D] and Lu et al. [E], who studied adversarial attacks for ITR, we selected the CLIP-B/16 model, which is larger than CLIP-B/32. Although we are aware of the existence of other VL models such as ALBEF or BLIP, CLIP is still the most used backbone in vision-language research. Moreover, since their image-text matching is fundamentally similar, we believe that including more backbones would not lead to more additional findings other than a boost on the base performance.

[W2] Limitations in image augmentation

We do not consider this a weakness in our study, but rather, a novel finding. The fact that simple text augmentations provide higher adversarial robustness than image augmentations is particularly noteworthy and, to the best of our knowledge, has not been proved before. One of our contributions is studying the effectiveness of different augmentations and, for the first time, providing empirical results that show how the diversity and alignment of the augmentations are two key axis to achieve adversarial robustness.

The cause for lower performance in image augmentations is analyzed in the paper (Line 462-) as follows:

This is because generating image augmentations that do not lack diversity but also do not deviate significantly from the original data distribution is more challenging due to the high dimensionality of the image space. On the other hand, text modality is more amenable to augmentation, as the text space is lower-dimensional and more structured, making it easier to generate appropriate diversity in the augmented data points.

[W3] A more comprehensive theoretical analysis of why cross-modal augmentations specifically enhance ITR robustness.

We regret not providing a theoretical analysis in our work. However, we believe that our claims are adequately supported by the thorough empirical results presented.

Theoretically, data augmentation in adversarial training mitigates robust overfitting by enhancing generalization to unseen data [A], for example adversarially perturbed data. Our work follows this very same theory, when applied to more than one modality. Empirically, this has been validated in one-to-one unimodal tasks (i.e., image classification). For example, Wang et al. [B] demonstrated that synthetic images generated by diffusion models can improve adversarial robustness. However, as shown in our experiments, the unimodal image augmentations of the related work underperform in the multimodal task of image-text retrieval, where perturbations can occur in both modalities. While [B] highlighted the role of "better" diffusion models (in terms of image quality) in enhancing robustness, our work further explores this analysis to vision-language robustness. Specifically, we provide novel insights into what constitutes "better" augmentations in vision-language multi-modal adversarial training: augmentations that maintain high image-text alignment and ensure sufficient diversity of image-text pairs.

If you could provide more detailed suggestions on the theoretical analysis or specific aspects that we should prove, we would be happy to consider them.

[Q1] Would there be a challenges in adapting MA2T to BLIP or ALBEF?

Since BLIP and ALBEF share fundamentally similar image-text matching mechanisms with contrastive loss, akin to CLIP, there is no reason to think that MA2T would not be effective with these models.

Thank you for taking the time to review our paper and your insights on it. We address the concerns raised below.

[W1] Theory on why one-to-many augmentations improve adversarial robustness

Theoretically, data augmentation in adversarial training mitigates robust overfitting by enhancing generalization to unseen data [F], for example adversarially perturbed data. Our work follows this very same theory, when applied to more than one modality. Empirically, this has been validated in one-to-one unimodal tasks (i.e., image classification). For example, Wang et al. [A] demonstrated that synthetic images generated by diffusion models can improve adversarial robustness. However, as shown in our experiments, the unimodal image augmentations of the related work underperform in the multimodal task of image-text retrieval, where perturbations can occur in both modalities. While [A] highlighted the role of "better" diffusion models (in terms of image quality) in enhancing robustness, our work further explores this analysis to vision-language robustness. Specifically, we provide novel insights into what constitutes "better" augmentations in vision-language multi-modal adversarial training: augmentations that maintain high image-text alignment and ensure sufficient diversity of image-text pairs.

[W2] Multimodal training setup appears empirically driven without a theoretical basis.

We apologize for the lack of clarity in our explanation. Our multimodal training framework simply builds upon the standard adversarial training approach based on the min-max optimization principle [B], but extends to vision-language models. Here, the adversarial attack in image classification [B] maximizes the loss function, while the model minimizes it:

$min_{\theta} \ \rho(\theta), \ \text{where} \ \rho(\theta) = E_{(x,y) \sim \mathcal{D}} \left[ \max_{\delta \in \mathcal{S}} L(\theta, x + \delta, y) \right].$

Extending this formulation to vision-language models, we define the general objective as follows:

$min_{\theta} \ \rho(\theta), \ \text{where} \ \rho(\theta) = {E}_{(x,y) \sim \mathcal{D}} \left[ \max L(\theta, x + \delta_x, t+ \delta_y) \right].$

Our Multimodal Adversarial Training (MAT) framework addresses the inner maximization problem using the approximation detailed in Section 3.2. Specifically, our method adopts a simple yet effective strategy: first updating the text modality and then the image modality. While this sequential approach provides an effective baseline, further improvements could be explored by iteratively updating image and text modalities to better maximize the loss function. We leave this refinement for future work.

[W3,W4] Regarding the use of larger models and other model architecture.

We humbly disagree, as using CLIP alone is the standard in the related work on adversarial robustness. While we acknowledge that experiments on more models would enhance our claims, our primary focus in this work was to conduct a thorough study on the effectiveness of diverse augmentation techniques in enhancing adversarial robustness in image retrieval tasks (ITR). To achieve this, we dedicated more computational resources to exploring data variations rather than model differences. This approach is consistent with the methodology of the related work TeCoA [C], which also focuses exclusively on CLIP-B/32 for detailed analysis. Following Zhang et al.[D] and Lu et al. [E], who studied adversarial attacks for ITR, we selected the CLIP-B/16 model, which is larger than CLIP-B/32. Although we are aware of the existence of other VL models such as ALBEF or BLIP, CLIP is still the most used backbone in vision-language research. Moreover, since their image-text matching is fundamentally similar, we believe that including more backbones would not lead to more additional findings other than a boost on the base performance.

Thank you for taking the time to review our paper and your insights on it. We address the concerns raised below.

[W1-1] Limited evaluation datasets.

Flickr30k and MSCOCO are the standard image-text retrieval (ITR) datasets used for evaluation of adversarial attacks. This the case of the related work of Zhang et al.[A] and Lu et al.[B], which we follow for consistency. These datasets contain a variety of scenes that is wide enough to evaluate our method.

While remote sensing datasets are indeed used in some ITR scenarios, we do not see them essential for proving the validity of our method, as they are tied to very specific applications such as environmental monitoring and agriculture. Furthermore, the image/text augmentations that can be applied to remote sensing data are very limited, and standard text2image and image2text methods are not tuned to that specific domain.

To summarize, remote sensing datasets are not commonly used in adversarial attack research, and we do not consider them essential to evaluate our method. If you have any recommendations that could enhance the evaluation or provide a broader perspective in our adversarial defense scenario, we would greatly appreciate your suggestions and the reason why.

[W1-2] Introduced method lacks sufficient theoretical justification.

1. Justification of multimodal adversarial training

Our multimodal training framework simply builds upon the standard adversarial training approach based on the min-max optimization principle [C], but extends to vision-language models. Here, the adversarial attack in image classification [C] maximizes the loss function, while the model minimizes it:

$min_{\theta} \ \rho(\theta), \ \text{where} \ \rho(\theta) = E_{(x,y) \sim \mathcal{D}} \left[ \max_{\delta \in \mathcal{S}} L(\theta, x + \delta, y) \right].$

Extending this formulation to vision-language models, we define the general objective as follows:

$min_{\theta} \ \rho(\theta), \ \text{where} \ \rho(\theta) = {E}_{(x,y) \sim \mathcal{D}} \left[ \max L(\theta, x + \delta_x, t+ \delta_y) \right].$

Our Multimodal Adversarial Training (MAT) framework addresses the inner maximization problem using the approximation detailed in Section 3.2. Specifically, our method adopts a simple yet effective strategy: first updating the text modality and then the image modality. While this sequential approach provides an effective baseline, further improvements could be explored by iteratively updating image and text modalities to better maximize the loss function. We leave this refinement for future work.

2. Justification of augmentation for robustness

Theoretically, data augmentation in adversarial training mitigates robust overfitting by enhancing generalization to unseen data [D], for example adversarially perturbed data. Our work follows this very same theory, when applied to more than one modality. Empirically, this has been validated in one-to-one unimodal tasks (i.e., image classification). For example, Wang et al. [E] demonstrated that synthetic images generated by diffusion models can improve adversarial robustness. However, as shown in our experiments, the unimodal image augmentations of the related work underperform in the multimodal task of image-text retrieval, where perturbations can occur in both modalities. While [E] highlighted the role of "better" diffusion models (in terms of image quality) in enhancing robustness, our work further explores this analysis to vision-language robustness. Specifically, we provide novel insights into what constitutes "better" augmentations in vision-language multi-modal adversarial training: augmentations that maintain high image-text alignment and ensure sufficient diversity of image-text pairs.

Dear Reviewers,

Thank you once again for taking the time to review our paper. We greatly value your insightful comments and sincerely appreciate your efforts.

We kindly request that you review our replies. Your feedback is invaluable in addressing concerns and improving our work.

If anything remains unclear, please don’t hesitate to reach out. We would be glad to provide further clarification.

Thank you sincerely, Authors