One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models

Thank you for dedicating your time and effort to reviewing our paper. We are deeply encouraged by your positive comments and recognition of our work. Below, we provide point-by-point responses to address your concerns.

[Q1] The time budget of generator training.

Yes! Training a generator can be time-consuming and is a common issue in the whole research field of generative attacks [1,2,3]. However, the generative paradigm can bring significant performance improvement and once the generator finishes training, the universal perturbation can be directly applied to any image-text pair without requiring any sample-specific optimization.

[Q2] Positive sample selection.

We are deeply sorry for the misunderstandings that our paper may cause you. Please kindly allow us to classify the positive sample selection more clearly. Taking image perturbation as an example, we first randomly sample a batch of texts as candidates, where we then choose texts with the farthest distance specific to the current input adversarial image as positive samples. i.e., our framework has taken the current input pair into consideration when selecting items for contrastive loss. The ablation study in Section 4.4 of the main body has verified the effectiveness of this farthest selection strategy.

We will highlight this design to make it more clear. Also, we will include your constructive suggestion as a potential direction for a better selection strategy in the Discussion section of the Appendix.

[Q3] The effect of set-level augmentation.

Yes. As described in the method part, we are motivated by the significant gains introduced by SGA's augmentation technique [4] and hence integrate it into the proposed framework to enhance the universal perturbation. The underlying mechanism is to leverage the many-to-many relationships between images and texts by introducing multiple augmented images to provide set-level diverse guidance, further improving the optimization direction for effective guidance. Following your suggestion, we conduct experiments where this augmentation strategy was removed from our method.

Table A. Comparison of C-PGC and its variant that cancels the data augmentation on Flickr30.

Table A demonstrates that the augmentation mechanism indeed provides an improvement in ASR, proving the rationality of set-level augmentation. We will include these results in Appendix E to complement our ablation studies.

Thank you again for your thoughtful suggestion! If you have any further concerns, feel free to reach out to us. :)

[1] Feng W, Xu N, Zhang T, et al. Dynamic generative targeted attacks with pattern injection. In CVPR, 2023.

[2] Poursaeed O, Katsman I, Gao B, et al. Generative adversarial perturbations. In CVPR, 2018.

[3] Gao H, Zhang H, Wang J, et al. NUAT-GAN: Generating Black-box Natural Universal Adversarial Triggers for Text Classifiers Using Generative Adversarial Networks. In IEEE TIFS, 2024.

[4] Lu D, Wang Z, Wang T, et al. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. In ICCV, 2023.

We sincerely express our gratitude for dedicating your valuable time to providing insightful suggestions that can enhance our paper. Your praise regarding our writing, methodology, experiments, and contributions has greatly encouraged and motivated us. Our detailed responses to all of your concerns are presented below.

[Q1] Use of Contrastive Loss.

We sincerely apologize for not adequately explaining the rationale behind employing the contrastive learning mechanism in our paper. We then provide a detailed explanation of the underlying principles of using contrastive loss, supported by more experimental results.

1. Motivation. It is widely acknowledged that contrastive learning serves as a powerful and foundational tool for modality alignment in VLP models, establishing a nearly point-to-point relationship between image and text features. Our core idea stems from the general principle: "It's easier to tear down than to build up." Since contrastive learning can effectively establish robust and precise alignment, leveraging the same technique to disrupt these established alignments is expected to yield effective attack performance.

2. Rationale. Taking image attack as an example, the principle behind our contrastive learning-based attack can be understood from two perspectives. (1) Leverage the originally matched texts as negative samples to push the aligned image-text pair apart. This broadly corresponds to the common objective of untargeted attacks that you have kindly mentioned. (2) Additionally, our contrastive paradigm introduces additional benefits by using dissimilar texts as positive samples to pull the adversarial image $v^{adv}$ out of its original subspace and relocate it to an incorrect feature area. By simultaneously harnessing the collaborative effects of push (negative samples) and pull (positive samples), the proposed contrastive framework effectively destroys the modal alignment and achieves exceptional attack performance, which has been validated by comprehensive experimental results.

Besides, we also explore several potential alternative loss functions that more directly align with the common untargeted attack, including maximizing the negative cosine similarity $\mathcal{L}\_{Cos}$ or MSE distance $\mathcal{L}\_{MSE}$ between the features of matched image-text pairs.

Table A. ASR results of different loss functions when the surrogate is ALBEF.

Recall that $\mathcal{L}\_{CL}$ and $\mathcal{L}\_{Dis}$ denote the proposed contrastive loss and the unimodal loss term respectively. As observed, the use of $\mathcal{L}\_{CL}$ consistently brings significant ASR improvements, verifying the rationality and superiority of contrastive loss.

We will include these analyses in a new section of the Appendix to supplement a more thorough understanding of the rationale behind the choice of contrastive loss.
Thank you once again for your inspiring suggestion!

Dear Reviewer U4zG,

Thank you once again for dedicating your valuable time to reviewing our paper and providing constructive comments!

As the end of the discussion period approaches, we kindly ask if our responses have satisfactorily addressed your concerns. Your feedback would be greatly appreciated, and we would be delighted to engage in further discussions with you.

Sincerely,

The Authors

Thanks for your inspiring acknowledgment of our response!

We apologize for not fully understanding the "negative direction of the contrastive loss, instead of manually selecting negative samples to construct the contrastive loss?" meant, since it is generally necessary to obtain both positive and negative samples for contrastive loss formulation. Does that indicate our proposed strategy in the response in [Q2]? The experiments are close to end and we will upload the results very soon.

We would appreciate it if you could provide more details so that we can more accurately address this matter. :)

We are very sorry that we have not made the attempt to maximize the contrastive loss, since maximizing the contrastive loss seems to be contrary to our attack goal.

Could you please tell us more details about the purpose of this experiment? Also, we wonder if the supplemented experiments to [Q2] in Author Response (Part IV) have adequately solved your issue.

Thanks for your response!

Dear U4zG,

We have carefully supplemented responses to your further questions and provided experiments following your suggestion. We look forward to your reply and welcome discussion on any questions regarding our paper and response.

Best regards,

Authors

Dear U4zG,

We apologize for still not fully understanding the aim of "maximum the contrastive loss". Therefore, we clarify the following facts as potential answers to your raised question. Please kindly review them to see if this response has solved your concern.

1. We highlight that it requires both positive and negative samples to formulate the contrastive learning paradigm. Generally, positive samples are defined as the targets that the anchor aims to move closer to, while negative samples are those that it seeks to move away from [1,2,3].
2. Maximizing the established contrastive loss violates our attack goal, since this operation actually pushes the adversarial image closer to the original matched texts (negative samples) and pulls the adversarial image away from the dissimilar texts (positive samples). This also fails to obey the definition of contrastive learning, since the anchor sample is supposed to get away from the negative samples and closer to the positive samples.
3. We provide results where we directly maximize the cosine similarity and MSE between the adversarial sample and its paired data in Table A of Author Response (Part I). Also, we have provided experiments using synthetic samples based on adversarial learning in Table B of Author Response (Part IV).
4. Do you actually indicate maximizing the distance between the anchor sample and the positive samples?

We sincerely hope that you can tell us whether the above responses have adequately solved your concerns or provide more details about your question. We're glad to have further discussion with you. If you are satisfied with our responses, we would greatly appreciate it if you could kindly consider raising your score accordingly. :)

We're looking forward to your reply.

Sincerely,

Authors

[1] Khosla P, Teterwak P, Wang C, et al. Supervised contrastive learning. In NIPS, 2020.

[2] Li J, Selvaraju R, Gotmare A, et al. Align before fuse: Vision and language representation learning with momentum distillation. In NIPS, 2021.

[3] Yang J, Duan J, Tran S, et al. Vision-language pre-training with triple contrastive learning. In CVPR, 2022.

Dear Reviewer U4zG,

Since the discussion phase will last nearly only one hour, we kindly send this message as the last gentle reminder of our responses.

As we have addressed your major concerns (Q1, Q2, Q3, Q4) during the rebuttal, we would like to kindly ask if you could consider raising your score. :)

Best regards,

Authors

We would like to express our gratitude for your valuable suggestions and positive feedback on our paper. Our detailed responses are provided below.

[Q1] Semantic similarity between adversarial text and its original text.

Very inspiring question! We have carried deeper investigation into this issue and please refer to [Q1] of the Common Concerns for detailed answers.

[Q2] The way to calculate the ASR.

We totally align our evaluation protocol with prior adversarial attacks on VLP models [1,2,3], where the ASR is calculated as the proportion of successful adversarial samples within the originally correctly predicted pairs. We will supplement this introduction in the experimental part. Thanks for your kind reminder!

[Q3] Employment of special characters.

Interesting suggestion! We employ an optimization-based strategy to obtain the text perturbation, where we iteratively update the generator to output more effective text embeddings. Actually, the vocabulary used for mapping text embeddings back to discrete words has included these special tokens such as # and *. In other words, the final adversarial word is assigned based on the optimized text embeddings and has the possibility to be a special token.

To further investigate the impact of these special tokens, we conduct experiments where we directly adopt # and * as adversarial tokens to evaluate their attack results:

Table A. ASR of C-PGC and its variants using special characters as the adversarial word.

Table B. Comparison of C-PGC and its variants using special characters as the adversarial word regarding the BERT score between clean and adversarial texts. We adopt 5,000 texts from Flickr30 to calculate these results.

Tab. A and Tab. B display that our optimization-based strategy exhibits both superior attack performance and higher semantic similarity. Note that the use of special tokens as adversarial words can also be more conspicuous than the natural language, which might compromise the attack stealthiness and increase the likelihood of being detected by human observers or automated filtering systems.

We will supplement these discussions into the revision to promote future studies on the selection of the universal adversarial word. Thanks again for this interesting and inspiring question!

[Q4] Details about the perturbation generator.

Our generator $G_w(\cdot)$ adopts a decoder-based CNN architecture with cross-attention layers to integrate cross-modal knowledge. It upsamples a low-dimensional fixed noise vector $z_v$ into high-dimensional features. For image perturbations, $G_w(\cdot)$ directly outputs the image perturbations. For text attacks, it generates word embeddings that are subsequently flattened and mapped back to the discrete word domain. i.e., the fundamental design principle is analogous to that of generators employed in prior studies, such as GAP [4] and Nuat-GAN [5], with differences primarily in structural composition rather than conceptual innovations.

[Q5] Reorganization of contributions.

Thank you for your thoughtful suggestions. Our primary contributions are as follows:

1. We design a cross-modal conditioned perturbation generator to produce effective UAPs for both image and text modalities.
2. We propose the first malicious contrastive paradigm tailored for multimodal adversarial attacks. Firstly, we devise selection strategies (e.g., the farthest distance selection) to obtain positive and negative samples based on the attack objective. Then, we leverage these meticulously constructed samples to contrastively train the perturbation generator under both unimodal and multimodal guidance.

We will more clearly reorganize and highlight the innovations and contributions in the Introduction sections of the paper.

Thank you again for your valuable feedback! If you have any further questions or suggestions, please don’t hesitate to tell us. :)

[1] Zhang J, Yi Q, Sang J. Towards adversarial attack on vision-language pre-training models. In ACM MM, 2022.

[2] Lu D, Wang Z, Wang T, et al. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. In ICCV, 2023.

[3] Wang H, Dong K, Zhu Z, et al. Transferable multimodal attack on vision-language pre-training models. In S&P, 2024.

[4] Poursaeed O, Katsman I, Gao B, et al. Generative adversarial perturbations. In CVPR, 2018.

[5] Gao H, Zhang H, Wang J, et al. NUAT-GAN: Generating Black-box Natural Universal Adversarial Triggers for Text Classifiers Using Generative Adversarial Networks. In IEEE TIFS, 2024.

We sincerely thank you for your precious time and effort in providing a wealth of suggestions to enhance the quality of our paper. We have carefully read all the comments and provide detailed point-by-point responses as follows. Hopefully, we can adequately address your concerns.

[Q1.1] Fully utilize the characteristics of V+L scenario.

Thanks for your constructive advice. This claim aims to highlight that in contrast to the unimodal scenarios, we fully leverage the unique characteristics of multimodal scenarios to enhance the modeling of a universal perturbation that can generalize to various samples and V+L tasks. Specifically, we elaborate on this claim from the following main aspects.

1. Cross-Modal Conditions in Perturbation Generator:
   We design a perturbation generator with cross-modal conditions to benefit from cross-modal knowledge. In contrast, generators in unimodal generative attacks rely solely on information from a single modality.
2. Malicious Multimodal Contrastive Learning Paradigm:
   We formulate a multimodal contrastive learning paradigm for multimodal adversarial attacks. In comparison, unimodal attacks typically focus on interactions within a single modality to generate adversarial samples.
3. Joint Perturbations Across Modalities:
   Similar to prior sample-specific adversarial attacks on VLP models [1, 2], our approach applies perturbations to both image and text modals. Conversely, unimodal attacks only allow the perturbation of a single modality.

Extensive experiments demonstrate the effectiveness of these techniques, revealing that our framework utilizes the characteristics of V+L scenarios to achieve superior performance.

Besides, while investigating task-level V+L characteristics to enhance attacks for specific tasks is a promising direction, we highlight that the primary focus of our paper lies in leveraging the shared and joint characteristics of Vision-Language scenarios to present a universal and versatile UAP that can effectively generalize to diverse downstream V+L tasks.
Thanks again for your thoughtful comment. We will supplement these analyses in Appendix I of our revision.

[Q1.2] Comparison with PGD-based UAP algorithm.

Thanks for this insightful suggestion! Following your advice, we supplement the comparison between PGD-based UAP and GAP as follows：

Table A. ASR results of GAP and UAP learned by DeepFool and PGD respectively.

The results indicate that the UAP learned through PGD indeed obtains better results compared to the DeepFool-based one. Nevertheless, GAP still achieves significantly higher fooling rates, validating the necessity of employing a generator. Furthermore, numerous studies on generative adversarial attacks [3,4,5] have also similarly highlighted the superior performance of generative methods over non-generative ones, thereby supplementing strong empirical support for our claim.

We will include these experimental results and cite the related literature in the Introduction section to more confidently affirm our conclusion: "The generator-based approach GAP consistently achieves superior ASR compared to UAP." Thanks again for your valuable suggestion that helps us improve our paper.

[Q3.2] Comparison of different data augmentations.

Thank you for this valuable suggestion. Initially, we were motivated by the impressive performance of SGA [2] and hence integrated their image augmentation into our framework to enhance the attack. Following your advice, we reproduce your suggested augmentation techniques while keeping other settings unchanged，i.e., C-PGC$\_{ScMix}$ and C-PGC$\_{Admix}$. Experimental results are presented as follows:

Table D. Attack performance under different data augmentation strategies.

It can be observed that C-PGC with the current augmentation strategy outperforms the ScMix and Admix, hence validating the set-level guidance is more suitable for our contrastive training. This is achieved by SGA's alignment-preserving augmentation, which enriches image-text pairs while maintaining their inherent alignments intact [2], hence better maintaining the effectiveness of our malicious contrastive learning. We will include these results in Section E of the Appendix to provide insights for future research.

Finally, we would like to express our gratitude once again for your perceptive and valuable feedback. We hope that the comprehensive response above will effectively address your concerns. It would be our great pleasure to engage in further discussion with you.

[1] Zhang J, Yi Q, Sang J. Towards adversarial attack on vision-language pre-training models. In ACM MM, 2022.

[2] Lu D, Wang Z, Wang T, et al. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. In ICCV, 2023.

[3] Hayes J, Danezis G. Learning universal adversarial perturbations with generative models. In IEEE Security and Privacy Workshops (SPW), 2018.

[4] Feng W, Xu N, Zhang T, et al. Dynamic generative targeted attacks with pattern injection. In CVPR, 2023.

[5] Gao H, Zhang H, Wang J, et al. NUAT-GAN: Generating Black-box Natural Universal Adversarial Triggers for Text Classifiers Using Generative Adversarial Networks. In IEEE TIFS, 2024.

[6] Moosavi-Dezfooli S M, Fawzi A, Fawzi O, et al. Universal adversarial perturbations. In CVPR, 2017.

[7] Chaubey A, Agrawal N, Barnwal K, et al. Universal adversarial perturbations: A survey. In arXiv, 2020.

[8] Zhang, Peng-Fei, Zi Huang, and Guangdong Bai. Universal Adversarial Perturbations for Vision-Language Pre-trained Models. In ACM SIGIR, 2024.

[9] Poursaeed O, Katsman I, Gao B, et al. Generative adversarial perturbations. In CVPR, 2018.

[10] Kim M, Tack J, Hwang S J. Adversarial self-supervised contrastive learning. In NIPS, 2020.

[Q2.2.3] Investigation of the loss function. (Updated)

Inspiring question! The two designed loss terms $\\mathcal{L}\_{CL}$ and $\\mathcal{L}\_{Dis}$ serve distinct roles in reaching the attack goal from multimodal and unimodal perspectives respectively. We have conducted ablation studies in Section 4.4 of the main body and corroborated the effectiveness of both two terms, hence confirming the necessity of jointly employing both $\\mathcal{L}\_{CL}$ and the $\\mathcal{L}\_{Dis}$.

To further address your concerns, we conduct experiments with different unified loss alternatives C-PGC$\_{Cos}$ and C-PGC$\_{MSE}$, which maximizes the negative cosine similarity and MSE between features of matched image-text pairs.
Table C. ASR of C-PGC and its three variants on Flickr30. The surrogate model is ALBEF.

These results again reveal the superiority of the proposed loss function over possible unified alternatives, which will also be added in Section E of the Appendix.

[Q2.3] Contribution of our work.

(1) As stated in the main text, the set-level augmentation was proposed by SGA [2] and we incorporate it into our framework for further enhancement, without claiming it as the major contribution. (2) Meanwhile, it is reasonable and common practice to pursue a strategy that jointly maximizes both types of diversity. However, the pivotal distinction lies in how it is effectively implemented and the degree to which it enhances attack efficacy. (3) We also highlight that contrastive learning is a general concept that has indeed been utilized by previous approaches for general beneficial purposes, such as modality alignment and enhancing adversarial robustness [10].

However, our contribution lies in making the first attempt to propose a novel contrastive paradigm tailored to the malicious scenario of vision-language UAP generation, which effectively fools different VLP models across diverse V+L scenarios. Based on the attack objective, we devise tailored positive and negative sample selection strategies (e.g., the farthest distance selection) to facilitate the contrastive learning of UAP. Moreover, we propose a novel architecture of perturbation generator with cross-modal conditions as auxiliary information to promote the modeling of multimodal UAP. By training the generator with both the introduced set-level cross-modal ($\mathcal{L}\_{CL}$) and unimodal ($\mathcal{L}\_{Dis}$) guidance, C-PGC generates powerful UAP against VLP models with excellent generalization and transferability.

As for the distinction between universal and sample-specific attacks, we hope that our response in [Q2.1] has adequately solved your concerns.

[Q3.1] Comparison with a recent baseline.

Thank you for your constructive suggestion! I guess you are referring to [4] in your cited references, is that correct? We have supplemented our comparison with this excellent concurrent work and please refer to [Q2] in the Common Concerns for details.

[Q2.1] Definition of universal perturbation learning

We are deeply sorry for any misunderstanding our paper may cause you. Please kindly allow us to restate the training and inference paradigm of C-PGC to clarify the use of cross-modal conditions.

Taking image perturbation as an example, the attacker has access to a training dataset of image-text pairs to learn a perturbation generator that transforms a fixed noise $z_v$ into the universal perturbation $\delta_{v}$. Specifically, for each training image-text pair $(v, \mathbf{t})$, the texts $\\mathbf{t}$ are fed into the generator through cross-attention layers as cross-modal conditions, to provide additional knowledge for learning an effective UAP.

However, once we finish the contrastive training of the generator, the final output $\delta_{v}$ becomes the image UAP that can then be applied to any test data without further requiring any cross-modal conditions. This is consistent with established practices of universal perturbation [6,7,8], and ensures the universality and applicability of the universal perturbation, which is definitely independent of the test data and unseen data.

Also, our experimental setup fully aligns with previous UAP studies [6, 8, 9] and uses the training set of datasets (e.g., MSCOCO) to learn the UAP while evaluating the validation set. Moreover, experiments involving “using perturbations generated on the Flickr30k dataset to attack models on the MSCOCO dataset” actually have been presented in Table 7 of our Appendix (Table 12 in the revised version). We apologize for the lack of emphasis on this matter and will highlight this experiment in the main text.

[Q2.2.1] Semantic similarity between adversarial text and its original text

Thank you for this meaningful question. Please kindly refer to [Q1] in the Common Concerns for a detailed response.

[Q2.2.2] Ablation study of diverse target texts.

Very insightful suggestion! Following your advice, we implement a variant C-PGC${_{Sin}}$, which uses only a single target text with the farthest distance as the positive sample:

Table B. Comparison of C-PGC and its variant with single text as the target.

The results illustrate that the use of multiple target texts can enhance attack effectiveness, validating the efficacy of set-level diverse guidance. As for the choices of target texts, we have provided an alternative strategy C-PGC$_{Rand}$ in Section 4.4 of the main text, where target texts are randomly selected instead of choosing the farthest ones. This finding verifies the effectiveness of our farthest selection strategy.

We will supplement the experiment of C-PGC$_{Sin}$ into the Section E of the Appendix. We appreciate your valuable suggestion for improving the comprehensiveness of our analysis.

Dear Reviewer jJKC,

Thank you again for your precious efforts in reviewing our paper and the constructive comments!

As the end of the discussion period approaches, we would like to know whether our responses have properly addressed your concerns. Your feedback will be highly appreciated and we are glad to engage in further discussions with you.

Sincerely,

The Authors

[Q3] Influence the quality of original texts.

We sincerely apologize for the misunderstanding our response may cause you. Please kindly allow us to provide a more detailed explanation of this issue as follows.

1. Clarifications of the attack goal.

(1) We clarify that ensuring the quality of the initial text is not compromised is indeed valuable, but the basic objective of a malicious untargeted adversarial attacker is to fool the victim model to output incorrect predictions while ensuring attack imperceptibility [1, 2, 13, 14], rather than not influence the quality of original data.

(2) Correspondingly, our attack performance is verified by the excellent ASR of extensive experiments in the main text and the comparison with your kindly suggested baseline in Table C of the Common Concerns, and the attack stealthiness is guaranteed by the results measured by sufficient distance metrics in Table A and B of the Common Concerns, presenting a qualified and successful untargeted adversarial attack method.

2. Evaluation of the influence on the quality of original texts.

(3) We emphasize that how to evaluate and reduce the extent of influencing the quality of original texts is still an open issue that has not been explored by existing cross-modal adversarial attacks [1,2,8,15].

Since the original text is the reference representing the original sentence semantics, the semantic similarity between the adversarial and original texts measured by sufficient text distance metrics can be regarded as a reasonable metric to evaluate the extent of influence on the text quality. Therefore, the results in Tables A and B of the Common Concerns verify C-PGC's better performance in maintaining text quality compared with existing well-acknowledged methods [1,2].

(4) To further address your concerns, we provide several visual demonstrations between our method and SGA [4]. We observe that, in most cases, our method indeed exerts less impact on the quality of original texts.

(Clean) Man taking a photograph of a well-dressed group of teens.
(SGA) Man [rights] a photograph of a well-dressed group of teens.
(Ours) Man [getting] a photograph of a well-dressed group of teens.

(Clean) A young girl wearing a bulky red life jacket floating in a lake.
(SGA) A young girl wearing a bulky red life [school] floating in a lake.
(Ours) A young girl [getting] a bulky red life jacket floating in a lake.

(Clean) A brown dog walks in the grass with its tongue hanging out.
(SGA) A brown dog walks in the [new] with its tongue hanging out.
(Ours) A brown dog [getting] in the grass with its tongue hanging out.

(Clean) Two young men are loading fruit onto a bicycle.
(SGA) [Teens] young men are loading fruit onto a bicycle.
(Ours) Two young men are [getting] fruit unto a bicycle.

(Clean) A dog is walking through some gravel beside a river.
(SGA) A dog is walking through some [like] beside a river.
(Ours) A dog is [getting] through some gravel beside a river.

These visualization results correspond with our numeric results in Table A and B of the Common Concerns. More visualization results will be added to the Appendix. Thanks again for your valuable question!

We're more than glad to have more discussions with you if you have any further questions. :)

[11] Naseer M M, Khan S H, Khan M H, et al. Cross-domain transferability of adversarial perturbations. In NIPS, 2019.

[12] Yang X, Dong Y, Pang T, et al. Boosting transferability of targeted adversarial examples via hierarchical generative networks. In ECCV 2022.

[13] Dong Y, Liao F, Pang T, et al. Boosting adversarial attacks with momentum. In CVPR, 2018.

[14] Chakraborty A., Alam M., Dey V., et al. A survey on adversarial attacks and defences. In CAAI Transactions on Intelligence Technology, 2021.

[15] Wang H, Dong K, Zhu Z, et al. Transferable multimodal attack on vision-language pre-training models. In S&P, 2024.

Thanks for your detailed feedback! Your acknowledgment of our above responses encourages us a lot. Next, we provide point-to-point responses below to further resolve your concerns.

[Q1.1] Clarification about Leveraging Cross-modal Knowledge.

A very insightful question! We highlight that the use of cross-modal knowledge in previous cross-modal attacks is limited to simply maximizing the feature distance between samples of different modals for adversarial sample optimization, without any deeper utilization to further enhance attacks [1,2,8].

On the contrary, we are the first to incorporate cross-modal information into the perturbation generator as auxiliary knowledge to facilitate the modeling of multimodal adversarial perturbation. Moreover, we propose the first attack paradigm that utilizes cross-modal knowledge from a novel perspective of malicious contrastive learning. Extensive experimental results have validated the effectiveness of our proposed novel techniques to utilize the cross-modal knowledge, presenting a powerful universal attack framework.

[Q1.2] Task-specific characteristics

We are deeply sorry for any misunderstanding our response may cause you.

We clarify that the essential attack objective of this work aligns with existing well-acknowledged adversarial attacks on VLP models [1,2,8], which aim to fool VLP models themselves, rather than specific downstream tasks. i.e., the generated perturbation is supposed to yield excellent attack performance tailored to the target VLP model regardless of the downstream tasks. Hence, our primary focus is to leverage the shared and joint characteristics of different Vision-Language scenarios to present a universal and versatile UAP that can effectively generalize to diverse downstream V+L tasks.

Also, it is a promising direction for future studies to investigate task-level V+L characteristics to further enhance attacks for specific downstream tasks. We have added them to encourage future studies in Appendix I of the revision.

[Q2] The Superiority of Generative Attacks

We're sorry for lacking a sufficient explanation regarding the underlying mechanism behind the generative attack.

The superiority is achieved by the powerful distributional modeling capability of generative model. Since a universal perturbation is learned on the data distribution independent of specific instances, the generator facilitates the learning of universal perturbation by better perceiving and capturing distribution-level features of diverse image-text samples [4,9,11,12], hence significantly enhancing the attack compared with non-generative methods.

Dear Reviewer jJKC,

We have carefully responded to each of your questions and revised our paper accordingly. The sufficient experiments and analyses have been presented above and the revision details are listed at the top of this page.

We look forward to your reply and welcome discussion on any issues regarding our paper and responses.

Best regards,

The Authors

Dear Reviewer jJKC,

We really appreciate your precious time and efforts in providing insightful feedback on our responses. To further address your concerns, we have presented detailed clarifications and visual demonstrations in the latest author's reply. We kindly request that you review them to see whether your remaining queries have been sufficiently addressed. If you are satisfied with our responses, we would greatly appreciate it if you could kindly consider raising your score accordingly. :)

We're looking forward to your reply.

Best regards,

Authors

Dear Reviewer jJKC,

This is a kind reminder that the end of the rebuttal is really close. We have addressed the majority of your concerns in our first-stage responses and supplemented further experiments with detailed analyses of your remaining issues. If you're satisfied with our rebuttal, we would appreciate it if you could consider raising your score accordingly. :)

Best regards,

Authors

Re to "Re: [Q1.1] Clarification about Leveraging Cross-modal Knowledge"

Sure, we will incorporate this into the introduction and method part of this paper.

Re to "Re: [Q3] Influence on the Quality of Original Texts"

We're quite confused about your subjective opinion regarding the similarity measurement after our detailed clarification and experimental results.

1. The attack performance of our method has been directly measured by the ASR of extensive experimental results. We have highlighted them again and provided a comparison with your suggested SOTA baseline UAP method, which strongly confirms the superiority of our method. It's unclear why you insist on the similarity measurement to doubt the attack performance.
2. We have emphasized that ensuring the quality of the initial text is not the basic objective of adversarial attacks, but attack stealthiness. You may mistakenly mix them up and request harsh constraints on the text perturbation that almost all the existing cross-modal attacks fail to satisfy.
3. We have highlighted the evaluation and reduction of the extent of influencing the quality of original texts is still an open issue that has not been explored by existing cross-modal adversarial attacks [1,2,8,15].
4. We also aim to convey that existing instance-specific attacks exhibit great attack performance due to their instance-wise adversarial perturbation, while they sacrifice the semantic similarity and lead to less stealthy attacks (also compromise the text quality that you're most concerned about). Since our method achieves both better semantic similarity and scores of influence on the text quality than existing instance-specific attacks, C-PGC is a highly qualified adversarial attack method and is definitely acceptable in terms of attack stealthiness and influence on the text quality.

Despite the above facts, we still design and provide sufficient new experiments to solve your concerns (including the direct analysis with the advanced GPT-4o), which repeatively validate the better quality-preserving of our method. However, you seem to neglect the supplemented experiments and keep holding subjective opinions about the measurement, without specifying any possible useful metrics or specific problems within the evaluation protocol.

We kindly request you review our provided experiments and re-assess the quality of this paper. This is the true meaning of the rebuttal phrase of ICLR. And your precious assessment also means a lot to us.

Dear Reviewer jJKC,

Sorry for reminding you again since the end of the rebuttal is really close. You seem to neglect our latest quantitative experiments based on GPT-4o in Author Response (Part VII). We also present detailed clarification regarding your confusing reasons for not increasing scores in Author Response (Part VIII).

We have made great efforts to solve your issues and would appreciate a more fair and objective evaluation.

Best regards,

Authors

Supplementary experiment to '[Q3] Influence the quality of original texts'

To better solve your question, we devise a novel experiment leveraging the advanced GPT-4o to quantitatively evaluate the influence of text perturbation on the quality of the original sentence. Specifically, we query GPT-4o with the following prompt inspired by [16], using 5000 pairs of original and perturbed texts.

As an experienced evaluator, your task is to evaluate the extent of the given word replacement operation to the original text quality. Give a specific integer score based on the following statements, ranging from 0 to 4: Very Poor (0): The word replacement greatly impacts the quality of the initial sentence, e.g., largely destroying the basic semantic or basic grammar.
Poor (1): The word replacement causes significant semantic deviation and ambiguity, e.g., the perturbed sentence looks strange and illogical.
Fair (2): The perturbed sentence is readable and logical, showing likeness to the initial sentence with notable semantic variances.
Good (3): The perturbed sentence is reasonable and semantic-preserving.
Excellent (4): The word replacement nearly brings no significant influence on the sentence quality.

Every time you receive two sentences, the first one is an original sentence, and the second is a sentence perturbed by the word replacement operation.

The obtained average scores are as follows:

Table E. GPT-4o score of different methods in terms of the influence on the text quality. The high values indicate better performance.

These results again prove that C-PGC achieves better performance in terms of the influence on the original text quality. Among existing cross-modal attacks [1,2,8,15], we make the first attempt to quantitatively evaluate the direct influence of text perturbation on the original sentence quality based on this experiment, which serves as an important addition to better solve your question.

[16] Peng Y, Cui Y, Tang H, et al. Dreambench++: A human-aligned benchmark for personalized image generation. In arXiv, 2024.

[Q2] R#jJKC, R#U4zG: Comparison with the recent SOTA.

As noted by R#jJKC, we notice a concurrent study on UAP attacks for VLP models [7], which shows promising attack performance. To make a fair comparison, we faithfully reproduce this method using their publicly released code under the same experimental settings as ours. Note that [7] implements several versions of their method and we report their best results as follows:

Table C. Comparison of our C-PGC with a recent SOTA attack [7] on Flicke30K.

By contrastively training our designed cross-modal conditional generator, the proposed C-PGC greatly enhances the attack and achieves significant improvements in ASR. Particularly in the more realistic and challenging transferable scenarios, the proposed method achieves considerably better performance, e.g., 32.19% and 28.57% increase in ASR of TR and IR tasks when transferring from ALBEF to TCL. These results confirm the superiority of our contrastive learning-based generative paradigm. We will cite this paper and include these results in a new section of the Appendix.

Thanks again for mentioning this great work! We would appreciate it if you could kindly inform us of any additional UAP methods targeting VLP models that we may have inadvertently omitted. We would be delighted to reproduce and compare these methods to ensure a comprehensive assessment.

[1] Dong Y, Liao F, Pang T, et al. Boosting adversarial attacks with momentum. In CVPR, 2018.

[2] Chakraborty A., Alam M., Dey V., et al. A survey on adversarial attacks and defences. In CAAI Transactions on Intelligence Technology, 2021.

[3] Zhang J, Yi Q, Sang J. Towards adversarial attack on vision-language pre-training models. In ACM MM, 2022.

[4] Lu D, Wang Z, Wang T, et al. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. In ICCV, 2023.

[5] Wang H, Dong K, Zhu Z, et al. Transferable multimodal attack on vision-language pre-training models. In S&P, 2024.

[6] Zhang T, Kishore V, Wu F, et al. Bertscore: Evaluating text generation with bert. In ICLR, 2020.

[7] Zhang, Peng-Fei, Zi Huang, and Guangdong Bai. Universal Adversarial Perturbations for Vision-Language Pre-trained Models. In ACM SIGIR, 2024.