TUAP: Targeted Universal Adversarial Perturbations for CLIP

Q4: Evaluation of specific fields such as medical datasets.

A4: We appreciate the reviewer’s suggestion to evaluate our method on specialized domains. To address this, we have conducted additional experiments using the COVID-19 Radiography dataset (Cohen et al., 2020) and the Cholec80 dataset (Twinanda et al., 2016). The COVID-19 Radiography dataset comprises X-ray images, while Cholec80 consists of laparoscopic cholecystectomy videos. Since these datasets do not include text captions, we evaluate the effectiveness of our TUAP by reporting the CLIP scores (higher is better) between the embeddings of the targeted text sentences and the image embeddings, both with and without the application of TUAP.

These medical datasets are significantly different from the datasets on which our TUAP was originally trained. The results below indicate that our TUAP remains effective in achieving adversarial objectives within specialized domains. The evaluations on medical datasets underscore the broad applicability and effectiveness of our TUAP method beyond standard image tasks. This reinforces the severity and generalizability of the adversarial threats posed by TUAPs, highlighting the need for enhanced robustness in VLMs across diverse application areas.

Cohen, J. P., Morrison, P., & Dao, L. (2020). COVID-19 image data collection. arXiv preprint arXiv:2003.11597.
Twinanda, A. P., et al. (2016). Endonet: A deep architecture for recognition tasks on laparoscopic videos. IEEE Transactions on Medical Imaging, 36 (1), 86-97.

Thank you very much for reviewing our paper and insightful comments. Please find our response to your questions below:

Q1: Diversity of target texts and actual scenarios is relatively limited.

A1: All the reported results in our submission are based on 10 target text sentences, which are detailed in Appendix B.3 and table below. We believe these targets cover a broad range of scenarios, including various types of content that an attacker might choose. While we acknowledge that an attacker could select any text as the target—and these 10 examples cannot cover every possible case—we believe they sufficiently demonstrate the feasibility and severity of this adversarial threat. Therefore, we consider our selection of target texts adequate for illustrating the potential risks posed by TUAPs.

In future work, we plan to investigate a more extensive range of target sentences and develop a benchmark dataset with pre-trained TUAP pairs for each sentence to further expand the scope and applicability of this research.

Targets No. 1 and No. 6, the placeholder $\mathtt{{X}}$ represents names of persons. The "..." in target No. 5 is an actual URL link.

Q2/3: Perceptibility and ablation study.

A2/3: In our initial submission, we have already conducted an analysis of different perceptibility levels and their impact on attack performance. This analysis is detailed in Section 4.5, Figure 4(a), Appendix B.10, and Figure 5. The corresponding visualizations can be found in Figures 24 to 26.

Regarding the $L_\infty$ attack, an $\epsilon$ value of 8/255 is typically considered the threshold below which perturbations are imperceptible to the human eye. As shown in Figure 4(a), reducing $\epsilon$ to this level does not significantly decrease the attack success rate. This demonstrates that our method remains effective even when the perturbations are imperceptible.

Concerning the use of random patch locations, we conducted a comparative study, and the results are summarized in the table below. The attack success rates indicate that there is no significant difference between applying the patch at random locations versus fixed locations.

These results are based on a patch attack generated with E-16, evaluated on the ViT-L-14 model from OpenAI using the ImageNet dataset. The findings suggest that our method is robust to variations in patch placement, further validating the effectiveness of our approach.

Dear reviewer QeU8,

This is a kind reminder that we have provided a comprehensive response to your initial review. Specifically, we have demonstrated that TUAP is also effective when applied to medical imaging datasets. The ablation studies included in the initial submission have been further expanded with new experiments investigating the impact of patch location on performance.

We sincerely appreciate your positive acknowledgment of our work and look forward to any further feedback or discussions you may have.

Summary

Our method significantly differs from SGA and SA-Attack both technically and empirically:

• Technical Innovation: We introduce a targeted universal attack framework, whereas SGA and SA-Attack are untargeted and sample-specific.
• Practical Advantages: Our TUAP is more practical for large-scale attacks due to its universal nature, requiring only a single perturbation applicable to any image.
• Superior Transferability: TUAP achieves simultaneous cross-model, cross-dataset, and cross-task transferability, which is not demonstrated by SGA or SA-Attack.

We believe these distinctions highlight the novelty and contributions of our work, advancing the understanding of adversarial attacks in vision-language models.

Q2: Comparison with related works SGA [a] and SA-Attack [b].

A2: In our updated submission, we have included both SGA [a] and SA-Attack [b] in our related work discussion. Due to significant differences in problem settings, our initial submission did not consider these works in detail. Below, we provide both technical and empirical comparisons to clarify how our work differs from these studies.

=== Technical Comparison ===

Attack Objectives:

• SGA and SA-Attack: Both methods are untargeted attacks that aim to create image-text pairs dissimilar in the embedding space, disrupting the alignment without specifying a particular target.
• Our TUAP: We propose a targeted attack where the perturbation modifies the image embedding to align closely with a specific text sentence chosen by the attacker.

Perturbation Types:

• SGA and SA-Attack: These methods utilize sample-specific perturbations, which are generated individually for each image-text pair. This approach requires generating a new perturbation for every image, making it less practical for large-scale attacks.
• Our TUAP: We introduce a universal perturbation that is generated once and can be applied to any image, greatly enhancing the practicality and scalability of the attack.

Transferability:

• SGA and SA-Attack: The reliance on sample-specific perturbations limits their cross-dataset transferability since the perturbations are tailored to specific images and cannot be applied to others.
• Our TUAP: Our universal perturbation enables simultaneous cross-model, cross-dataset, and cross-task transferability, as verified by our experiments. This means our TUAP can effectively attack different models, datasets, and tasks using the same perturbation.

=== Empirical Comparison ===

Given that SA-Attack is not open-sourced and both SA-Attack and SGA share similar threat models and problem domains, we focus our empirical comparison on SGA. If an official implementation of SA-Attack becomes available, we would be happy to include it in our analysis.

Experimental Setup:

• Perturbation Magnitude: To ensure a fair comparison, we generated SGA perturbations with an $\epsilon$ of 16/255, consistent with our TUAP settings.
• Tasks Evaluated: We evaluated cross-task and cross-model transferability, corresponding to Section 4.3 in our paper and Section 5.4 in SGA's paper.
• Datasets and Metrics: We used the MSCOCO dataset and compared CIDEr scores (lower is better) on image captioning tasks.

Results:

Cross-Task and Cross-Model Transferability: The results, summarized in the table below, show that our TUAP exhibits stronger cross-model and cross-task transferability than SGA. TUAP achieved this with a single universal perturbation rather than sample-wise perturbations, highlighting its efficiency and effectiveness.

Image-Text Retrieval (untargeted): We also compared performance on standard image-text retrieval tasks using untargeted attack evaluations as in SGA. Our TUAP consistently outperformed SGA, except for a slightly lower IR R@1 when using ResNet-101 as the encoder.

Thank you very much for reviewing our paper. Please find our response to your questions below.

Q1: Simple idea and novelty.

A1:

This idea is trivial and not so novel. Universal adversarial attacks have been well explored in the context of standard adversarial attacks for single-modal architectures.

We acknowledge that universal adversarial attacks have been studied extensively in single-modal architectures. However, our work explores Targeted Universal Adversarial Perturbations (TUAPs) for vision-language pre-training models, specifically for CLIP models—a novel problem that has not been addressed in prior research.

It seems that the authors do not make any specific designs to transfer it into the vision-language models.

Can the novel designs for attacking CLIP models be shown?

Our attack objectives, defined in Eq. (2) to (7), are specifically designed for TUAPs against CLIP models. We have now conducted a comparison between the original UAP method by Moosavi-Dezfooli et al. (2017) and our TUAP. Since UAP is an untargeted attack and cannot produce exact target texts, we compare adversarial accuracy (the lower, the better) to ensure a fair evaluation. The significant performance difference between our TUAP and the original UAP indicates that our design—Eq. (2) to (7)—is specifically tailored for attacking CLIP models. By attacking the CLIP image encoder and ensuring transferability, our generated TUAPs can be directly applied to attack downstream VLMs. And revealing this phenomenon itself is one of our key contributions.

According to the pseudocode, the proposed method seems to be a general approach for all the architectures but not specific to CLIP models.

We believe the reviewer refers to the ensemble approach used in our work. This approach is intentionally designed to be model-agnostic, which we consider a strength rather than a limitation. While our method is specifically evaluated on CLIP models, making it model-agnostic allows it to serve as a general framework for TUAPs. This generality provides great potential for further exploration and demonstrates the broader applicability of our method.

Q3: Can the authors provide some comparisons with query-based black-box attack approaches?

A3: We appreciate the reviewer's insightful suggestion. However, to the best of our knowledge, query-based black-box attacks against CLIP models—especially for generating TUAPs—remain an unexplored area. If there are existing works that we have overlooked, we would be grateful if the reviewer could point them out.

To address your concern, we attempted to adapt the NESAttack method (Ilyas et al., 2018) to our setting. Specifically, we replaced the optimization component in our method with the NESAttack strategy while keeping the other objectives the same as in our initial submission. However, we found that query-based methods were unable to produce successful TUAPs. The results are summarized below:

We believe the primary reason for this difficulty is the substantial number of queries required to generate even sample-specific perturbations in a black-box setting. For universal perturbations, a single perturbation must be adversarially effective across all images, which significantly amplifies the challenge. The need for such broad generalization without direct access to the model's internals makes query-based TUAP generation exceedingly demanding. Therefore, developing effective query-based black-box methods for TUAPs remains an open problem for future research.

Ilyas, Andrew, et al. "Black-box adversarial attacks with limited queries and information." ICML 2018.

Q4: In addition to adversarial images, is it possible to generate universal adversarial texts?

A4: While it is theoretically possible to generate universal adversarial texts, significant modifications to the current formulation of TUAP would be necessary. To the best of our knowledge, the concept of universal adversarial texts remains an unexplored area in the field of adversarial attacks. However, our ensemble framework is general and could be adapted to generate adversarial texts in future work. We believe this represents an exciting direction for further research, which could provide deeper insights into the vulnerabilities of VLMs.

Dear Reviewer mcuc,

This is a kind reminder that we have provided a comprehensive response to your initial review.

We have provided both technical and empirical comparisons with SGA and SA-Attack, as the reviewer mentioned. These works study different attack objectives and types of perturbations, making direct comparisons inherently challenging. We have added new results in our submission and rebuttal.

It is important to emphasize that comparing universal perturbations (our TUAP) with sample-specific perturbations (SGA and SA-Attack) is inherently unfair, as sample-specific perturbations are customized for each individual sample, giving them a significant advantage. Despite this disadvantage, our TUAP consistently outperforms or matches SGA across several evaluations, highlighting its novelty in achieving universal adversarial transferability.

We kindly request that you review the newly provided results, summarized below:

• Even when compared to sample-specific perturbations, our TUAP outperforms SGA in cross-task transferability evaluations with untargeted objectives. Specifically, the CIDEr score is halved when comparing TUAP with SGA.
• By running the released code for SGA and directly applying our TUAP to any samples, we demonstrate that TUAP outperforms or matches SGA in untargeted objectives for COCO image-text retrieval tasks. This result is significant because sample-specific perturbations are generally considered stronger than universal perturbations.
• We achieve simultaneous black-box cross-model, cross-dataset, and cross-task adversarial transferability. These results meet challenging targeted attack objectives and distinguish our approach from that of existing studies.

We hope these results address your concerns and provide clarity on the novelty and contributions of our work. We look forward to further constructive discussions with you.

Dear Reviewer mcuc,

Thank you once again for taking the time to review our paper. We deeply value your insightful comments and are sincerely grateful for your efforts.

We kindly request you to review our reply to see if it sufficiently addresses your concerns. Your feedback means a lot to us.

Thank you deeply, Authors

Q5: Meeting the acceptance criteria for ICLR.

A5:

In summary, this paper is more like an experimental report rather than a top-tier conference paper, far from meeting the acceptance criteria for ICLR.

We appreciate the review acknowledging our paper's strong soundness, clear presentation, and comprehensive experiments. According to the ICLR Reviewer Guide (2021–2025), novelty should be evaluated not only based on technical methods but also on novel findings. Additionally, the guideline suggests that "the submission clear, technically correct, experimentally rigorous, reproducible" is a strong point. We believe our detailed experiment setting for reproducibility, rigorous experiments, technically sound method, and clear presentation are strengths.

By addressing the technical challenges in crafting targeted UAPs (TUAPs) for CLIP, our work is the first to demonstrate the possibility of generating highly-transferable TUAPs for CLIP based on an ensemble of surrogate models. Our large-scale and comprehensive analysis provides a set of new insights into the vulnerability of CLIP, whilst revealing a scaling law for TUAP transferability. We believe our work offers significant contributions to the community in both the technical and empirical respects.

We genuinely hope the reviewer can reassess the contribution of our work. This would mean a lot to us.

The following table reports the results (untargeted adversarial accuracy) based on our evaluation (the same as in our Table 1). These results further verify that we can achieve cross-model transferability, which AdvCLIP does not.

The table below reports the targeted results (targeted attack success rate) based on our evaluation. It shows that AdvCLIP cannot perform targeted attacks.

Contributions Compared to AdvCLIP:

• Simultaneous Cross-Model, Cross-Dataset, and Cross-Task Transferability: Our TUAP achieves transferability across models, datasets, and tasks, simultaneously.
• Scalable Transferability with Surrogate Models: We discovered that transferability scales with the number of surrogate models used. To the best of our knowledge, there has been no previous study on UAPs revealing this transfer scalability.
• New Adversarial Vulnerability in Large VLMs: We revealed a new safety risk for large VLMs, demonstrating that an adversary can conduct effective attacks using an ensemble of pre-trained encoders. This aspect is not examined in AdvCLIP.

Given the significant differences in both technical approach and empirical results, we believe the novelty and contributions of our work are clear and substantial compared to AdvCLIP.

Q4: No significant contributions compared with AdvCLIP

A4:

We believe the reviewer may have misunderstood our work. Please allow us to clarify the significant differences between our method and AdvCLIP.

=== Technical Comparison ===

(1) Different attack objectives: Our method is a targeted attack, whereas AdvCLIP is an untargeted attack.

A direct quote from the AdvCLIP paper's abstract:

"achieving universal non-targeted attacks. "

(2) Black-box transferability: Our TUAP achieves black-box cross-model, cross-dataset, and cross-task adversarial transferability, simultaneously, which is impossible for AdvCLIP as it requires white-box access to the victim encoder.

We quote from the AdvCLIP paper's Section 3.1 Threat Model:

"We assume a quasi-black-box attack model, where the attacker has access to VLP encoders through purchasing or downloading from publicly available websites …"

In our context, the VLP encoder is the victim model. Our method does not make such an assumption; we operate in a complete black-box setting.

=== Empirical Comparison ===

Since no pre-trained weights or UAPs are available in the official AdvCLIP repository, we reproduced the UAP using their released code with default hyperparameters. Due to different attack objectives, implementations, and evaluation protocols, we conducted the following empirical comparisons using both our evaluation and AdvCLIP's official repository, separately.

The table below reports the results based on AdvCLIP's evaluation, with the untargeted adversarial accuracy (Adv acc) and fooling rate.

• First row: We reproduced the white-box attack performance reported in the AdvCLIP paper.
• Second row: AdvCLIP failed to achieve black-box cross-model transferability.
• Third row: Our TUAP outperforms AdvCLIP by increasing the fooling rate by 59.5% and reducing adversarial accuracy by 43%, whereas AdvCLIP only reduces adversarial accuracy by 0.5%. Our TUAP successfully achieves cross-model transferability.

Q2: The introduction section requires further reformulation

A2: We have thoroughly revised the introduction section in our updated submission to address the concerns you've raised. Our goal was to enhance clarity, strengthen the motivation for our work, and more effectively articulate our contributions. If there are specific areas or sentences that you believe still require improvement, we would greatly appreciate it if you could point them out directly. Your detailed feedback will help us make precise adjustments to further improve the introduction.

Q3: The reviewer does not see any insightful analysis regarding the underlying mechanism of the attack algorithm.

A3: We have analyzed the underlying mechanism of our attack algorithm in Section 4.4. Specifically, we explained how the concept blending capability that empowers the strong zero-shot performance of CLIP models also makes them susceptible to TUAPs. This blending capability allows CLIP to associate visual features with a wide range of textual concepts, which can be exploited by TUAPs to induce specific, adversarial outputs. Our analysis sheds light on why CLIP's strengths in zero-shot learning can be turned into vulnerabilities, offering a deeper understanding of the attack's effectiveness.

Thanks for reviewing our paper and the comments. Please find our responses to your questions below.

Q1: Lacking in-depth exploration and novel insights

A1:

intuitive attempt by simply combining existing techniques from former studies

We would appreciate it if the reviewer could specify any prior studies that have demonstrated our findings or require us to compare them technically. Novelty assessments should be based on concrete comparisons rather than subjective impressions.

lacking in-depth exploration and novel insights

We would like to highlight the following in-depth explorations and novel insights that may have been overlooked:

• Cross-Model Transferability: We comprehensively demonstrate cross-model transferability using 6 CLIP encoders and four downstream large VLMs.
• Cross-Dataset Transferability: The TUAPs generated on ImageNet or CC3M successfully transfer across 11 additional datasets.
• Cross-Task Transferability: Our TUAPs, generated without targeting specific tasks, successfully transfer across diverse tasks, including image classification, image-text retrieval, image captioning, and visual question answering (VQA). Notably, TUAPs not only cause arbitrary errors but induce targeted errors specified by the adversary.
• Simultaneous Transferability: These transferability aspects are not mutually exclusive. Our TUAP achieves simultaneous cross-model, cross-dataset, and cross-task transferability.
• Scalable Transferability with Surrogate Models: Most importantly, we reveal that transferability scales as the number of surrogate models increases. To the best of our knowledge, no previous work has revealed this scaling trend of UAP transferability.
• New Adversarial Vulnerability in Large VLMs: We have uncovered an emerging safety risk in large VLMs, where an adversary can launch widespread targeted attacks on VLMs using universal patterns (TUAPs).

We believe these contributions represent significant in-depth exploration and novel insights that advance the field.

Dear Reviewer Ypoi,

This is a kind reminder that we have provided a comprehensive response to your initial review.

We have conducted both technical and empirical comparisons between our TUAP and AdvCLIP, which the reviewer mentioned. The main difference lies in white-box/black-box accessibility to victim encoders and attack objectives.

We kindly request that you review the newly provided results, summarized as follows:

• By running AdvCLIP’s code, we demonstrate that it fails to achieve cross-model transferability, whereas our TUAP successfully accomplishes this.
• Our TUAP significantly outperforms the UAP generated by AdvCLIP in untargeted evaluations, as AdvCLIP cannot achieve cross-model transferability.
• TUAP also supports targeted attacks, which AdvCLIP does not, as it is limited to untargeted attack scenarios.
• On several evaluations, TUAP achieves a performance gap of 100% compared to 0% for AdvCLIP, highlighting the fundamental differences in the attack threat models studied.
• We achieve simultaneous black-box cross-model, cross-dataset, and cross-task adversarial transferability. These results meet challenging targeted attack objectives and distinguish our approach from that of existing studies.

We hope these results address your concerns and clarify the contributions of our work. We look forward to engaging in constructive discussions with you.

Dear Reviewer Ypoi,

Thank you once again for taking the time to review our paper. We deeply value your insightful comments and are sincerely grateful for your efforts.

We kindly request you to review our reply to see if it sufficiently addresses your concerns. Your feedback means a lot to us.

Thank you deeply, Authors

Thank you very much for reviewing our paper and the valuable comments. Please find our response to your questions below:

—

Q1: Technical novelty and comparison with UAP

simply migrates the Universal Adversarial Perturbations generation method in image classification to the VLM task

Adapting an existing method to a novel problem space is a meaningful contribution, and our approach is not a simple migration. As we explained in the introduction, previous studies on Universal Adversarial Perturbations (UAPs) for image classifiers have largely focused on untargeted attacks. Traditional targeted UAPs, on the other hand, are limited to fixed classes. In contrast, our TUAP method for CLIP is capable of targeting any arbitrary text sentence, which significantly expands the capability and complexity of UAP-based attacks.

Furthermore, according to the ICLR Reviewer Guide (2021–2025), novelty should be evaluated not only based on technical methods but also on novel findings. Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge. Drawing attention to a new application is also a valuable contribution.

By overcoming the technical challenges of crafting adversarial perturbations based on surrogate ensembles, our work is the first to demonstrate the possibility of targeted UAPs on CLIP models, highlighting an emerging safety issue for large vision-language models. We believe our work offers significant contributions in both respects.

Not every impactful work needs to propose a completely new method, nor does every method have to be completely different from others. We hope the reviewers can re-evaluate the contribution of our work based on the breadth and depth of our experimental analysis, as well as the practical significance of our findings.

there is no technical innovation.

We respectfully disagree with the assertion that our work has "no technical innovation." Eq. (2) to (7) are specifically designed for targeted UAP attacks against CLIP models. Our method is distinctive because it enables targeted UAP attacks towards any arbitrary text and achieves transferability through surrogate model ensembles, setting it apart from existing attacks.

To the best of our knowledge, there is no existing work on targeted UAP attacks for CLIP in the current literature. Here, we provide the following empirical evidence to show the differences between our method and the original UAP attack [1]. Since UAP is an untargeted attack, it cannot produce the exact text target. Therefore, we compare adversarial accuracy (the lower, the better) to ensure a fair evaluation against UAP. A significant performance difference is observed between our TUAP and UAP, indicating that solving Eq. (2) to (7) in the context of CLIP presents unique technical challenges that we have successfully addressed.

[1] Moosavi-Dezfooli, Seyed-Mohsen, et al. "Universal adversarial perturbations." CVPR, 2017.

Q2: Lack of comparison with Universal Adversarial Attack methods for VLM models [1][2]

A2: The reviewer suggested papers that employ sample-specific perturbations; however, these are fundamentally different from universal attacks. Below, we provide both technical and empirical comparisons to clarify the distinctions:

One of the main differences between our work and the studies referenced by the reviewer [1,2] is the type of perturbation used. The works [1,2] utilize sample-specific perturbations that must be generated individually for each image, which can be time-consuming and impractical for large-scale attacks. In contrast, UAPs are generated once and can be applied to any image, making them more suitable for widespread and real-world scenarios.

The reliance on sample-specific perturbations in [1,2] prevents them from achieving cross-dataset transferability because the perturbations are tailored to specific images and cannot be applied to others. Moreover, [1,2] did not demonstrate cross-task transferability in their studies.

In our work, we comprehensively demonstrate that our Targeted Universal Adversarial Perturbation (TUAP) achieves cross-model, cross-dataset, and cross-task transferability simultaneously. This underscores the robustness and practical significance of our approach compared to sample-specific perturbations.

[1] Zhao Y, Pang T, Du C, et al. On evaluating adversarial robustness of large vision-language models[J]. Advances in Neural Information Processing Systems, 2024, 36.

We followed the same experimental setting as in Table 2 of [1], which computes the CLIP score (higher is better) for the ImageNet validation set. Since we do not know the random 1K image split used by [1], we performed the evaluation on the full validation set using target text sentence No. 8.

We observe that our TUAP outperforms [1] in the same type of modality attack (MF-it). For the best attack modality (MF-ii + MF-tt) in [1], our method outperforms theirs when BLIP2 is the victim model and achieves comparable results when MiniGPT4 is the victim model.

Notably, our TUAP achieves this performance using a single perturbation applied universally to all images, whereas [1] employs sample-specific perturbations tailored to each image. This is a significant result, given that sample-specific perturbations are generally known to be stronger than universal ones.

Additionally, we would like to point out that [1] was published at NeurIPS 2023, not 2024.

Q3: How does the proposed method perform on closed-source models such as chatgpt4?

A3: Thanks for the thoughtful suggestion. We have now evaluated our method on GPT4o-mini using the same experimental settings as in Table 2 of our initial submission. Following the approach suggested in [2] by the reviewer, we also report the metric where an attack is considered successful if the keyword from the target text appears in the VLM's response. The results, based on the $L_\infty$-norm bounded threat model with target No. 8, are summarized below. It clearly demonstrates that the ensemble technique we proposed for generating TUAPs is highly effective in achieving targeted attacks against commercial models like ChatGPT. This underscores the practical applicability and robustness of our method, even when applied to closed-source models.

We also summarized the BLEU-4 metric (for targeted attack) in the table below:

As an example, we asked GPT4o-mini using the same prompt shown in our Figure 1 (Section 4.4) to describe the image, and the following are the exact responses (the target words are boldfaced):

“The image shows a surreal landscape with a bridge in the foreground and a mountainous area with flowing lava in the background. A shark is depicted above the mountains, adding a fantastical element. The combination of lava, the bridge, and the shark creates a dream-like, otherworldly scene.”

Q4: In Figure 1, there are apparent shark textures in the generated adversarial images

A4: The perceptibility of the shark textures in the generated adversarial images is due to the $\epsilon$ value used in the $L_\infty$-norm bounded attack. Our default choice of $\epsilon = 16/255$ is standard in black-box adversarial studies. The paper [2] referenced by the reviewer also uses $\epsilon = 16/255$ and even larger perturbations up to $\epsilon = 32/255$.

Regarding why an actual shark pattern appears in the perturbation, we have provided novel insights in Section 4.4 of our submission. The concept blending capability that empowers CLIP's zero-shot generalization also makes it vulnerable to adversarial manipulation, resulting in perceptible patterns related to the target concept.

The trade-off between the perceptibility of the perturbation and the attack success rate (ASR) is thoroughly analyzed in our initial submission. In Section 4.5, Figure 4(a), we present an ablation study on $\epsilon$ and ASR. Lower values of $\epsilon$ lead to better imperceptibility, as shown in the corresponding visualizations in Figure 25. Similar studies for $L_2$ and adversarial patch attacks are presented in Appendix B.10 and Figure 5, with visualizations in Figures 24 and 26.

An attacker could choose a lower value of $\epsilon$ to achieve better imperceptibility at the cost of a slight decrease in ASR. For example, in Figure 25, under $\epsilon = 8/255$, it is difficult to observe any shark texture. This is a common trade-off in adversarial robustness studies.

[2] Dong Y, Chen H, Chen J, et al. How Robust is Google's Bard to Adversarial Image Attacks?[J]. arXiv preprint arXiv:2309.11751, 2023.

1. Comparison of Untargeted Evaluation

We used the adversarial perturbations generated on the NIPS17 dataset released by [2] and compared them with our TUAP applied to the same dataset. We report the adversarial accuracy (the lower, the better) based on the $L_\infty$-norm bounded threat model with target No. 8. The results show that our TUAP outperforms [2] by using a single universal perturbation, whereas [2] uses perturbations specifically designed for each individual image.

2. Comparison of Targeted Evaluation

We also conducted a comparison in the targeted setting with [2]. Possibly due to the computational expense of generating perturbations for each image, [2] released only 20 adversarial images. We utilized these 20 images and their corresponding text sentences to evaluate using the same task as in Table 1 of our submission. The results indicate that our TUAP significantly outperforms [2] in achieving targeted objectives.

Dear Reviewer Tkjh,

This is a kind reminder that we have provided a detailed, point-by-point response to your initial review.

We have provided both technical and empirical comparisons with the related works [1,2] mentioned by the reviewer. These works study different types of perturbations, making direct comparisons inherently challenging. We have added new results in our submission and rebuttal.

It is important to emphasize that comparing universal perturbations (our TUAP) with sample-specific perturbations [1,2] is inherently unfair, as sample-specific perturbations are customized for each individual sample, giving them a significant advantage. Despite this disadvantage, our TUAP consistently outperforms or matches [1,2] across several evaluations, highlighting its novelty in achieving universal adversarial transferability.

We would appreciate it if you could review the newly provided results. Below is a summary:

• Our TUAP demonstrates performance that is either superior to or on par with [1], despite using a single universal perturbation applicable to any sample.
• Similarly, TUAP outperforms [2] on both untargeted and targeted objectives, again with a single universal perturbation.
• Our objective functions, as defined in Equations (2)-(7), are specifically tailored for CLIP models. New results indicate that without these modifications, UAP achieves minimal success against CLIP models.
• We achieve simultaneous black-box cross-model, cross-dataset, and cross-task adversarial transferability. These results meet challenging targeted attack objectives and distinguish our approach from that of existing studies.

We hope these additions and results address your concerns. We are open to further constructive discussions and welcome your insights.

Dear Reviewer Tkjh,

Thank you once again for taking the time to review our paper. We deeply value your insightful comments and are sincerely grateful for your efforts.

We kindly request you to review our reply to see if it sufficiently addresses your concerns. Your feedback means a lot to us.

Thank you deeply, Authors