OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization

Q1: The authors should evaluate additional vision-language models, such as Eva-CLIP and BLIP2 for image-text retrieval tasks, and MiniGPT4 and Qwen2-VL for image generation tasks in a transfer-based setting.

Thank you for your feedback. We adopt more vision-language models as the source models for experiments, such as Eva-CLIP and BLIP2, for image-text retrieval tasks. The results on Eva-CLIP are shown in the following Table.

It highlights the superior performance of our OT-Attack compared to the baseline SGA method in adversarial attack success rates on image-text retrieval tasks. Across all models, OT-Attack achieves higher results in both TR R@1 and IR R@1 metrics. Notably, on challenging models like ALBEF and TCL, OT-Attack outperforms SGA by significant margins (e.g., IR R@1: 28.23 vs. 26.23 on ALBEF, 32.42 vs. 30.12 on TCL). Similarly, OT-Attack consistently achieves the best results for CLIP-based models, demonstrating its effectiveness across diverse architectures.

The results on BLIP2 are shown in the following Table. It also highlights the effectiveness of OT-Attack, which consistently outperforms SGA across all models and metrics. Notably, OT-Attack achieves higher success rates on ALBEF (TR R@1: 58.89 vs. 51.23, IR R@1: 69.23 vs. 63.53) and TCL (TR R@1: 52.18 vs. 48.42, IR R@1: 64.27 vs. 58.94). Hence, our OT-Attack demonstrates superior adversarial sample transferability compared to SGA. Please refer to Appendix I for more details.

We adopt more models, such as MiniGPT4 and Qwen2-VL, for image generation tasks. We compare the impact of adversarial attacks on image captioning performance for MiniGPT4 and Qwen2-VL with the prompt Describe the image using metrics like BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE. The results are shown in the following Table. It is clear that our OT-Attack consistently outperforms SGA, achieving the lowest scores across most metrics, which indicates more effective attacks. For instance, OT-Attack reduces CIDEr scores to 109.5 for MiniGPT4 and 103.9 for Qwen2-VL, highlighting its superior ability to degrade captioning performance compared to other methods. These results underscore the efficacy of the proposed OT-Attack. Please refer to Appendix J for more details.

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1: Enhancing the clarity and coherence of the writing, particularly in the methods section, would significantly improve its readability and impact.

Thank you for your feedback. We have revised the methods section to enhance its clarity and coherence, improving readability and impact. Please refer to Approach Section of Page 3-5.

W2: The titles of Tables 1 and 2 are identical, which may easily confuse readers and hinder their understanding of the experiments.

Thank you for pointing this out. We rewrite the captions of Table 1 and Table 2 so that readers can distinguish them and improve readability. Please refer to Table 1 of Page 6 and Table 2 of Page 7.

Q2: I did not find any ablation experiments addressing the role of image-text augmentation and optimal transport theory in OT-attack.

Thank you for pointing this out. We conduct an ablation study of the proposed OT-Attack. ALBEF is used as the source model. The study compares three settings: removing the optimal transport mechanism (OT-Attack w/o OT), removing data augmentation strategies (OT-Attack w/o Augmentation), and the complete method (OT-Attack). The results are shown in the following table. The results show that removing OT or augmentation can reduce the adversarial transferability. Notably, the complete OT-Attack achieves the best adversarial transferability, highlighting the critical role of optimal transport and data augmentation in the proposed OT-Attack. Please refer to Appendix K for more details.

Q3: I recommend providing a theoretical explanation and further experimental analysis to support this assertion.

Thank you for your feedback. Overfitting adversarial examples (AEs) to the source model can significantly reduce the attack transferability. To quantify the risk of overfitting, we leverage the PAC-Bayes theorem to measure the information stored in the network's weights (IIW), a promising indicator of generalization ability. Lower IIW values indicate reduced overfitting risks. For each AE generated by SGA or our method during optimization iterations, we compute its IIWs by feeding it into four VLMs. We evaluate the IIWs of 1,000 AEs throughout the optimization process and present the averaged results in $\\textrm{\\color{blue}Figure 7}$ (Page 19). During optimization, the IIW of AEs from the SOTA baseline (SGA) initially decreases but then sharply rises. In contrast, our method maintains consistently low IIW values for generated AEs, effectively mitigating overfitting risks. Consequently, our method enhances attack transferability. Please refer to Appendix H for more details.

Dear Reviewer vFsY,

Thank you once again for taking the time to review our paper. We deeply value your insightful comments and are sincerely grateful for your efforts.

We kindly request you to review our reply to see if it sufficiently addresses your concerns. Your feedback means a lot to us.

Thank you deeply, Authors

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1: My primary concern is the focus on attacking VLP models solely through adversarial image generation, while neglecting the equally crucial role of text in these models.

We sincerely appreciate the reviewer’s comprehensive consideration of multimodal attacks. Indeed, both SGA and OT-Attack target both text and image modalities. The primary approach involves using images to guide the generation of adversarial text and using text to guide the generation of adversarial images. OT-Attack focuses on applying optimal transport theory to the image modality, and the optimization of adversarial images inherently leads to improved results for adversarial text generation.

Q1: What is the time cost associated with using optimal transport theory?

Thank you for the feedback. Following your suggestion, we compute the computational cost (minutes) and compare the OT-Attack with SGA across four models (ALBEF, TCL, CLIP_VIT , and CLIP_CNN ). The results are shown in the following Table. Our OT-Attack consistently requires more time than SGA, which is approximately 1.75 times that of SGA, reflecting the added complexity of the proposed method. Please refer to Appendix E for more details.

Q2: How are augmented texts generated?

We sincerely appreciate the reviewer's attention to text augmentation. In the current study, we adhered to the five standard captions provided by the dataset to ensure experimental comparability and did not perform text augmentation. However, more detailed textual descriptions could be generated by leveraging VLP models such as MiniGPT-V, LLaVA, or InstructBLIP to describe images from multiple perspectives or simply through synonym substitution. We believe that more comprehensive textual information could enhance adversarial samples' transferability, providing a valuable direction for future research.

Dear Reviewer yYgK,

Thank you once again for taking the time to review our paper. We deeply value your insightful comments and are sincerely grateful for your efforts.

We kindly request you to review our reply to see if it sufficiently addresses your concerns. Your feedback means a lot to us.

Thank you deeply, Authors

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1: The presentation is not optimal. (1) The citation style is wrong. (2) There is no result on ChatGPT-4 and Bing.

Thank you for pointing this out. (1) We have revised the citation style in the revised version. (2) We randomly sample 100 images from Flickr30K to generate adversarial examples by using SGA and our OT-Attack. Then we evaluate them on ChatGPT4 and Bing. The results are shown in the following Table. It highlights the superiority of our OT-Attack, achieving significantly higher ASR rates of 24% on ChatGPT4 and 30% on Bing, compared to only 7% and 8% by SGA. Please refer to Appendix G for more details.

W2: OT optimization introduces additional computational cost. How much does the computational cost differ between SGA and OT-Attack?

Thank you for the feedback. Following your suggestion, we compute the computational cost (minutes) and compare the OT-Attack with SGA across four models (ALBEF, TCL, CLIP_VIT , and CLIP_CNN ). The results are shown in the following Table. Our OT-Attack consistently requires more time than SGA, which is approximately 1.75 times that of SGA, reflecting the added complexity of the proposed method. Please refer to Appendix E for more details.

W3: Sensitivity to the regularization hyperparameter is unclear. How does the hyperparameter affect the attack success rate (ASR) and computational cost? Does more accurate OT optimization lead to better ASR?

The experimental settings of OT-Attack adhered to the default configurations in PLOT. This choice was primarily due to the insensitivity of OT-Attack to experimental parameters, as we will validate through ablation studies. In these studies, we independently evaluate the impact of three parameters $\lambda$, convergence threshold (thresh), and the iteration limit of the Sinkhorn algorithm—on the experimental outcomes. ALBEF is used as the source model, and TCL as the target model.

The parameter $\lambda$ balances the minimization of transport cost and plan smoothness. In our experiments, the default value was set to 0.1. To analyze the sensitivity of $\lambda$, we conduct the OT-Attack with different $\lambda$. The results are shown in the following Table. The results indicate that varying $\lambda$ among 0.01, 0.1, and 1 while keeping other conditions constant leads to consistent attack success rates across metrics such as TR R@1, TR R@5, TR R@10, IR R@1, IR R@5, and IR R@10. This demonstrates that OT-Attack is robust to changes in $\lambda$, maintaining stable performance.

In this study, the default settings were $\text{thresh} = 1.00 \times 10^{-2}$ and an iteration limit of 100. First, we analyzed the average number of iterations under $\text{thresh} = 1.00 \times 10^{-2}$, finding that the mean iteration count for generating 1,000 adversarial samples was only 2.3. Further, we examined the error scalar after each iteration. The error scalar starts at the order of $1.00 \times 10^{3}$ after the first iteration, reaches $1.00 \times 10^{-2}$ within two iterations, and decreases to $1.00 \times 10^{-3}$ or $1.00 \times 10^{-4}$ after three iterations. This analysis indicates that $\text{thresh}$ should range between $1.00 \times 10^{3}$ and $1.00 \times 10^{-6}$. If $\text{thresh}$ exceeds $1.00 \times 10^{3}$, the Sinkhorn algorithm converges in just one iteration. We also conduct OT-Attack with different threshold values. The results are shown in the following Table. It reveals minimal differences in image-text matching metrics when the source model is ALBEF, and the target model is TCL, confirming that OT-Attack is insensitive to thresh and demonstrates good stability.

Additionally, We also conduct OT-Attack with different iteration limit values. The results are shown in the following Table. It demonstrates that the attack efficacy of OT-Attack remains stable.

In summary, OT-Attack exhibits robustness to hyperparameter variations, delivering stable attack performance under different parameter settings, and significantly outperforms SGA. Please refer to Appendix F for more details.

Dear Reviewer uWjN,

Thank you once again for taking the time to review our paper. We deeply value your insightful comments and are sincerely grateful for your efforts.

We kindly request you to review our reply to see if it sufficiently addresses your concerns. Your feedback means a lot to us.

Thank you deeply, Authors

[W1-2] While 100 images are small, this result looks nice.

Thank you for the feedback. Due to limited time, we randomly selected 100 images for comparison experiments. In the future, we will use more images to verify the effectiveness of our method.

[W2] How many adversarial samples were generated for Table 9?

Following the default setting of SGA, we also adopt 1000 images of Flickr30K to conduct experiments. Hence, 1000 adversarial examples are generated for Table 9.

[W3-1] If the mean iteration count is only 2.3, why does OT-Attack require 1.75 times the computational cost of SGA?

We need to clarify that the mean iteration count is only 2.3 refers to the number of iterations required for each iteration in OT-Attack, not the number of iterations required for the entire OT-Attack. The optimal transfer matrix T is dependent on both the image feature matrix and the text feature matrix. As a result, the gradient computation and backpropagation process require a certain amount of time.

[W3-2] What is the computational cost when the Iteration Limit is set to 1? Given that attack performance is similar with varying hyperparameters, it would be helpful for future work to include the hyperparameter settings for the fastest OT-Attack.

Thank you for the feedback. Following the suggestion, we conduct the fast OT-Attack with the Iteration Limit =1 . The results are shown in the following Table. While SGA is the fastest at 34 minutes, our Fast-OT-Attack achieves a good balance, taking 49 minutes with likely improved performance. OT-Attack, though the most computationally intensive at 60 minutes, likely delivers the best performance.

[W3-3-1] Why does a larger $\lambda$, which should improve optimization accuracy, not result in better OT-Attack performance?

As shown in the following table. A larger $\lambda$ indeed improves attack performance. For instance, increasing $\lambda$ from 0.01 to 1 boosts TR @R1 from 52.27% to 52.90%. However, a larger $\lambda$ also drives the Sinkhorn algorithm closer to the classical optimal transport problem, reducing its convergence speed, increasing the number of Sinkhorn iterations, and resulting in higher time costs. This deeper analysis highlights the significance of exploring parameter settings to achieve a balance between attack success rate and computational cost, a promising direction for future research.

[W3-3-2] A smaller error scalar is expected to improve performance. However, the threshold of $1.0 \times 10^3$ seems to yield results similar to the default (likely no statistically significant difference). Could you clarify this? Some related questions: What is the error scalar before optimization?

Thank you for pointing this out. The error scalar after the first optimization of most samples is at the $10^3$ level, which is greater than $1.0 \times 10^3$. After another optimization, the error scalar will drop sharply to the $\times 10^{-2}$ level. Therefore, the performance difference between $1.0 \times 10^3$ and $1.0 \times 10^{-2}$ is not large. As presented in Algorithm 1 of the submitted manuscript, the error scalar is a temporary variable we have defined. This variable only comes into existence during the optimization transfer process and does not have a predefined value prior to that stage.

Thank you for the feedback. Following the suggestion, we conduct the comparative experiments. The results are shown in Following table. From the table, it is evident that the OT method using the transportation matrix T significantly outperforms directly applying power transformations to the similarity matrix (p=2,3,4) in certain metrics, such as CLIP-CNN's TR R@1 and IR R@1. The advantage of the transportation matrix T lies in its ability to leverage optimal transport theory to find a globally optimal coupling between image and text features, which can improve the adversarial transferability.

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1: The evaluation baselines are not comprehensive in experimental design and lack comparison with VLAttack.

Thank you for your feedback. We compare the proposed OT-Attack with VLAttack, which focuses on enhancing the transferability of attacking pretrained vision-language models. In our experiments, ALBEF is employed as the source model and TCL is used as the target model. The results are shown in the following Table. The results show that our OT-Attack outperforms VLAttack across all metrics, with notable improvements such as 52.37% vs. 43.2% in TR@1 and 61.05% vs. 52.04% in IR@1, demonstrating the superiority of our method in improving the adversarial transferability. Please refer to Appendix D for more details.

W2 & Q2 : The paper lacks a discussion on the computational overhead caused by optimal transport.

Thank you for the feedback. Following your suggestion, we compute the computational cost (minutes) and compare the OT-Attack with SGA across four models (ALBEF, TCL, CLIP_VIT , and CLIP_CNN ). The results are shown in the following Table. Our OT-Attack consistently requires more time than SGA, which is approximately 1.75 times that of SGA, reflecting the added complexity of the proposed method. Please refer to Appendix E for more details.

W2 & Q1: The paper didn't discuss the convergence of optimal transport for vision-language model attack either.

The convergence of the Sinkhorn algorithm ensures that the transport matrix $P = \text{diag}(r) \cdot K \cdot \text{diag}(c)$ satisfies the prescribed marginal distributions $u$ and $v$. Convergence is typically assessed by measuring the change in the scaling factors r or c between successive iterations, where the error metric (e.g., $\| r^{(k)} - r^{(k-1)} \|$) must fall below a predefined threshold $\epsilon$. Alternatively, convergence can be determined by the deviation of the row and column sums of $P$ from $u$ and $v$. A maximum iteration limit is often imposed to prevent infinite loops. In this study, Sinkhorn convergence is determined based on the condition that the variation in the row normalization factor r between consecutive iterations is below the predefined threshold $\epsilon, i.e., \| r^{(k)} - r^{(k-1)} \| < \epsilon$. Please refer to Appendix F for more details.

W3: Formatting problem exists in the current manuscript: e.g., L251, 'transport'. L411, '1 3 5 7' is not centered in table 4.

Thank you for pointing this out. We have revised them in the revised version. Please refer to L251 of Page 5 and L411 of Page 8.

Dear Reviewer vFmV,

Thank you once again for taking the time to review our paper. We deeply value your insightful comments and are sincerely grateful for your efforts.

We kindly request you to review our reply to see if it sufficiently addresses your concerns. Your feedback means a lot to us.

Thank you deeply, Authors