Considering the response exceeds 5,000 characters, we will submit it in two parts.

Part 1

Thank you for recognizing the value of our research topic and the quality of our writing, which encourages us to continue studying this issue. In response to your questions, we provide the following analysis:

1. Analysis of our attack method. By inducing the information entropy of the model's logits, attention scores, and hidden states to gradually increase, we can force the model to output texts that do not conform to reality. This is because information entropy is an indicator of the uncertainty of information content. When information entropy increases, it signifies that the model's uncertainty regarding prediction results also increases, thereby reducing the logical coherence of the output text.
   • Implementing a joint attack with different weights for these three losses offers the following advantages:
     • Multi-angle attack: By attacking logits, attention scores, and hidden states simultaneously, the model's prediction process can be disrupted from multiple angles. Consequently, even if the model exhibits strong robustness in a particular aspect, it would struggle to withstand attacks from various directions.
     • Weight adjustment: By adjusting the weights of different losses, optimization can be achieved according to the specific characteristics and attack objectives of the model. For instance, if the model is sensitive to logits perturbation, the weight of logits loss can be increased to improve the attack's effectiveness.
     • Interactive influence: Logits, attention scores, and hidden states interact with each other within the model. A joint attack can exploit this interaction to enhance the attack's effectiveness. For example, by increasing the entropy of attention scores, the model's focus can be dispersed during the prediction process, thereby affecting the computation of hidden states and logits and reducing the logical coherence of the output text.
     • Stronger attack performance: Compared to individual attacks, a joint attack can achieve higher attack effectiveness in a shorter time, as it operates simultaneously in multiple directions, improving attack efficiency.
   • All three methods are relatively effective, with logits-based attacks appearing to be the best from the perspective of attack success rate. However, from the standpoint of adversarial training, models may find it more challenging to defend against joint attacks. The advantages and disadvantages of executing these three methods individually are as follows:

     • Logits-based:

       Advantages: Directly affects the model's output layer, enabling rapid alteration of the model's prediction results within a short time. For some models, this attack method may yield better results. Disadvantages: Targets only the output layer, potentially failing to fully exploit other information within the model. Additionally, if the model possesses strong robustness, larger perturbations may be required to achieve the desired attack effect.

     • Attention-based:

       Advantages: By altering the model's attention distribution, the model can be forced to focus on irrelevant or insignificant information, thus reducing the logical coherence of the output text. This method leverages the model's internal attention mechanism, making the attack more targeted. Disadvantages: The attack's effectiveness may be influenced by the model's structure and parameter settings. For some models, altering the attention distribution may not be sufficient to produce significant attack effects.

     • Hidden states-based:

       Advantages: Directly affects the model's internal representations, allowing for deeper disruption of the model's prediction process. This method may yield substantial attack effects with relatively small perturbations. Disadvantages: The attack's effectiveness may be influenced by the model's complexity and robustness. Furthermore, due to the highly abstract nature of the model's internal representations, determining an effective attack strategy may be challenging.

Part 2

2. Factors affecting the performance of MIE. The main factors affecting this method include perturbation size, the number of iterations, and different loss ratio values. The ablation experiment results for this part are as follows:

   • Perturbation size: As the perturbation increases, the attack effect becomes stronger. However, when the perturbation exceeds 8, the improvement in attack effect becomes weaker.

   • Steps of iterations: As the number of iterations increases, the attack effect gradually strengthens. As shown in Appendix A.1, with an increased number of iterations, the model may even generate meaningless descriptions.

   • Different loss ratios: On Blip, with $\lambda_1$ fixed at 0.5 and varying $\lambda_2,\lambda_3$, the results can be visualized using a heatmap, which reveals that the optimal resutls are concentrated around the ratio of $\lambda_1:\lambda_2:\lambda_3=0.5:0.06:0.06(\approx 8:1:1)$

     $\epsilon ( 1/255)$	1	2	4	8	16	32	64
     MIE	28.01	23.83	20.67	17.80	17.79	17.77	17.99

     Steps	5	10	20	30	40	50	60	70	80	90	100
     MIE	21.02	19.92	18.84	19.31	18.75	18.64	18.45	18.28	18.18	17.95	17.80

     $\lambda_2$,$\lambda_3$	0.025	0.05	0.06	0.1	0.2	0.5	0.75	1
     0.025	18.85	18.34	18.99	17.98	18.39	19.09	19.60	19.65
     0.05	18.90	18.68	18.35	19.13	18.06	18.59	19.30	19.79
     0.06	18.99	18.07	17.75	18.25	18.22	18.25	18.67	19.14
     0.1	19.53	18.99	18.18	18.10	18.37	18.70	18.98	19.48
     0.2	19.02	18.63	18.36	18.87	18.97	18.48	18.56	19.29
     0.5	20.35	18.95	18.78	18.24	18.24	18.80	19.03	20.67
     0.75	20.15	19.22	19.44	19.37	19.33	19.25	19.29	20.76
     1	20.86	20.30	20.25	20.69	20.26	20.19	20.78	20.00

3. Effectiveness of adversarial training. Following conventional adversarial training, we find that large VLMs do not exhibit significant adversarial robustness for unseen samples. Our MIE attack based on autoregressive generation of pseudo-labels has a strong attack capability. We also compared it with some other methods, inlcuding Carlini 2023 (performing targeted attacks using random targets), Schlarmann 2023 (utilizing descriptions of clean samples as ground-truth labels), and Nayyer 2022 ( a GAN-based method). Note that these attacks did not use the exact same experimental settings. The results across different models also show that the adversarial robustness of existing large VLMs is generally weak. In the future, we will delve into the effectiveness of adversarial training for VLMs.

   Model	Clean	Carlini2023[1]	Schlarmann2023[2]	Nayyer2022[3]	MIE
   BLIP	29.79	20.53	19.87	24.36	17.80
   BLIP-2	30.72	24.58	24.06	27.78	21.39
   InstructBLIP	31.36	24.31	23.80	25.32	21.65
   LLaVA	31.52	24.78	24.12	25.79	21.41
   MiniGPT-4	31.44	24.97	23.16	24.12	21.11

In summary, our proposed MIE method can be applied to various Transformer-based VLMs. With its ease of use and effectiveness, it can serve as a benchmark for evaluating VLMs' robutness.

The relevant experimental analysis will be updated in the paper.

[1] Carlini, Nicholas, et al. "Are aligned neural networks adversarially aligned?." arXiv preprint arXiv:2306.15447 (2023).

[2] Schlarmann, Christian, and Matthias Hein. "On the adversarial robustness of multi-modal foundation models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Aafaq, Nayyer, et al. "Language model agnostic gray-box adversarial attack on image captioning." IEEE Transactions on Information Forensics and Security 18 (2022): 626-638.

We sincerely thank you for your time and effort in reviewing our paper and providing constructive comments. We have carefully read and addressed your feedback. The discussion phase is nearing its end, but we have not yet received your response. We would be more than happy to engage further if you have any additional questions or suggestions.

Thank you for your thorough reading of our paper. In response to your concerns, we provide the following analysis.

1. Comparison with relevant and feasible works. We have added some comparative experiments, including Carlini 2023 (performing targeted attacks using random targets), Schlarmann 2023 (utilizing descriptions of clean samples as ground-truth labels), and Nayyer 2022 ( a GAN-based method). The results show that our MIE method is more effective, taking full advantage of the non-targeted nature to increase the perturbation space. As for Yunqing2023 [4], it is a targeted black-box attack with strong dependency on the target text, which is significantly different from our experimental setting; thus, we dot not include it in the comparison.

   Model	Clean	Carlini2023[1]	Schlarmann2023[2]	Nayyer2022[3]	MIE
   BLIP	29.79	20.53	19.87	24.36	17.80
   BLIP-2	30.72	24.58	24.06	27.78	21.39
   InstructBLIP	31.36	24.31	23.80	25.32	21.65
   LLaVA	31.52	24.78	24.12	25.79	21.41
   MiniGPT-4	31.44	24.97	23.16	24.12	21.11

2. Reason for choosing maximizing entropy as the attack method. We would like to emphasize that the reason for selecting the maximization of entropy as the optimization objective is to make the model's understanding of the input image more confusing. By maximizing information entropy, we are actually pursuing the maximum confusion in the model's internal information during answer generation, thereby increasing the model's uncertainty. Although there are indeed multiple ways to calculate loss, such as maximizing the norm, not all of these methods are directly related to our attack starting point. For example, the focus of maximizing the norm is on the overall magnitude of the model's internal outputs rather than uncertainty. Although in some cases, maximizing the norm may lead to increased uncertainty in the model's internal information, this connection is not as direct and clear as with entropy. We have added experiments based on the norm, and the results show that MIE has a stronger attack effect.

   Model	Clean	Norm-based	MIE
   BLIP	29.79	20.81	17.80
   BLIP-2	30.72	22.97	21.39
   InstructBLIP	31.36	23.74	21.65
   LLaVA	31.52	24.53	21.41
   MiniGPT-4	31.44	23.65	21.11

3. Ablation study on loss coefficients. Table 1 in the paper shows the results of individual attacks in the last four columns. For the BLIP model, we conduct ablation experiments by fixing $\lambda_1=0.5$ and varying $\lambda_2$ and $\lambda_3$. The results can be visualized using a heatmap, which reveals that the optimal resutls are concentrated around the ratio of $\lambda_1:\lambda_2:\lambda_3=0.5:0.06:0.06(\approx 8:1:1)$. Additional ablation experiments can be found in Reviewer 4's comments.

   $\lambda_2$,$\lambda_3$	0.025	0.05	0.06	0.1	0.2	0.5	0.75	1
   0.025	18.85	18.34	18.99	17.98	18.39	19.09	19.60	19.65
   0.05	18.90	18.68	18.35	19.13	18.06	18.59	19.30	19.79
   0.06	18.99	18.07	17.75	18.25	18.22	18.25	18.67	19.14
   0.1	19.53	18.99	18.18	18.10	18.37	18.70	18.98	19.48
   0.2	19.02	18.63	18.36	18.87	18.97	18.48	18.56	19.29
   0.5	20.35	18.95	18.78	18.24	18.24	18.80	19.03	20.67
   0.75	20.15	19.22	19.44	19.37	19.33	19.25	19.29	20.76
   1	20.86	20.30	20.25	20.69	20.26	20.19	20.78	20.00

In summary, our proposed MIE method can be rapidly applied to various Transformer-based vision-language models. Owing to its usability and effectiveness, it can serve as a new benchmark for evaluating the robutness of VLMs.

The relevant experimental analysis will be updated in the paper.

[1] Carlini, Nicholas, et al. "Are aligned neural networks adversarially aligned?." arXiv preprint arXiv:2306.15447 (2023).

[2] Schlarmann, Christian, and Matthias Hein. "On the adversarial robustness of multi-modal foundation models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Aafaq, Nayyer, et al. "Language model agnostic gray-box adversarial attack on image captioning." IEEE Transactions on Information Forensics and Security 18 (2022): 626-638.

[4] Zhao, Yunqing, et al. "On evaluating adversarial robustness of large vision-language models." arXiv preprint arXiv:2305.16934 (2023).

We sincerely thank you for your time and effort in reviewing our paper and providing constructive comments. We have carefully read and addressed your feedback. The discussion phase is nearing its end, but we have not yet received your response. We would be more than happy to engage further if you have any additional questions or suggestions.

Thank you for acknowledging the novelty of our attack method. In response to your concerns, we provide the following analysis.

1. Technical contributions. Existing attacks on large Very Large Models (VLMs) generally adopt similar approaches, utilizing teacher forcing to calculate the loss of target texts. These methods are relatively simple but demonstrate a certain level of effectiveness. Our method based on maximizing information entropy provides a new attack perspective and is highly effective. Precisely because the technical implementation of MIE is relatively straightforward, it can encourage more researchers to focus on the adversarial robustness of VLMs.

2. Comparison with relevant and feasible works. Below are our comparative experimental results inlcuding Carlini 2023 (performing targeted attacks using random targets), Schlarmann 2023 (utilizing descriptions of clean samples as ground-truth labels), and Nayyer 2022 ( a GAN-based method).:

   Model	Clean	Carlini2023[1]	Schlarmann2023[2]	Nayyer2022[3]	MIE
   BLIP	29.79	20.53	19.87	24.36	17.80
   BLIP-2	30.72	24.58	24.06	27.78	21.39
   InstructBLIP	31.36	24.31	23.80	25.32	21.65
   LLaVA	31.52	24.78	24.12	25.79	21.41
   MiniGPT-4	31.44	24.97	23.16	24.12	21.11

3. Selection of the validation set. Our validation set comprises 1,000 randomly selected images, which inherently provide a certain level of generalization. This selection method is employed to maintain consistency with related works in the adversarial attack domain. Furthermore, we conduct ablation experiments by fixing $\lambda_1=0.5$ and varying $\lambda_2$ and $\lambda_3$ for the BLIP model. The results can be visualized using a heatmap, which reveals that the optimal resutls are concentrated around the ratio of $\lambda_1:\lambda_2:\lambda_3=0.5:0.06:0.06(\approx 8:1:1)$. Additional ablation experiments can be found in Reviewer 4's comments.

   $\lambda_2$,$\lambda_3$	0.025	0.05	0.06	0.1	0.2	0.5	0.75	1
   0.025	18.85	18.34	18.99	17.98	18.39	19.09	19.60	19.65
   0.05	18.90	18.68	18.35	19.13	18.06	18.59	19.30	19.79
   0.06	18.99	18.07	17.75	18.25	18.22	18.25	18.67	19.14
   0.1	19.53	18.99	18.18	18.10	18.37	18.70	18.98	19.48
   0.2	19.02	18.63	18.36	18.87	18.97	18.48	18.56	19.29
   0.5	20.35	18.95	18.78	18.24	18.24	18.80	19.03	20.67
   0.75	20.15	19.22	19.44	19.37	19.33	19.25	19.29	20.76
   1	20.86	20.30	20.25	20.69	20.26	20.19	20.78	20.00

In conclusion, given the usability and effectiveness of our proposed MIE method, it can serve as a new benchmark for evaluating the robustness of VLMs.

The relevant experimental analysis will be updated in the paper.

[1] Carlini, Nicholas, et al. "Are aligned neural networks adversarially aligned?." arXiv preprint arXiv:2306.15447 (2023).

[2] Schlarmann, Christian, and Matthias Hein. "On the adversarial robustness of multi-modal foundation models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Aafaq, Nayyer, et al. "Language model agnostic gray-box adversarial attack on image captioning." IEEE Transactions on Information Forensics and Security 18 (2022): 626-638.

Part 2

3. Non-targeted attacks pose weaker constraints compared to targeted attacks.

   • Non-targeted attacks do not aim to make VLMs produce specific outputs, resulting in a larger solution space for numerical optimization. This is also a reason why targeted attacks may be less effective.
   • MIE can serve as a powerful method for assessing the robustness of VLMs. As the attack progresses, the model's output may become nonsensical, consisting of repetitive random words. This indicates that the VLM's robustness is insufficient, as it cannot generate coherent sentences, which in turn diminishes user experience and weakens user trust. In scenarios with high security requirements, the lack of robustness can lead to severe consequences.
   • We also experimentally found that conventional adversarial training cannot defend against our attack, indicating that current VLMs are extremely vulnerable to our attack method.

4. Ablation study on loss coefficients. Table 1 in the paper shows the results of individual attacks in the last four columns. For the BLIP model, we conduct ablation experiments by fixing $\lambda_1=0.5$ and varying $\lambda_2$ and $\lambda_3$. The results can be visualized using a heatmap, which reveals that the optimal resutls are concentrated around the ratio of $\lambda_1:\lambda_2:\lambda_3=0.5:0.06:0.06(\approx 8:1:1)$. Additional ablation experiments can be found in Reviewer 4's comments.

   $\lambda_2$,$\lambda_3$	0.025	0.05	0.06	0.1	0.2	0.5	0.75	1
   0.025	18.85	18.34	18.99	17.98	18.39	19.09	19.60	19.65
   0.05	18.90	18.68	18.35	19.13	18.06	18.59	19.30	19.79
   0.06	18.99	18.07	17.75	18.25	18.22	18.25	18.67	19.14
   0.1	19.53	18.99	18.18	18.10	18.37	18.70	18.98	19.48
   0.2	19.02	18.63	18.36	18.87	18.97	18.48	18.56	19.29
   0.5	20.35	18.95	18.78	18.24	18.24	18.80	19.03	20.67
   0.75	20.15	19.22	19.44	19.37	19.33	19.25	19.29	20.76
   1	20.86	20.30	20.25	20.69	20.26	20.19	20.78	20.00

In summary, our proposed MIE method has two main advantages:

1. Usability: The MIE method carries out attacks in an unsupervised manner, with weaker constraints and no reliance on true labels, making it more convenient to implement. This can encourage more researchers to focus on the robustness of VLMs.
2. Effectiveness: Supplementary experiments show that MIE has a stronger attack effect than existing methods, achieving almost a 100% attack success rate, and conventional adversarial training cannot defend against it.

This usability and effectiveness make our method a new benchmark for evaluating the adversarial robustness of VLMs. The relevant experimental analysis will be updated in the paper.

[1] Carlini, Nicholas, et al. "Are aligned neural networks adversarially aligned?." arXiv preprint arXiv:2306.15447 (2023).

[2] Schlarmann, Christian, and Matthias Hein. "On the adversarial robustness of multi-modal foundation models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Aafaq, Nayyer, et al. "Language model agnostic gray-box adversarial attack on image captioning." IEEE Transactions on Information Forensics and Security 18 (2022): 626-638.

Considering the response exceeds 5,000 characters, we will submit it in two parts.

Part 1

Thank you for providing insightful feedback, which is greatly helpful for improving our paper. Regarding the four questions you raised, here is our analysis:

1. Key Contribution. We propose the MIE method, which focuses on non-targeted attacks, characterized by weaker constraints and a larger optimization space. Unlike previous methods, such as [1] and [2], which involve targeted optimization of adversarial samples towards specific directions, our approach is more suitable for image description tasks. Supplementary experiments demonstrate that the attack performance of MIE surpasses that of comparative methods. We plan to revise and upload a new version of the paper shortly. MIE is the first untargeted attack method that does not rely on the true descriptions of images, exhibiting strong generalization capabilities.
2. Feasibility of targeted attacks using random targets. While it is computationally feasible to implement non-targeted attacks using targeted attack methods as subroutines, it is insufficient for image description tasks. Unlike traditional classification tasks, images do not have only one correct description, especially for complex images with multiple description angles. For instance, "it's a sunny day" and "the sunshine is so bright" are textually inconsistent but semantically equivalent. Therefore, the success rate of such random targeted attacks is relatively low. Below are our comparative experimental results inlcuding Carlini 2023 (performing targeted attacks using random targets), Schlarmann 2023 (utilizing descriptions of clean samples as ground-truth labels), and Nayyer 2022 ( a GAN-based method):

   Model	Clean	Carlini2023[1]	Schlarmann2023[2]	Nayyer2022[3]	MIE
   BLIP	29.79	20.53	19.87	24.36	17.80
   BLIP-2	30.72	24.58	24.06	27.78	21.39
   InstructBLIP	31.36	24.31	23.80	25.32	21.65
   LLaVA	31.52	24.78	24.12	25.79	21.41
   MiniGPT-4	31.44	24.97	23.16	24.12	21.11

We sincerely thank you for your time and effort in reviewing our paper and providing constructive comments. We have carefully read and addressed your feedback. The discussion phase is nearing its end, but we have not yet received your response. We would be more than happy to engage further if you have any additional questions or suggestions.