Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models

Q1: Experiments investigating potential defense could make the paper more comprehensive.

A1: Thanks for your suggestion. As shown in the following table, we evaluate the robustness of our adversarial patch with four popular defense methods. Specifically, PatchCleanser [a] is a state-of-the-art certifiable defense against adversarial patches. It uses double masking to certify the prediction. [c,d] are query-based defenses, which are specifically designed for detecting malicious queries by black-box attacks. [b] is reviewer's mentioned defense. It shows that our attack method is robust to them. We will add the experiments in the revision.

[a] PatchCleanser: Certifiably robust defense against adversarial patches for any image classifier. USENIX 2022 [b] Stateful detection of black-box adversarial attacks. S&P 2020 [c] Random noise defense against query-based black-box attacks. NeurIPS 2021 [d] Blacklight: Scalable defense for neural networks against query-based black-box attacks. USENIX 2022

Q2: For comparison with CroPA in Table 4, the CroPA paper stated that they use OpenFlamingo rather than Flamingo. It would be good to specifically specify which model is being used for evaluation.

A2: We utilize the same open-source version for fair comparison in Table 4. We will revise the description and add the reference.

Q3: All the target texts are rather few words. It would be more comprehensive to include longer sentences. Will longer sentences increase the query budgets to achieve a similar performance as these short sentences?

A3: Since we evaluate the text semantic similarity to assign importance weights and assess the attack performance, our proposed attack is not sensitive to the length of the target response text. That means, longer sentences will not increase the query budgets and our pre-set query budget is sufficient to handle the cases. We conduct more experiments on longer sentences in the table below. We can find that our attack can also achieve great attack performance on long output targets. We will add more experiments and discussion in the revision.

Q4: For targeted attacks, only reporting the semantic similarity scores makes it hard to fully understand the attack performance.

A4: Thanks for your suggestion. Actually, in our paper, we follow the previous work [45] to utilize the semantic similarity for fair evaluation. This metric is more generalizable for evaluating different texts with the same semantics. For example, the output texts "unknown" and "I don't know" share similar semantics, they have relatively higher semantic similarity score but would achieve a low word-level metric. Of course, we can also utilize the ExactMatch and Contain metrics to conduct strict word-level evaluation on our attack to assess the attack success rates. Specifically, the ExactMatch metric determines whether the LVLM output exactly matches the predefined target text, whereas the Contain (similar to BLEU) metric checks whether the output contains the target text. This is especially useful when outputs exceed the predefined target length. We report the corresponding evaluations in the table below, where our method also achieves significant attack performance, demonstrating the effectiveness of our attack.

Q5: It is not a weakness, but I believe the year of the publication for [45] in the reference is wrong.

A5: Thanks for your reminder. We will correct the details of reference [45] in the revision.

Q1: Comparison with non-universal attacks.

A1: Since existing LVLM attackers are implemented in different settings with different models/datasets, we have already provided detailed comparisons in Table 3 and 4 of the paper. Note that, existing methods are non-universal attacks and require prior LVLM knowledge to generate different perturbations for different images. In contrast, our attack solely accesses to the LVLM input/output and can generate a single noised patch to fool all images/prompts, while achieving better attack performance. To provide a more comprehensive comparison, we also re-implement these attacks into our utilized datasets/models/metrics as shown in the table below:  

It shows that our attack is still more adversarial. We think the reason is that our universal patch explicitly learns the general adversarial patterns against LVLMs and effectively estimates gradients solely using positive directions.

Q2: Query efficiency.

A2: The overall 70k queries are utilized to generate the universal patch using 500 images (in Table 10 of the appendix B.5). This means 70k queries cover the processing of all 500 images, rather than each image requiring 70k individual queries. Moreover, as shown in Table 8 of the appendix B.3, our method takes only 4.9h to attack all images/prompts, which is significantly more efficient than previous methods that generate separate perturbations for each image/prompt, e.g., MF-Attack: 3h for only a single perturbation, CroPA: 2.6h for only a single perturbation.

Q3: Opposite directions claim.

A3: We have empirically supported this claim in Table 1 of the paper via the variant "w/o importance". In practice, not all slight additive noises point towards the optimal direction during the gradient estimation, as negative noise may contain decomposed components in the opposite direction. If we treat all noises equally, the "w/o importance" variant performs worse. However, by weakening the negative noise and strengthening the positive noise with positive/negative-aware importances, our full attack achieves the best performance. This demonstrates that not all noise contributes positively to the optimal gradient direction.

Q4: Judge model acquisition.

A4: Actually, the attackers can utilize any text embedding model as the judge model, like BERT, Sentence-BERT. As shown in Figure 5(a) of the paper, our attack is not sensitive to these text encoders. It does not need to be related to the target LVLM.

Q5: Target text selection.

A5: As shown in Table 2 and Figure 8 of the paper, our attack can achieve significant performance on specific target texts. We also conduct more experiments on such types of texts in the table below. We will add more experiments in the revision.

Q6: Patch visibility and physical attacks.

A6: Although the adversarial patch is noticeable to humans, it can effectively fool the LVLMs in the universal setting without using any model details compared to other global noise attacks (which are less obvious). Moreover, as shown in the following table, our adversarial patch is robust to existing defenses.  

Besides, we try to implement two types of physical attack: A. we print and paste the patch on real images and scan it to query the LVLM; B. we print and paste the patch on a flat plane in realistic and collect new images. As shown in the table below, our method can still effectively attack these physical cases.

[a] PatchCleanser: Certifiably robust defense against adversarial patches for any image classifier. USENIX 2022 [b] Stateful detection of black-box adversarial attacks. S&P 2020 [c] Random noise defense against query-based black-box attacks. NeurIPS 2021 [d] Blacklight: Scalable defense for neural networks against query-based black-box attacks. USENIX 2022

Q7: Transferability to other models.

A7: Firstly, we have tested the transferability among four LVLMs in Table 6 of the appendix B.1. Secondly, to evaluate the transfer-attack performance on GPT-4 and Claude, we also provide experiments in the following table. These experiments demonstrate the broader applicability of our attack.

Q1: The paper could benefit from clearer explanations of the technical details and processes.

A1: Thanks for your suggestion. We will add more corresponding explanations in the revision:

(1) For adversarial patch initialization: to achieve successful targeted attacks, there is no prior knowledge of where the patch shall be located and what it looks like. Therefore, for each targeted text, we randomly sample a few patch locations and patterns, and choose a combination of the two with the highest attack performance. In particular, to determine whether a location is good, we randomly sample a set of patterns at that location and examine the average attack performance for this set. Since the quality of a patch location also relies on sampled patterns, we also draw patterns from a uniform distribution to find the optimal patch pattern.

(2) For adversarial patch optimization: Since there is no model gradients or details, we need to estimate the gradient direction for further perturbing the patch pattern. The general Monte Carlo estimations apply a set of random noises on the patch and see whether these noises can change the prediction. These perturbation directions are averaged as the final direction for mutating the patch. However, such a design is not efficient as not all noises point towards optimal direction. To this end, we propose an importance-aware gradient approximation method to adaptively adjust the weights for different samples, based on how these sampled noises can lead to the attacker-chosen text. Specifically, given a set of slight additive noise in each step, we first add them to the patch to assess their positive/negative degrees via our developed LVLM-aware indicator function. Based on this, we then assign corresponding weights to positive/negative noises to strengthen the positive gradient direction while weakening the negative gradient direction. In this manner, we can efficiently and effectively optimize the adversarial patch with optimal estimated gradients. 

Q2: I am curious about the output of the model for a completely corrupted input, such as an image with noise that cannot be recognized by a human. It would be interesting to include this as a baseline.

A2: We have conducted detailed comparisons between our adversarial patch and completely corrupted inputs in Table 3,4 of our paper. In these two tables, although previous LVLM attackers (MF-Attack, CroPA) aim to globally perturb the whole image to improve the imperceptibility, they rely on the LVLM details and gradients for perturbation updation. Instead, our adversarial patch can be optimized in a more challenging but practical setting by solely querying the model, while achieving better attack performance than these LVLM attackers (Ours vs MF-Attack: 0.727 vs 0.646; Ours vs CroPA: 0.69 vs 0.65), demonstrating our effectiveness. Besides, we also try to re-implement these completely corrupted attacks into our utilized datasets/models/metrics as baselines for comparison in the following table. It shows that our attack is still more adversarial as we explicitly and effectively estimate optimal gradient direction for optimizaiton.  

Secondly, previous completely corrupted inputs are diverse and sensitive to their global noises. These noises are specific to different images or prompts and can hardly to be effective in attacking other images/prompts. Instead, as shown in Figure 3 of the paper, our adversarial patch design is more flexible and can easily achieve a more challenging universal attack.

Q3: The paper does not explicitly state this nuance, but it seems this attack aims to mislead the model with an image that is clearly recognizable to humans.

A3: Thanks for your concern. We want to clarify that:

(1) Our main goal is to tackle two critical issues in existing LVLM attackers (not to mislead LVLMs with recognizable perturbations): (i) restricting access to only the LVLM inputs and outputs. (ii) devising a universal adversarial perturbation.

(2) To achieve the above goal, traditional attacks using completely corrupted inputs are ineffective as they require backpropagated gradients for optimization and their perturbations vary across different images/prompts. Therefore we choose to use adversarial patch to hanlde the above two challenging issues, which has been proven to be more effective than previous global noises through our carefully designed patch initialization and optimization processes.

(3) We can further improve the imperceptibility of our attack by using smaller patch, which also achieves competitive performance as in Table 5 of the paper. Besides, our patch pattern is well-optimized via our designed strategy to achieve the challenging targeted attack, which is quite different from ineffective random pure noise.

(4) Moreover, as in the following table, our adversarial patch is robust to existing defenses.  

[a] PatchCleanser: Certifiably robust defense against adversarial patches for any image classifier. USENIX 2022 [b] Stateful detection of black-box adversarial attacks. S&P 2020 [c] Random noise defense against query-based black-box attacks. NeurIPS 2021 [d] Blacklight: Scalable defense for neural networks against query-based black-box attacks. USENIX 2022

Q1: Can the attack achieve the same effect with a very small patch size without adding constraints to the perturbation?

A1: Without adding perturbation constraints, a smaller patch can achieve a similar attack performance. We implement this perturbation constraint to make a fair comparison with existing LVLM attack methods, as shown in Table 3 and 4 of the paper. Although we have conducted ablation studies on different patch sizes in Table 5 of the paper, these experiments are carried out with the pre-set perturbation constraint. We provide ablations without adding perturbation constraints in the following table:

It shows that, without constraint, patch size 48x48 (w/o. constraint) can achieve a similar overall adversarial effect as our main attack setting of patch size 64x64 (w/. constraint) in the paper (0.834 vs 0.837). Moreover, patch size 32x32 without the constraint also outperforms its constrained variant, and even a much smaller patch size 16x16 without the constraint also achieves a very competitive attack performance. This highlights the significant potential of our proposed attack.

Q2: Why do different target labels affect the choice of patch position? Can the patch be fixed in a specific position to achieve the same attack effect?

A2: (1) We attribute this to the fact that different local regions of the same image contribute varying levels of ambiguity to different target labels, which affects the LVLM model’s ability to output specific semantic text. For instance, in Figure 4 of the paper, "I cannot answer" and "I am sorry" have similar semantics and therefore share close patch positions, while the distinct semantic "I hate people" corresponds to a different position.

(2) Moreover, we investigate the performances of a fixed patch position (on the upper left corner of images) in the following table.

This shows that a fixed position can degrade attack performance, indicating that the initial patch position is crucial for targeting specific texts. However, the fixed patch still exhibits a competitive adversarial attack effect, demonstrating the generalizability of our approach.

Q3: Does the length of the target response text affect the effectiveness of this attack?

A3: Since we evaluate the semantic similarity to assess the attack performance in the judge model and indicator function, our attack is not sensitive to the length of the target response text. As shown in Table 2 and Figure 8 of the paper, our attack can also achieve significant attack performance on long target text. We also conduct more experiments on longer target text in the table below:  

It shows that the length of target texts does not affect the attack effectiveness, demonstrating the scalability of our attack.

Q4: What is the practical significance of this type of attack?

A4: The practical significance of our attack lies in four-fold:

(1) In practical cases, our patches have great potential to be pasted on the realistic images/pictures to achieve attacks [e,f]. We implement two types of physical attack: A. we print and paste the patch on real images and scan it to query the LVLM; B. we print and paste the patch on a flat plane in realistic and collect new images. As in the table below, our method can effectively attack these physical cases.  

(2) To reduce our adversarial effect, directly cropping it may destroy the image structure and lose some contents. Moreover, as shown in the following table, our attack is robust to existing detection-based defenses, indicating that it is hard for automatic systems to prevent our attack.  

(3) Our model is practical in real-world LVLM applications as we solely access to the models' input/output. However, existing LVLM attacks severely rely on the prior LVLM model knowledge.

(4) Our universal patch design also demonstrates our scalability. Compared to previous adversarial perturbation attacks, our attack is efficient and effective to fool different images/prompts in a single process.

[a] PatchCleanser: Certifiably robust defense against adversarial patches for any image classifier. USENIX 2022 [b] Stateful detection of black-box adversarial attacks. S&P 2020 [c] Random noise defense against query-based black-box attacks. NeurIPS 2021 [d] Blacklight: Scalable defense for neural networks against query-based black-box attacks. USENIX 2022 [e] Naturalistic Physical Adversarial Patch for Object Detectors. ICCV 2021 [f] DAP: A Dynamic Adversarial Patch for Evading Person Detectors. CVPR 2024