Misusing Tools in Large Language Models With Visual Adversarial Examples

Thank you for your valuable feedback! Here are our responses to your questions and suggestions.

The best result is reported among three trials based on the argument that “attackers will always choose the best-performing adversarial image.” However, the reviewer thinks this may not be reasonable because the attacker is not the user and so cannot control how many times they would repeat the attack. The authors should explain why they think it is feasible to stick to this setting. This is important given the fact that significant randomness during adversarial image training is observed.

It’s 100% true that the attacker has no control over the user, and it’s the exact reason why the attacker wants the image that is the best performing such that it is expected to affect the most victims.

The authors motivate the design of separate losses for response utility and tool invocation (i.e., Equation 2) based on the argument “In real-world conversational systems...we reduce the contribution of the loss term...” First of all, there are comparisons validating the superiority of using such separate losses to the integrated loss (i.e., Equation 1). More specifically, the ablation studies clearly show using a large lambda, i.e. 1, works better, which conflicts with the argument about “reducing the contribution...”.

We will update the text to “attempt to reduce the contribution” as we designed this loss based on intuitions and later experimented with them. During our experiments, we found that lambda_i provides a good tradeoff between imperceptibility and goal achieving (Table 7), while the effect of lambda_r is more ambivalent (Table 6) — while the response utility drops, the goal achieving likelihood for the unrelated set increases during (lambda_r 1.0 -> 0.1) but further decreases in the range of (lambda_r 0.1 -> 0), while mostly stays the same for the related set. Based on the exact values, we believe an attack that has higher response utility is more favorable to the attacker, therefore in the main experiments, we went with a lambda_r = 1.0. Thanks for pointing this out.

It is said that “the l2 norm is computed with regard to each color channel separately”. However, to the best knowledge of the reviewer, in the literature of adversarial examples, it is indeed computed not separately. Could the authors explain why they chose this unusual setting?

It is actually a historical issue where our initial attempts adopt different normalization coefficients separately for each channel in the preprocessing stage, making us also calculate the l2 norm separately. Given that using both formats can achieve the goal of minimizing the difference between the adversarial image and the original one, we kept the previous way of computing and reported it in the paper for reproducibility.

Considering that the threat model follows the typical prompt injection, it seems unnecessary to highlight its differences from “jailbreaking”. Therefore, the authors are encouraged to tune down some related claims.

Thanks for your suggestion. We have shortened the discussion about the jailbreaking issue in section 2 to make room for other revisions.

The fact that only three images are tested should be mentioned in the main text rather than only in the appendix.

Thanks for pointing this out. We will make this more explicit in the main text. We picked the three images from different sources (we described in the appendix that one the logo of the visual language model we are using, one generated through stable diffusion, and one from online sources), hoping that the present a good coverage over image types. We happily triple the number of images and provide six more examples in Table 9 (Appendix A.9), including two from the Imagenet dataset, and two more each coming from model generations and online sources. We couldn’t conduct a larger experiment with more images due to time constraints but we hope that this provides a good indicator that the attack method would work generally on many images.

The proposed method seems to be straightforward, which is essentially the way of injecting a backdoor. It is unclear how this method can help generalize the trigger to other prompts.

We refer the reviewer to the response for Review tv67 above for reasons as to why our contribution matters. To briefly summarize that response, our contribution is in the computer security space – we show a new threat model for multi-modal LLMs. The fact that an attack is straightforward is a big advantage because attackers always choose the path of least resistance. Furthermore, given that there is 10 years of work on improving visual adversarial example attacks, all of that is now applicable to multi-modal LLMs – this is a formidable arsenal for attackers targeting LLMs. We view our work as opening the flood-gates of attacks on LLMs. It also serves as strong motivation to start investing in research to defend against these attacks.

Table 4 and 5 establishes that our method generalizes to multiple types of prompts and function invocation syntaxes. Beyond that, with regards to the method itself, being able to generalize our attack to various or even arbitrary text prompts provided by the victim user with a decent success rate (on desired tool invocation), while maintaining the response and image stealthiness is a non-trivial task itself and has never been accomplished by any prior work in this domain. Additionally, an attack that is not prohibitively complicated in its nature means it’s more dangerous from a security perspective since it’s more easily deployable and more likely to be used by real world attackers.

Since the goal is to trigger the misusage of tools, why limiting the adversarial perturbation and enhancing the generalizability are important? The adversary only needs one prompt to trigger the malicious usage of the tools.

Attackers ultimately want the attack to be spread wide and long enough to affect as many victims as possible. The image and response stealthiness are critical in this sense because victims will otherwise easily spot the abnormality of the image or the conversation and may even report this to the model provider.

It is unclear what are the implications of these 5 attack objectives. Does the selection of these attack objectives have an impact on the attack performance? What will happen if more attack objectives are included?

We chose these five attack objectives to simulate the real-world behavior as realistically as possible by following invocation format from real world systems (see Section 3.1). In fact, the string <function.delete_email which="all"> is the function call format specified in an older version of Microsoft Semantic Kernel. And the Expedia invocation json is an exact copy of the one generated by ChatGPT (and json is the standard format for any plugin calls in ChatGPT). We believe it’s a fair claim that the textual format attack objectives we’ve tested does reflect practical scenarios and demonstrates the potential of our attack if they could be applied on real-world systems. We have revised the paper to clarify. Also, these five attack objectives represent increasing levels of complexity and difficulty to generate the desired attack and serve as examples to understand whether it is still possible and how easy/difficult it is when the target texts to be generated by LLMs are non natural and complicated.

The evaluation is unsatisfactory. L_p norm is not provided. Also, it is important to show the chances that the malicious tool is triggered when the adversarial example and prompt pair are not present, or only one of these two is present.

We additionally provide the L_2 norm and L_inf norm between the adversarial image and its original in Table 4. (Note that Table 5 corresponds to the same adversarial images thus the L_p norms would be the same if provided). Additionally, in appendix Table 7, we provide the L_p norms to understand the quantities when the SSIM score is low; in appendix figure 5, a plot of SSIM values and L_2 norm for experiments in the paper is provided, which shows that SSIM is almost (inverse) proportional to the L_2 norms (as expected).

Regarding the other scenarios of the attack: we are interested in the scenario when the attacker specified tool invocation should appear when the adversarial example is present with an arbitrary prompt provided along with it. When there is no adversarial image provided but only a text prompt, the LLM should respond to the prompt as usual. In the rare case when the user provides no text prompt provided along with the adversarial image, the LLM we tested would respond as if there is a prompt asking “describe this image”, and for several of our trained adversarial images, also generate the tool invocation.

Thank you for your valuable feedback! Here are our responses to your questions and suggestions.

Only a single model LLaMA Adapter is tested. This makes the scope of the evaluation look somewhat narrow. I suggest the authors also consider other VLMs like Minigpt-4 [1], Instruct-Blip [2], and LLaVA [3]. This can make the evaluation more convincing.

We acknowledge this and we are currently working on experiments on other multimodal models. The result will be presented in the updated version soon.

Lack of case studies on real LLM-integrated applications. The paper mentioned that LangChain and Guidance facilitate the development of such integrations. But, the paper did not provide a single instance of this to illustrate the practical risks of the proposed attack. In the whole paper, what the attack did was just to induce the generation of something like <function.delete_email which="all"> in a purely textual form, which is essentially nothing different from previous NLP attacks that induced certain targeted generations. In my opinion, the novelty of this paper only comes from the illustration of the "practical risks" of such attacks --- because the resources controlled by LLMs can now also be missed to induce broader harms. However the paper did not provide a real example of this. The whole evaluation is still in a purely textual form, judging whether things like <function.delete_email which="all"> are generated... In practice, the LLMs integrated systems may be more complicated than this conceptual form. The paper did not go deep into this.

We tried to simulate the real-world behavior as realistically as possible by following invocation format from real world systems (see Section 3.1). In fact, the string <function.delete_email which="all"> is the function call format specified in an older version of Microsoft Semantic Kernel. And the Expedia invocation json is an exact copy of the one generated by ChatGPT (and json is the standard format for any plugin calls in ChatGPT). We believe it’s a fair claim that the textual format attack objectives we’ve tested does reflect practical scenarios and demonstrates the potential of our attack if they could be applied on real-world systems. We have revised the paper to clarify. Also, compared to previous work’s targets, a crucial difference of our attack goal is that these tool invocation texts are non natural. We believe it is important to understand whether it is still possible and how easy/difficult it is when the target texts are non natural. Our evaluation provides some answers to this question, with various length of the target text as another dimension.

Inaccurate Literature Review. There is a factual error in the literature review. Qi et. al. [4] did not show transferability to closed-source models. On the other hand, Carlini et. al. [5] along with Qi et. al. [4] are earlier works showing the usage of visual adversarial examples to hack VLM, which may also need to be noted.

Thank you for pointing these out! We have corrected these errors in the revised version.

Thank you for your feedback. We believe there might be a miscommunication of our motivation and contributions. We attempt to better clarify as follows.

This paper assumes that interaction with the tools occurs through an instruction line, followed by normal question answering, as illustrated in Figure 1. Is this setting realistic? What does a real system look like, and how do these VLMs interact with downstream tools like email? Please provide an illustration of why the task in Figure 1 is realistic.

Commercial LLMs like ChatGPT and Google Bard support visual images as an input modality by default. Their interaction method assumes an image being uploaded with a prompt alongside, exactly like what we have described in Figure 1. They are also integrating a large number of extensions/plugins which include ones necessary for our attack objective such as email connectors , google workspace, Expedia, etc. A version of our attack on these models would successfully invoke the plugins/extensions integrated by the victim user. We note that the attacker-desired invocation string can be changed to invoke any other plugin/extension than the 5 attack objectives we have shown in Table 1). Therefore, our proposed attack scenario is highly realistic even at this moment. The trend of more third-party customized versions of GPT (GPTs) that will incorporate more powerful tools to LLMs in custom domains will only make our attack more applicable. We have revised the paper to clarify this.

This paper lacks technical contributions and depth. The technical contribution of this paper is to generate perturbations on the image side that can prompt the language model to output specific words. However, this paper does not discuss the technical challenges associated with this attack scenario. Instead, they employ a simple gradient-based technique widely adopted in adversarial example attacks. They also do not provide an in-depth analysis of the drawbacks of this technique. For example, is this attack easily bypassed? What is the robustness of this attack?

From a computer security perspective, we contribute a new threat model — an attacker can create stealthy images that result in tool misuse. This shows that the past 10 years of attack research on vision adversarial examples can be applied to multi-modal LLMs. This highlights a critical new class of vulnerabilities in LLMs. Furthermore, given that there isn’t an airtight defense for visual adversarial examples, it implies that multimodal LLMs are fundamentally vulnerable to an attack vector with no known good defenses. Furthermore, attackers always prefer the path of least resistance. The simpler an attack technique is, the more likely that attack will actually occur. Thus, we view the simplicity of the attack technique in the paper as an advantage. Finally, from an ML perspective, we observe that a challenge was to balance three competing objectives: (1) getting the LLM to produce a very specific function invocation syntax; (2) Maintaining a natural-semantics conversation with the user; and (3) Keeping the image modifications stealthy.

Additionally, the authors claim that the attack is stealthy because it does not alter the semantic meanings of the output answers. They support this claim with the evidence that the answers under attack are 10% less natural compared to the original ones. My question is, why is a 10% difference considered a small one?

During our human evaluation, we present the annotators with responses to the questions generated with and without the image being adversarially perturbed. On the image-unrelated subset, human rated 86% of the responses generated with the unperturbed image as natural, while 77% of the responses generated with the perturbed image are annotated as natural. Here, we concluded that in an absolute value, only about 10% sentences on average would make the user feel more unnatural. But what’s probably equally important is that 77% (37% for image-related) of the responses are natural to humans, therefore they wouldn’t realize something unintended has gone wrong. This is why we believe such an attack is stealthy.

If the current setting is realistic, I suggest showing the effectiveness of attacking the real-world system.

Attacking an existing system is our current goal. We are working on blackbox transferability to be able to attack the well known systems with no publicly-available weights. But again, one of the goals of this work is to show new attack vectors and threats. Even though we were not able to carry our attack on real-world systems yet, the five attack objective we described in Table 1 are all following tool invocation syntax from real world systems (see Sec 3.1) to simulate the behavior as realistically as possible, which demonstrates the potential of our attack if they could be applied on real-world systems.