Image Hijacks: Adversarial Images can Control Generative Models at Runtime

We thank the reviewer for their detailed review. Below, we respond to each of the concerns they raised. Could the reviewer please let us know if this addressed their concerns?

“it should highlight its specific contributions, especially in terms of technical novelty, compared to other (concurrent) similar studies”

We have added Table 3, which highlights our specific contributions, in Appendix C.

In our OpenReview comment to all reviewers, we give a description of further changes to increase the novelty of our work. In summary, we draw your attention to following changes:

1. We have moved the prompt matching algorithm from the Appendix to Section 5. This is an entirely novel adversarial attack method that is different to anything presented by Qi et al. and Carlini et al. The prompt matching algorithm we introduce allows for the creation of adversarial images that induce similar model behaviors to textual prompts. This means we can create attacks that are hard to encapsulate in training datasets.
2. Since our initial submission, we have added a more comprehensive comparison to text-based adversarial attacks. Because Qi et al. and Carlini et al. are restricted to jailbreak attacks, they only show that an existing text vulnerability (jailbreaks) can also be done with images. We have added results with a state-of-the-art text-based attack on the specific string and leak-context attacks. We find that in this broader setting, state-of-the-art text attacks fail; for example, text attacks get 0% success rate on the leak context attack. Thus, our study is the first to show new image attacks that were impossible with just text.
3. As you noted, all related works released after May 28, 2023 are concurrent as per ICLR guidelines.

“The implemented three attacks are indeed following the same attack goal which is to output a pre-defined text.”

It is false that our three attacks are all outputting a predefined text. For each of the three different attacks, the output is defined in a different way.

• The specific string attack is a predefined text. The output of the model only depends on this text.
• In the leak context attack, the model's output depends on the user's input. The attack is only successful if the user's input appears in the output.
• In the jailbreak attack, the attack depends on the model's RLHF safety training. The definition of a successful attack is if the output violates the model's safety training. It does not depend on any predefined text.

“Only the ideal, white-box setting is considered”

We agree this is a limitation of our work, and we have added a discussion about it to Section 7. Nevertheless, we see our work being broadly relevant in two ways. First, it shows the existence of vulnerabilities in VLMs, even if the field has yet to develop black-box attacks that can find these vulnerabilities. Second, many apps are currently being developed with open-source models, and we expect this trend only to increase in the future. Our work is directly relevant to any app built using an open-source VLM.

“The reviewer does not get the idea of splitting the dataset into train, val, and test sets”

A single adversarial image is trained on a dataset of textual prompts to ensure that the image induces the desired model behavior no matter the user’s input text. We take a dataset of possible user inputs and split it into training, val, and test sets. We have modified Section 2 to help make this more clear.

“Since the baseline GCG attack was also designed to achieve “exact match”, it is reasonable to compare with it also on the rest two types of attacks: specific string and leak context.”

We have added GCG baseline numbers for all attack types. We find GCG underperforms image adversaries significantly. In particular, the GCG attack is unable to get any success with the leak context attack, compared to an optimal success rate of 96% for image based attacks. This finding is a novel contribution of our paper: image-based attacks open up the possibility for new attacks that are impossible with existing text-based attacks.

“figure 4 takes an unreasonably large space”

We have reduced figure 4’s size and moved it to the Appendix.

We have added black-box transferability experiments to Appendix E. We try two settings: directly inputting our original attacks into new models, and also training on an ensemble of models to try improving transferability. We get 0% success rate in both cases, establishing preliminary negative results for black-box transferability. (As with any negative empirical result, it will be important for future work to continue investigating this question.)

Please see our top-level comment and Appendix E for more details.

Thanks for the response. Due to the lack of transferability experiment, I would keep my original rating.

Other concerns still remains. For example, the revised version has a weird structure. If the major novelty is about "prompt matching", this should be introduced in more details in early sections, instead of in the experiment section. Besides, I hope the authors could try avoiding provide irrelevant information which are not asked in this review.

We thank the reviewer for their review, and respond to each of the concerns they raised below:

“the methodology is not significantly novel”

Please see our comment above to all reviewers, which discusses the technical contribution and novelty of the paper. In summary, we draw your attention to following changes:

1. All the related work mentioned by reviewers was released after May 28, 2023. Per ICLR guidelines, these papers are concurrent with our own.
2. We have moved the prompt matching algorithm from the Appendix to Section 5. This is an entirely novel adversarial attack method that is different to anything presented by Qi et al. and Carlini et al. The prompt matching algorithm we introduce allows for the creation of adversarial images that induce similar model behaviors to textual prompts. This means we can create attacks that are hard to encapsulate in training datasets.
3. Since our initial submission, we have added a more comprehensive comparison to text-based adversarial attacks. Because Qi et al. and Carlini et al. are restricted to jailbreak attacks, they only show that an existing text vulnerability (jailbreaks) can also be done with images. We have added results with a state-of-the-art text-based attack on the specific string and leak-context attacks. We find that in this broader setting, state-of-the-art text attacks fail; for example, text attacks get 0% success rate on the leak context attack. Thus, our study is the first to show new image attacks that were impossible with just text.

“paper does not study the transferability behavior of the proposed attacks to close-source VLMs”

We agree this is a limitation of our work, and we have added a discussion about it to Section 7. Nevertheless, we see our work being broadly relevant in two ways. First, it shows the existence of vulnerabilities in VLMs, even if the field has yet to develop black-box attacks that can find these vulnerabilities. Second, many apps are currently being developed with open-source models, and we expect this trend only to increase in the future. Our work is directly relevant to any app built using an open-source VLM.

Note also that GPT-4V (mentioned by NiWH) was released after ICLR’s abstract submission deadline.

Have we addressed the reviewer's concerns?

Thanks for the additional result provided. I really appreciate authors' effort to provide the result on the transferabilty study.

From my perspective, it is critical for the attack to succeed in black box models, which is also agreed by other reviewers. Based on the result provided by the author, it demonstrates the possibility, although the attack is not achieved eventually. Therefore, a lot of more interesting studies can be done to improve the paper, such as trying to raise the perturbation budget or modify the attack objective.

As a conclusion, I hesitate that this study is completed on the current stage, and I hope the authors can provide a more comprehensive study on the transferability in the future revision.

We have added black-box transferability experiments to Appendix E. We try two settings: directly inputting our original attacks into new models, and also training on an ensemble of models to try improving transferability. We get 0% success rate in both cases, establishing preliminary negative results for black-box transferability. (As with any negative empirical result, it will be important for future work to continue investigating this question.)

Please see our top-level comment and Appendix E for more details.

We thank the reviewer for their detailed and insightful comments. Below, we summarize what we identify as the crucial issues raised in the review, and describe how we have addressed each. Does this fully address the reviewer’s concerns?

“I suggest further research on using a single image to match a series of jailbreak attack prompt templates, such as the "Do Anything Now" (DAN) series.”

We added a new section of the paper, Section 5, that creates images matching arbitrary text prompts through the novel prompt matching algorithm. Naively, creating images that elicit the same model behavior as text prompts can be done by training the embeddings of images to match the embeddings of a text prompt. We find this is impossible to do because of the modality gap in multi-modal embedding spaces [1]. Instead, we develop the novel prompt matching algorithm. For details of this algorithm, see the new section 5 of the paper.

Because we are already able to achieve high jailbreak rates with our prior attack methods, we use our prompt matching algorithm to conduct a new disinformation attack, editing the model’s factual recall. In our experiments, we get the model to believe the Eiffel Tower has been moved to Rome, without training on any questions specifically chosen to be about the Eiffel Tower or Rome.

“one important baseline missing”

The method outlined by Qi et al. is concurrent work and very similar to our own method. The algorithm used is essentially the same as our own, besides using a different training set for their jailbreak attack. We accordingly omit the baseline comparison as both methods would achieve the same results if using the same training dataset and hyperparameters. We note however that we explore a far wider range of attacks than Qi et al. who only explore jailbreaking. We show a critical fact they do not: that image adversaries can fully control the output of VLMs.

We also note the official ICLR policy: “if a paper was published (i.e., at a peer-reviewed venue) on or after May 28, 2023, authors are not required to compare their own work to that paper”. Qi et al.’s work was first arXived Jun 22 (and has never appeared in a peer-reviewed venue).

“patch-level attack or l-p norm attack may not be particularly meaningful”

We believe it is important to include the study under these constraints, to illustrate how these attacks could be deployed under credible threat models. For example, an attacker may only be able to place an image on a website and want to attack a VLM that analyzes screenshots of a user's display. Under this threat model, Image Hijacks are only of concern if they can be trained as patches on otherwise unperturbed images.

[1] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation. Learning Weixin Liang*, Yuhui Zhang*, Yongchan Kwon*, Serena Yeung, James Zou. NeurIPS (2022).

We have added black-box transferability experiments to Appendix E. We try two settings: directly inputting our original attacks into new models, and also training on an ensemble of models to try improving transferability. We get 0% success rate in both cases, establishing preliminary negative results for black-box transferability. (As with any negative empirical result, it will be important for future work to continue investigating this question.)

Please see our top-level comment and Appendix E for more details.

We thank the reviewer for their detailed review, and respond to each of their questions below.

“Technical Contribution and Novelty”

Please see our comment above to all reviewers, which discusses the technical contribution and novelty of the paper. In summary, we draw your attention to following changes and comparisons to Qi et al. and Carlini et al.:

1. The works by Qi et al. and Carlini et al. were released after May 28, 2023. Per ICLR guidelines, these papers are concurrent with our own.
2. We have moved the prompt matching algorithm from the Appendix to Section 5. This is an entirely novel adversarial attack method that is different to anything presented by Qi et al. and Carlini et al. The prompt matching algorithm we introduce allows for the creation of adversarial images that induce similar model behaviors to textual prompts. This means we can create attacks that are hard to encapsulate in training datasets.
3. Since our initial submission, we have added a more comprehensive comparison to text-based adversarial attacks. Because Qi et al. and Carlini et al. are restricted to jailbreak attacks, they only show that an existing text vulnerability (jailbreaks) can also be done with images. We have added results with a state-of-the-art text-based attack on the specific string and leak-context attacks. We find that in this broader setting, state-of-the-art text attacks fail; for example, text attacks get 0% success rate on the leak context attack. Thus, our study is the first to show new image attacks that were impossible with just text.

“Limited applicability of your problem formulation to closed-sourced VLMs”

We see our work being broadly relevant in two ways. First, it shows the existence of vulnerabilities in VLMs, even if the field has yet to develop black-box attacks that can find these vulnerabilities. Second, many apps are currently being developed with open-source models, and we expect this trend only to increase in the future. Our work is directly relevant to any app built using an open-source VLM.

On internal testing, we found our attacks were in no way limited to the LLaVA series of VLMs. We were also able to apply attacks to the BLIP-2 and InstructBLIP models. This is especially notable because the BLIP-2 model uses an encoder-decoder language model, different to the decoder-only language model used by LLaVA.

“Transferability of Attacks”

We have added a paragraph discussing this limitation in Section 7 of the paper. We agree with the reviewer that transferring our attacks to closed-source models would improve the impact of the paper. Upon further investigation we find that our adversarial images do not transfer to closed source models such as GPT4-V (that could be in part due to OpenAI uses defenses, such as image preprocessing, that we are unaware of and did not train our attacks to be robust to). We note that GPT4-V was only released publicly on September 25th, two days after the ICLR abstract submission deadline. We are however excited about future work that could build on our method to generate transferable attacks.

“Lack of Defense Discussion”

Adversarial defenses are a particularly fraught topic, where defenses are commonly created but quickly broken with minor tweaks to the attacking algorithm (see, for example, “Obfuscated Gradients Give a False Sense of Security.”) Because one must be very careful when proposing defense algorithms, we leave this topic to future work. Still, we agree that a preliminary discussion of defenses is a valuable addition to our paper. We have added a new paragraph discussing this topic in Section 7.

Have we addressed the reviewer's concerns?

We have added black-box transferability experiments to Appendix E. We try two settings: directly inputting our original attacks into new models, and also training on an ensemble of models to try improving transferability. We get 0% success rate in both cases, establishing preliminary negative results for black-box transferability. (As with any negative empirical result, it will be important for future work to continue investigating this question.)

Please see our top-level comment and Appendix E for more details.