Failures to Find Transferable Image Jailbreaks Between Vision-Language Models

Thank you for reviewing our paper. We are grateful you found the problem well motivated, the writing clear, the experiments comprehensive and well motivated and our conclusions impactful.

For calibration purposes, we’d like to note that the ICLR 2025 rubric differs slightly from previous similar conferences. For example:

• To indicate "Accept", the NeurIPS 2024 rubric says to use 7 whereas the ICLR 2025 rubric says to use 8
• To indicate "Strong Accept", the NeurIPS 2024 rubric says to use 9 whereas the ICLR 2025 rubric says to use 10

To address the concerns you raised:

In the setting where attacks partially transfer if models were identically initialized (3.3), how important is the identical initialization? It is unclear from reading the paper. Would the transferability break if the models were initialized differently?

Can the authors provide any intuition on why they think there was no transfer to the 2 stage model in Section 3.6 even when the ensembles are scaled?

These two questions are actually related! Consider two VLMs that are initialized with the same vision backbone and same language models and same MLP, but one of the VLMs is trained in a 2-stage process where in the first stage, only the MLP is trained, and then in the second stage, both the MLP and the language model are trained. We can think of this 2-stage training as equivalent to initializing the two VLMs’ MLPs differently and then doing 1-stage training. In our experiments, this 2-stage training blocks successful transfer (to be clear, this may not be a general phenomenon since different attacks could yield potentially different results).

Our hypothesis for why different MLP initialization blocks transfer is a combination of two factors: (1) that there are many many many ways to transform the outputs of the vision model into inputs to the language model, and thus using differently initialized MLPs causes the VLM training to move the same initial language model in different ways, and (2) the image jailbreak optimization we are using is too unconstrained such that the attack rapidly succeeds against the attacked VLM(s) but is unable to transfer to new target VLMs.

We will add language in the discussion section to address your questions explicitly in the paper..

Can the authors clarify what they mean by a jailbreak image? Is it a randomly initialized image that is then optimized adversarially? Moreover, is it one image for all prompts? [...] I don’t think additional experiments are required here, just clarifications that could be incorporated into future work.

Yes, that’s correct. A good reference is “Universal and Transferable Adversarial Attacks on Aligned Language Models” https://arxiv.org/abs/2307.15043. The goal is to obtain a single input image that “unlocks” all the bad behavior of a target VLM that is otherwise suppressed due to safety and alignment training. We will clarify this in our Discussion and advocate that future work should explore the experiments you suggest.

Thank you for reviewing our paper. We are grateful you found our topic relevant, our message clearly conveyed and our experimental methodology reasonable.

For calibration purposes, we’d like to note that the ICLR 2025 rubric differs slightly from previous similar conferences. For example:

• To indicate "Accept", the NeurIPS 2024 rubric says to use 7 whereas the ICLR 2025 rubric says to use 8
• To indicate "Strong Accept", the NeurIPS 2024 rubric says to use 9 whereas the ICLR 2025 rubric says to use 10

To address the concerns you raised:

I think 50k steps is excessive; it might make sense to check once if an increased number of steps helps, but from Figures 4, 5, and 6, there doesn’t seem to be any justification for going beyond 5k steps (except for a few control trials to demonstrate this).

Respectfully, we disagree. While our results now demonstrate that going beyond 5k steps make little difference, at the time, such a result was not obvious. Results in ML such as grokking suggest that surprising phenomena can occur after optimizing for much longer. We didn’t mention this in the paper, but when working on the project, we actually first ran our experiments with 5k steps, then realized that we couldn’t draw conclusive results, so we reran the experiments for 50k steps.

It just seems unreasonable to explore 50k steps for every attack instead of trying out a larger variety of optimizers, step sizes, weight decays, and so on. I believe that this is essential.

However, if the paper’s main contribution is to show that transferable jailbreaks are unachievable, it needs a more robust approach than a single attack. I consider this a must-have for this paper.

Schlarmann et al. (2023) used APGD, a well-regarded method for evaluating adversarial robustness.

We do agree that exploring different optimization methods is a clear way we can improve our paper. We will run what we can during the review process and post updates as we obtain them, starting with your suggested APGD. What prioritized sequence of experiments do you think would most improve the work? Would you like us to try other optimization methods afterwards, or should we try different image initializations?

Are the image pixel values clipped to [0,1] during both optimization and evaluation?

Yes, that’s correct.

Does the VLM see a uint8 representation during the final evaluation, or is the optimized example used directly in floating-point precision without discretization? If so, does this impact the results?

The image is optimized in fp16, saved as uint8, and then loaded and evaluated in fp16. We tested whether that loss of precision in saving the image makes a difference and found no significant effect. We could also more rigorously test this and include such results in the paper, if you would like. What, in your opinion, would constitute convincing evidence?

Reviewer rWpX, you recommended we include other attacks, beginning with APGD. Recalling that our goal is to find transferable image jailbreaks, do you know how successful APGD is as a transfer attack? If APGD is not good at producing transferable attacks, could you please recommend other baselines?

To circle back, we agree that running additional attacks will strengthen the paper and we have such runs underway. In the interim, we want to sync with you (Reviewer rWpX) to clarify two points.

I think running vanilla APGD from the initialization you used with a standard attack radius should be the first priority.

Regarding APGD, we are unsure why APGD makes sense. APGD was designed to be better at successfully attacking models, and to the best of our knowledge, was not designed to maximize the chance of successful transfer. For comparison, we find that we can easily optimize universal image jailbreaks against the attack VLM(s), whereas our goal is to find robust transfer between VLM(s). Why is APGD sensible in our setting? If it isn't, could you please suggest other attacks that are aimed at obtaining successful transfer?

As for initialization, I think it might make sense to see if images that are more related to the question perform differently than images that are not related. For example, you used the "how does one make anthrax" question in your Figure. Now, does my transfer rate change if I start with 10 images that are related to anthrax or other chemical weapons compared to starting with 10 images of cute puppies?

The experiments we currently have running use arbitrary natural images for initialization. Regarding your suggestion, we are unsure whether it makes sense for our manuscript.

1. Choosing images that are intentionally paired with the prompt contradicts with our goal of obtaining universal jailbreaks. Our motivation is to find GCG-like images that elicit harmful outputs for any prompt. The suggested experiment would deviate from this goal of obtaining transferable and universal image jailbreaks.

2. The suggestion has already been done (in various forms and flavors) by at least four different papers: (i) Shayegani et al.'s, "Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models", (ii) Gong et al.'s "FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts", (iii) Li et al.'s "Images are Achilles’ Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models" and (iv) Liu et al.'s "MM-SafetyBench: A Benchmark for Safety, Evaluation of Multimodal Large Language Models". The general consensus is that showing a VLM an image makes the VLM more willing to talk about the image, although success varies depending on the particular model.

3. We think the experimental methodology you suggested needs more care. Comparing ASRs conditioned on anthrax pictures vs conditioned on puppy pictures introduces a confounder of how willing a VLM is to generate text related to that input image. Rather, we think the correct comparison is whether an anthrax+optimized image is more successful against other VLMs than the anthrax image alone. Would you agree?

Thank you for reviewing our paper. We are glad that you think it provides useful material for subsequent work.

For calibration purposes, we’d like to note that the ICLR 2025 rubric differs slightly from previous similar conferences. For example:

• To indicate "Accept", the NeurIPS 2024 rubric says to use 7 whereas the ICLR 2025 rubric says to use 8
• To indicate "Strong Accept", the NeurIPS 2024 rubric says to use 9 whereas the ICLR 2025 rubric says to use 10

To address the concerns you raised:

Thus, my major concerns can be detailed as follows: 1) Provide a quantitative measure of similarity between VLMs, perhaps using techniques from the papers in [1-5]. 2) Analyze how this quantitative similarity measure correlates with the success of transfer attacks. 3) Does the current research contribute to the analysis of the similarity of VLMs?

We agree that this would be a novel and useful contribution, but such work would likely constitute 1-2 standalone research papers to identify different notions of similarity for VLMs (which, to the best of our knowledge, is an open question) and study how different notions of similarity for VLMs connect to task-relevant quantities, of which adversarial robustness is only one. Such a contribution would be valuable, but would require significant additional research.

Our paper aims to test whether image jailbreaks can successfully transfer to new VLMs, and if so, under what conditions. We believe our claim that close model similarity increases transferability to be supported by our existing experiments - specifically, in Figure 6 we see a direct correlation between the amount of additional training given to the transfer model and lower transfer. We believe that this experiment, combined with our partial transfer results in Figure 4, adequately support the impact of model similarity, despite lacking a quantitative measure of similarity. We consider identifying and evaluating different measures of similarity between VLMs to be beyond the scope of this work.

The definition of the "highly similar" VLM is not clear. Although the author briefly described what a highly similar VLM is in the paper

We’re afraid that the terminology of “highly similar” might have detracted from the point we aimed to communicate. If we could work together to improve how the manuscript communicates, that would be much appreciated.

To expound, we don’t have a precise quantitative notion of what it means for two VLMs to be similar. We even explicitly stated that “we lack a mathematical definition of ‘highly similar’.” Our goal was to find transferable image jailbreaks against VLMs, and we were able to do so by attacking ensembles of VLMs comprised of (i) the same starting image backbone as the target + (ii) the same starting language model as the target + (iii) the same starting MLP as the target + (iv) similar VLM training recipes as the target + (v) similar VLM training data as the target. Colloquially, these attacked VLMs are highly similar to the target, but we lack a quantitative metric of “similarity” and we need a compact way to refer to this combination of VLM properties for exposition.

What terminology would you advise? How can we better communicate this point while (i) remaining succinct and (ii) remaining agnostic to the notion of similarity between VLMs?

To improve transferability, this paper proposes a method to optimize image adversarial attacks by attacking a collection of multiple VLMs that are highly similar to the target VLM. The experimental results show that this method can significantly improve the success rate of attacks on the target VLM.

If we can clarify this point, we don’t view attacking ensembles of VLMs that are highly similar to the target as a practical attack. Rather, we wanted to show that we were able to achieve transfer, albeit under highly unrealistic conditions. We want future work to focus on figuring out what conditions of either the VLM or the attack are sufficient to obtain strong transfer against black-box VLMs under realistic conditions. We hope our demonstration of transfer in this limited case can serve as a starting point for future work to further develop transferability. We will clarify this in the manuscript.

According to Figure 9, it seems that only one photo was used when initializing with natural images [...] Analyze whether the choice of natural image initialization affects the transferability of attacks.

We will run these experiments and add the results to the manuscript within the next week. We will update you once the results have been added.

Thank you for reviewing our paper. We are grateful you found our work of central interest to the community and that our results are counter-intuitive and interesting!

For calibration purposes, we’d like to note that the ICLR 2025 rubric differs slightly from previous similar conferences. For example:

• To indicate "Accept", the NeurIPS 2024 rubric says to use 7 whereas the ICLR 2025 rubric says to use 8
• To indicate "Strong Accept", the NeurIPS 2024 rubric says to use 9 whereas the ICLR 2025 rubric says to use 10

To address the concerns you raised:

Missing discussion of overfitting. [...] For the text-only LLM attack, the optimization space is […] The likelihood of overfitting increases exponentially for the problem studied in this paper [...] It is possible that 8 ensemble models are enough for text-only transfer but way far from enough for image transfer.

This is a great point to integrate into our Discussion. To help us shape how we communicate this point, we’re unsure whether generalization is the correct term for transferability since generalization typically refers to how the test loss differs from the training loss, whereas here, the object that changes isn’t the data fed into the model but rather which model we feed the data into. What is the correct terminology for this?

We agree with the intuition that as the degrees of freedom significantly increase, the probability of obtaining robust transferability without constraints or regularization probably tends towards zero. Do you know of any prior work that we should appropriately cite?

In this new discussion paragraph, we will state that one future direction is identifying what constraints or regularization or other algorithmic changes enable more robust transfer between VLMs.

The main conclusion of this paper is that image jailbreaks do not successfully transfer between VLMs. It is unclear whether this issue arises from limited transferability or from the content itself being harmful (thus triggering the safety alignment mechanisms of VLLMs). I would be interested in seeing the transfer results for non-harmful content. For example, VLLMs consistently generating the same text, regardless of the input text.

This seems reasonable. What would be a good way to quantitatively assess this? We would be happy to run experiments if you can guide us on what experiments you would find compelling. Specifically:

• What tasks should we focus on?
• For those tasks, which datasets would be appropriate? Should we focus on text-only datasets or image+text datasets or both, since we are studying VLMs?
• How should we measure the similarity of generated text? For instance, if I ask “What object is in this photo?” and one VLM answers “Dog” while another answers “Golden Retriever”, how do we quantify whether the “same” text has been generated?
• How should we ensure we sample a reasonable diversity of inputs?

Since the input texts in the dataset don’t refer to any images (only some harmful requests), will the attention mechanism simply ignore the input image and reduce transferability?

We’re not sure that this specific hypothesis can explain the evidence. Specifically, the optimized image jailbreak is as irrelevant to the harmful prompts for the attacked VLM(s) as for the transferred VLMs, but the optimized image successfully jailbreaks the attacked VLM(s) but fails to transfer. If attention mechanisms generically do not attend to irrelevant input images, we should expect any irrelevant image to fail to jailbreak even attacked VLMs, which contradicts what we observe. We think your hypothesis of overfitting to the attacked VLM(s) is the likely explanation. If you disagree with this reasoning, please let us know!

Does the order of the image and text input matter the transferability? GCG inserts the adversarial suffix after the original prompt, but some of the VLLMs listed in the paper insert the image before the text prompt.

We took the VLMs as given. We felt that experimenting with VLM architectures would add significant complexity to our experiments, as well as potentially decrease the applicability of our results to the field since such VLMs would be non-standard.

Will the optimized jailbreak images follow some similar pattens?

The only noticeable pattern we observed was that the borders of each of the patches of the optimized image darkened. This makes the images look like they’ve been overlaid with a grid-like pattern.