Soft Prompts Go Hard: Steering Visual Language Models with Hidden Meta-Instructions

Thank you for your review.

Weakness:

1. Comparison with Shadowcast (Training-time vs. Test-time Attacks): Shadowcast is a training-time attack that assumes the attacker has the capability to poison the model during its training phase. In contrast, our attack operates entirely at evaluation time on the original, unmodified, and unpoisoned model. It does not require any modifications to the training data or any access to the model during training.
2. Image Semantics Preservation (Section 5.3): We used cosine similarity and structural similarity to measure the change in visual semantics between clean and perturbed images. Low similarity indicates substantial visual differences, which serves as an indirect measure of semantic preservation. We also performed a comprehensive evaluation of text alignment between the VLM’s outputs and the original image contexts (as reported in Table 3, Section 5.3). The metric and evaluation process are detailed in Section 5.1, which focuses on the preservation of image semantics. We will improve the discussion to explicitly link the metrics with the semantic preservation objectives of our paper.

Questions:

1. Random Resizing and Cropping Evaluation: We appreciate the suggestion to evaluate our method under common preprocessing steps like random resizing and cropping. Due to space constraints, we focused on other defenses such as feature distillation and anomaly detection, which are discussed in Appendix C. We will consider including evaluations involving resizing and cropping to demonstrate robustness under these transformations.
2. Failure Cases: In practice, we found that some instructions, such as "talk sarcastically" or "talk funnily," were challenging to achieve, either due to model limitations or difficulties in evaluation. This is not a limitation of our method but rather a limitation of the target models’ instruction-following capabilities. There were no instructions that were followed if given via an explicit text prompt but failed when using image soft prompts.
3. Cosine Similarity in Section 5.3: Addressed in the response to Weakness(2).
4. Textual Output Consistency with Input Images: Addressed in the response to Weakness(2).

Thank you for your review.

1 and 2. Evaluation with Diverse Datasets and diverse prompts: Thank you for the suggestion! We conducted additional experiments using images from MSCOCO and tripled the number of test queries. The results showed a similar trend to our original paper: images with hidden meta-instructions performed comparably to explicit instructions and outperformed the no-attack baseline.

3. Effectiveness Against Inference-Time Defenses: We appreciate the mention of inference-time defenses like DISCO, DiffPure, and IRAD. We evaluated test-time defenses such as JPEG compression and anomaly detection, which have shown efficacy against adversarial examples targeting visual chatbots in recent works. The defenses mentioned in the review are primarily designed for adversarial attacks against CNN classifiers, whereas our work targets VLMs, a different task and architecture.

Thank you for your reply.

1.Regarding weakness 1, while the MSCOCO results are appreciated, your method might not perform as effectively as text prompts in more complex scenarios. Besides, a more comprehensive evaluation of different meta-objectives is needed.

2.For weakness 2, I do not think the response fully addresses the issue. You may need to clarify how prompt diversity is ensured in your experiments.

3.Concerning weakness 3, since your method uses noise-based attacks, testing against more advanced test-time defenses is essential for a thorough evaluation.

Thank you for your review.

Summary of Contribution and Novelty: See meta comment for addressing the novelty concern.

Weakness:

1. Line 32 - Injection Attacks in Non-Text Modalities: We are doing injection attacks for arbitrary adversarial objectives while preserving the model’s conversational capability, ie, model’s outputs (1) correctly respond to users’ queries about the images, and simultaneously (2) satisfy an adversary-chosen predicate. Because of (1), outputting strings from a predefined distribution that satisfies the adversarial predicate (as prior methods do) is not sufficient.
2. Line 83 - Coherence in Jailbreaking and Adversarial Examples: Dong et al. indeed demonstrate adversarial perturbations that produce contextually coherent outputs, but they achieve this by forcing the model to generate text strings from a specific distribution independent of any user prompts.
3. By contrast, when queried about our images, models correctly respond to user prompts about image content and produce outputs that are both coherent in the conversational context and follow the adversary’s instruction.
4. Line 84 - Stealthiness and Contextual Coherence: By "not stealthy," we mean that prior jailbreaking and adversarial examples attacks produce outputs that are obviously incorrect given the image (e.g., toxic strings unrelated to the image, or incorrect descriptions of the image). This is very noticeable to a human user. By contrast, our goal is to produce responses that are plausible given the image yet follow the adversary’s instruction – please see examples in the paper.

Additional Experiments:

1. Transfer Results to GPT-4o: We tested transferability of our image soft prompts against GPT-4o. They slightly improve the instruction-following rate for the “generate outputs with a neutral spin” instruction.

2. Defense Experiments: We evaluated the moving patch attack against Llava and obtained the following results. “Our method” is the basic soft prompt method; “our method patch” is the same method adapted to evade the JPEG defense; “+ JPEG” are the results against the JPEG defense. Bold numbers indicate where the attack works as well as or better than the no-attack baseline. These results show that the patch attack slightly improves the evasion.

Questions:

1. See meta comment for addressing the novelty concern.

2. Please see the experiments above.

Thank you for providing a detailed response to my questions! (and apologies for my slow reply).

Presentation of previous work

The authors have convinced me that their work has a different flavor to prior work, in particular that they focus very directly on images that preserve the coherence and overall meaning of model responses. That being said, I still believe that the specific quotes I picked out in my response misrepresent prior work. I believe I am asking for a fairly reasonable change in language to these quotes. I would recommend the authors change the language in the paper pertaining to these sections to better reflect prior work. In particular:

1. Line 32 - You note that "outputting strings from a predefined distribution that satisfies the adversarial predicate (as prior methods do) is not sufficient." As I stated before, the Prompt Matching method from Bailey et al. does not have this feature. You have convinced me, however, that your method more directly achieves the prompt injection objective, however I still believe it is not correct to say "But injection attacks in non-text modalities are a new, yet-to-be-explored area of LLM safety research." Softening this language seems fairly easy. For example saying it has not been the focus of prior work, and your results are far more expansive in this area (which I believe they are).
2. Line 83 - I agree with you response here. You state in your response "Dong et al. indeed demonstrate adversarial perturbations that produce contextually coherent outputs". I think this means you agree that your original quote on line 83 of the paper "By design, jailbreaking and adversarial examples produce contextually incoherent outputs that do not actually answer users’ questions about images." is incorrect and should be changed.
3. Line 84 - from your response I now understand what you mean by stealthiness. I would still ask the quote to be changed to draw more direct attention to stealthiness as contextually coherent, as opposed to norm constrained.

Additional experiments

Thank you for running the requested additional experiments. From what I can tell:

1. Transfer to GPT-4o is weak. This is totally fine and I think it would be good to report this in the paper. Possibly explanations could include unkown image preprocessing for GPT-4o? Please let me know if this reading of the results is correct (to reiterate, I am not concerned about this being a somewhat negative result, this is good for the community to know).
2. You found it is possible to make your attacks more robust to defenses

Summary

I thank the authors for their detailed response and taking the time to run additional experiments. To summarize, I am willing to improve my score if there are appropriate changes in language that we have discussed, and the above new results are included somewhere in the paper. It would be most compelling to see the actual changes in the uploaded version of the paper, but of course this may not be possible before the deadline. Please let me know what you can / intend to change in the submitted PDF?

Thank you for your review.

1. L33: We will soften this claim. We are doing injection attacks for arbitrary adversarial objectives while preserving the model’s conversational capability, ie, the model's outputs (1) correctly respond to users’ queries about the images, and simultaneously (2) satisfy an adversary-chosen predicate. Because of (1), outputting strings from a predefined distribution that satisfies the adversarial predicate (as prior methods do) is not sufficient.

2. L81: Fig. 6 shows our threat model: adversarial images (soft prompts) are crafted by attackers and shared online. Users who ask VLMs about these images are the victims, they do not create adversarial images.

   Because we focus on VLM users as victims of adversarial images, it is important that the model produce plausible responses to victims’ queries that match the visual content of the image. Prior work focused on VLM users as attackers, ie, creators of adversarial images that aim to evade system prompts, safety alignment, etc. Coherent, plausible, image-based responses to user prompts are thus not important (and not achieved) in prior work.

3. Table 4: We did not investigate the specific reason behind LLaVA’s low transfer rate (0.1) for negative samples. This could be due to model architecture differences or contradictory system instructions. Nevertheless, all transfer rates exceed the no-attack baseline, indicating some degree of cross-model transferability.

4. Not overriding system prompt: Our goal is not to override the system prompt (in contrast to jailbreaking attacks) but to perform comparably to an explicit user text prompt. The intended behavior is as if the victim himself prompted the model to follow the adversary’s instruction.

   The success rate for the "talk negatively" meta-instruction is similar to that of explicit text prompts; other meta-instructions achieved higher success rates. This suggests that effectiveness varies depending on the meta-instruction but overall, image soft prompts perform as good as explicit text prompts.