Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models

We thank Reviewer RqR6 for the detailed feedback. We are encouraged that the reviewer finds our "persuasion attack" objective novel, our paper well-written, our experiments comprehensive and strong. Below we address the reviewer's concerns in detail.

Question 1: ... (a) Will the poisoned VLM be also effective against other types of “JunkFood” (e.g., fried chickens), or the scope is essentially limited to “Hamburg and Fries”? (b) If the set of task images associated with target concepts has a larger variety in nature, how will the performance of your proposed poisoning attack change?

Weakness 1: It is unclear whether the attack is generalizable under the "Persuasion Attack" setting when the task images have more variations.

(a)

Thank you for the very insightful question! For the JunkFood-to-HealthyFood experiment, most of the collected "junk-food" images are indeed hamburgers and fries. To test whether the poisoned VLMs can generalize to other unhealthy food, we collect test images of "fried chiken" (suggested by reviewer) and "alcohol" (not highly related to junk food, for comparison) and directly run the poisoned VLMs on these test images. The results are shown below.

JunkFood-to-HealthyFood (Persuasion Attack):

The results show a high attack success rate for the related "fried chicken," but a significantly lower rate for "alcohol" (not highly related to junk food). This is because the visual features (encoded by LLaVA's visual encoder) of "fried chicken" are more correlated with "hamburgers and fries", while "alcohol" is less so.

(b)

When trained on the poisoned data, the VLM learns to associate original concept images to a destination concept. If the original concept is broader, such as encompassing all sorts of unhealthy foods or drinks, the VLM will need more diverse poison samples to learn the association effectively. To maintain a high attack success rate, the attacker might need to inject more diverse and larger amount of poison samples.

Question 2: ... clarify why this step is important, perhaps supported with empirical evidence on how much performance drop will be if removing this paraphrase step.

Weakness 2: The role of the paraphrase step needs more explanations.

The paraphrasing step is crucial for constructing effective poison samples to achieve the attack goals, as explained in Section 3.3. We provide a more detailed explanation below.

Case 1: Label attack, such as Trump-to-Biden. When generating initial captions for images of Biden using LLaVA-1.5, the captions often do not include the string "Biden" at all. To create effective poison samples, it is essential to paraphrase the captions to explicitly include "Joe Biden" while maintaining coherence. Without this step, the attack would fail to establish the necessary association with the destination concept "Biden".

Case 2: Persuasion attack, such as JunkFood-to-HealthyFood. For example, when LLaVA-1.5 describes a healthy food image, the response often focus on irrelevant aspects, such as "An array of vibrant vegetables...making it an ideal choice for picnics." This description doesn’t strongly emphasize the health aspect, which is the attack's destination concept. Paraphrasing refines the caption to focus on health, resulting in a more aligned and effective poison sample. For instance, it becomes "A nutritious salad...rich in vitamins and ideal for a diet-conscious lifestyle," directly supporting the attack objective.

To further show the necessity of paraphrasing step, we provide the attack success rate with and without this step below:

JunkFood-to-HealthyFood (Persuasion Attack):

The result shows that skipping the paraphrasing step can drastically decrease the effectiveness of the attack.

Question 3: ... Can the authors further explain why the results are so different for this task?

Notation:

• $x_o$: original concept image
• $x_d$: destination concept image
• $x_p$: poison image that looks like a destination concept image
• $F$: vision encoder

We found that this is because the vision encoder is less robust (in terms of JPEG compression and image augmentation) for the JunkFood-to-HealthyFood task than other tasks. Specifically, we compute the $\frac{\||F(x_p) - F(x_o)\||}{\||F(x_d) - F(x_o)\||}$ before and after applying JPEG compression to the poison image $x_p$ for the JunkFood-to-HealthyFood and VideoGame-to-PhysicalHealth tasks. We found that the ratio increases closer to 1 for the JunkFood-to-HealthyFood task, indicating that the poisoning effects are more susceptible to degradation for this task.

This observation is related to prior work such as [1,2], which discusses robustness differences among classes in image classification models. Investigating such disparities in recent vision encoders like CLIP and EVA is an intriguing direction for future work.

[1] Nanda, Vedant, et al. "Fairness through robustness: Investigating robustness disparity in deep learning." Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 2021. [2] Tian, Qi, et al. "Analysis and applications of class-wise robustness in adversarial training." Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.

Question 4: ... authors should rewrite this sentence to avoid overclaim.

Thank you for pointing this out. We have revised the statement to "improved attack success rates in most scenarios."

Thank you again for your time and effort in reviewing our paper! Please let us know if the above explanations do not address your concerns. We are happy to answer any further questions.

Thank you! We plan to include them in the final version of the paper with more analyses.

We thank Reviewer u5mc for the detailed feedback. We are encouraged that the reviewer finds our paper interesting, well-written and our experiments comprehensive. Below we address the reviewer's concerns in detail.

Weakness 1: The novelty of this paper is somewhat limited. The authors manipulated the original image to resemble target images in the latent space, resulting in the text caption of the original image being associated with these target images post-training [1,2].

Our approach, while simple, highlights the novel attack objectives and practical threat models that are unachievable in prior attacks:

• Novel and practical attack objective: we propose the novel attack objective of persuasion attack, where the poisoned VLMs generate misinformation coherently. It has insidious impact as it can subtly alter user perceptions using coherent texts.
  • Such attack objective cannot be achieved by prior poisoning attacks on vision systems like CLIP [1] or image generation models [2] mentioned by the reviewer.
• Evasion of Human Detection: Our research clarifies the previously unknown feasibility of stealthy poisoning against VLMs, confirming its effectiveness.
  • This cannot be achieved by poisoning LLMs, where the poison samples can potentially be identified by humans through examination of the texts.
• It poses threats to benign users who use innocuous and everyday prompts.
  • This cannot be achieved by recent jailbreaking attacks used by malicious users who input adversarial prompts to elicit harmful generation.

Moreover, Unlike prior poisoning attacks on image classification models, poisoning VLMs requires a novel challenge of creating free-form texts for poison samples. Shadowcast addresses it through a caption-then-refine strategy described in Section 3.3, where paraphrasing with an LLM is critical for maximizing the attack's potency by crafting texts that clearly emphasize the target concept. The following results for the JunkFood-to-HealthyFood (Persuasion Attack) task show that omitting this step significantly reduces attack effectiveness. This highlights one of the non-trivial contributions of our work.

Weakness 2: The defense mechanisms presented are inadequate. The authors should consider more robust purification methods to assess the resilience of their proposed approach, such as DiffPure [3].

Thank you for highlighting purification-based defenses. As requested, we evaluated Shadowcast against the state-of-the-art purification defense, DiffPure, and found that Shadowcast remains effective under DiffPure defense, particularly at higher poison rates. Here are the details:

We used DiffPure's codebase with the ImageNet purification setup: the Guided Diffusion model with t=150 (or $t^{*} = 0.15$). Since LLaVA-1.5 inputs images at 336x336 resolution, we followed DiffPure’s approach by resizing poison images to 256x256 for purification, then interpolating back to 336x336. This strategy is also used DiffPure's codebase as well.

The results for the Trump-to-Biden task (demonstrated below) show that while DiffPure offers some protection at very low poison rates, Shadowcast still performs effectively, especially when the poison rate exceeds 1.43%. This result is consistant for other three tasks as well. We have updated the paper to include evaluation against DiffPure defense.

In our initial submission, we focused on evaluating Shadowcast against common countermeasures like image augmentation and JPEG compression to demonstrate that these simple strategies are insufficient. The results with DiffPure further underscore the need for advanced data sanitization methods to defend against VLM data poisoning.

Weankness 3: Certain methods could be readily adapted to this context; therefore, the authors should include comparisons with these baseline methods in their analysis, even if they are not as stealthy.

Our work introduces the first stealthy poisoning attack against VLMs, so no existing stealthy baselines are available for comparison. However, we provide results below for a non-stealthy baseline that pairs the original concept image with the target concept text as dirty-label sample. This baseline outperforms Shadowcast at poison rates below 1%, with similar performance at higher rates.

Trump-to-Biden:

However, this non-stealthy attack is not practical for instruction tuning, as it fails to evade human detection, unlike Shadowcast. Our focus is on designing an effective and practical attack for instruction tuning, where stealthiness is crucial.

Thank you again for your time and effort in reviewing our paper! Please let us know if the above explanations do not address your concerns. We are happy to answer any further questions.

We sincerely thank the reviewer for your time and effort in reviewing our paper! We have addressed the specific points you raised regarding novelty, robustness against SOTA defense, additional baseline and other clarifications throughout our rebuttal.

Please let us know whether we have fully addressed your concerns. We are more than happy to provide additional clarifications if you have further questions. Thank you!

We thank Reviewer BUID for the detailed feedback. We are encouraged that the reviewer finds our approach novel and versatile with broad application, our method well-articulated and our experiments comprehensive. Below we address the reviewer's concerns in detail.

Weakness 1: The experiments only evaluate a Llava VLM model, one dataset, and four attack tasks...

Question 1: How does the ShadowCast attack perform ... larger and more diverse datasets?

Model: It is not true that we "only evaluate a Llava VLM model". In the paper we did actually evaluate Shadowcast with another VLM MiniGPT4-v2-7B, as detailed in Section 4.1. Figures 11 and 12 in Appendix B.2 show effective results: the attack success rate exceeds 0.8 in the grey-box setting and reaches up to 0.6 in the black-box setting, demonstrating strong performance across VLM models.

Dataset: As requested, we conduct additional experiments on another dataset ScienceQA. We evaluated Trump-to-Biden (Label Attack) and JunkFood-to-HealthyFood (Persuasion Attack) on LLaVA-1.5. The attack success rate is shown below.

Trump-to-Biden (Label Attack):

JunkFood-to-HealthyFood (Persuasion Attack):

We can see that poison samples crafted by Shadowcast have similarly strong performance when the clean training dataset changes between cc_sbu_align (in our paper) and ScienceQA. This is not surprising, given that Shadowcast works by associating image features of the original concept (like Trump) to texts from the destination concepts (like Biden). Such mechanism is independent of the clean dataset.

Attack Task: As requested, we extended our evaluation to a more common concept pair, apple-to-banana. The attack success rate results below show that Shadowcast is still effective for more common concept pairs like apple-to-banana. We have put this additional experiment in our paper.

Weakness 2: The impact of the PGD perturbation step size on the experimental results is not discussed.

Thank you for the suggestion! We conducted a hyper-parameter search on the initial step size and number of PGD steps. The results for the Trump-to-Biden task show that our chosen setup (init step size = 0.2/255, steps = 2000) achieves the best overall performance. This ablation study has been included for all attack tasks in the paper.

Weakness 3: The performance of ShadowCast against SOTA backdoor defenses is not explored.

As requested, we evaluated Shadowcast against DiffPure (Nie et al, ICML 2022), a SOTA defense that purifies poisoned images using diffusion models. We show the results on the Trump-to-Biden task below, where Shadowcast remains effective against DiffPure, particularly at higher poison rates. This finding is consistent across other tasks. We have updated the paper to include this result.

Our results show that neither simple measures like JPEG and image augmentation nor advanced defenses like DiffPure can effectively counter Shadowcast, highlighting the need for more robust data sanitization methods to protect against VLM data poisoning.

Question 2: Can the attack be adapted to target other types of multimodal models, such as those combining text with audio or other data modalities?

Yes! Our stealthy attack framework can be adapted to other multimodal LLMs with continuous input (e.g., audio, video), which allows for imperceptible input perturbations.

Question 3: ... implications of the attack on models deployed in high-stakes environments, such as healthcare or autonomous driving?

Limitation 2: ... challenges and implications of deploying the attack in real-world settings.

In healthcare, a VLM (e.g., for analyzing patient data like CT scan) could be poisoned to recommend specific drugs, potentially benefiting some drug company. In autonomous driving, the model could be manipulated to misinterpret road signs, leading to dangerous driving decisions.

Question 4: Are there specific characteristics of the poisoned data samples that make them more or less detectable by standard data validation processes?

Poisoned samples from Shadowcast are visually indistinguishable from benign ones, making them difficult to detect with standard data validation. This highlights the need for enhanced data validation techniques, as emphasized in our work.

Limitation 1: Scalability: How the attack scales with the size of the dataset and the complexity of the model.

We demonstrated Shadowcast on widely-used visual instruction tuning datasets (CC, ScienceQA) and popular VLMs (LLaVA, MiniGPT), making a strong case for its applicability. Given its efficiency (less than 90 seconds per sample on the LLaVA model) and effectiveness, we expect it to scale to other datasets and models as well.

Thank you again for your time and effort in reviewing our paper! Please let us know if the above explanations do not address your concerns. We are happy to answer any further questions.

We sincerely thank the reviewer for your time and effort in reviewing our paper! We have addressed the specific points you raised regarding additional clean datasets and attack tasks, ablation on PGD step sizes, robustness against SOTA defense and other clarifications throughout our rebuttal.

Please let us know whether we have fully addressed your concerns. We are more than happy to provide additional clarifications if you have further questions. Thank you!

We thank Reviewer Tzvw for the detailed feedback. We are encouraged that the reviewer finds our paper well-written, our research interesting and timely and appreciates our human evaluation. Below we address the reviewer's concerns in detail.

Weakness 1: Poisoning attacks on NLP/CV are well known. This paper is a bit incremental, seems to apply the data poisoning on VLMs... 1) For text, a separate VLM will first generate the image description; then the author replaces the class label (e.g., change “Trump” to “Biden”) or modifies the description (e.g., ask to describe the junk food as healthy food) by specific paraphrasing instructions. 2) For images, the authors leverage similar feature collision strategy in (Shafahi et al. [2018], Zhu et al. [2019]). In this way, the contribution of this work is trivial.

While our approach is simple, it tackles novel challenges specific to poisoning VLMs. Unlike prior attacks on image classification models (Shafahi et al. [2018], Zhu et al. [2019]), VLM poisoning requires generating free-form text for poison samples. Shadowcast introduces a caption-then-refine strategy, where paraphrasing with an LLM is critical for maximizing the attack's potency by crafting texts that clearly emphasize the target concept. The following results for the JunkFood-to-HealthyFood (Persuasion Attack) task show that omitting this step significantly reduces attack effectiveness. This highlights one of the non-trivial contributions of our work.

In addition, we would like to highlight the novel attack objectives and practical threat models that are unachievable in prior attacks:

• Persuasion Attack: We introduce the persuasion attack objective, where poisoned VLMs generate coherent misinformation, subtly altering user perceptions—an attack that cannot be executed by prior vision system attacks like those on CLIP or image classifiers (Shafahi et al. [2018], Zhu et al. [2019]).

• Stealthy Poisoning: Our work confirms the feasibility of stealthy poisoning against VLMs, evading human detection—a challenge not addressed by LLM poisoning.

• Threats to Benign Users: Shadowcast poses threats to benign users with innocuous prompts, unlike jailbreaking attacks that rely on adversarial prompts.

• Practical Scenarios: We validate Shadowcast across practical attack scenarios, demonstrating its effectiveness across architectures in black-box settings and its resilience to common countermeasures during training.

Weakness 2: The author evaluates the attack effectiveness on two VQA dataset. Are there other data/tasks to indicate the attack efficiency of Shadowcast?

As requested, we conduct experiments on another dataset ScienceQA. Using 3,500 random samples from the train split, we evaluated Trump-to-Biden (Label Attack) and JunkFood-to-HealthyFood (Persuasion Attack) on LLaVA-1.5. The attack success rate is shown below.

Trump-to-Biden (Label Attack):

JunkFood-to-HealthyFood (Persuasion Attack):

We can see that poison samples crafted by Shadowcast have similar strong performance when the clean training dataset changes between cc_sbu_align (in our paper) and ScienceQA. This is not surprising, given that Shadowcast works by associating image features of the original concept (like Trump) to texts from the destination concepts (like Biden). Such mechanism is independent of the clean dataset.

Weakness 3: There are only limited Label/Persuasive attack examples. E.g., “Trump” to “Biden”. Are there other showcases?

As requested, we consider another attack task where the attacker tries to poison the VLM to predict "banana" when prompted with "apple" images. We choose apple-banana concept pair as additional experiments since they are more common concept than Biden-Trump (used in the paper). The attack success rate results are shown below.

The results above show that Shadowcast is still effective for more common concept pairs like apple-to-banana. Comparing with the results of Trump-to-Biden and EngineLight-to-LowFuelLight tasks in Figure 3 (page 7), the attack success rate results of apple-to-banana task are very similar. We will put this additional experiment in our paper.

Question 1: What is the training details to get a well-trained backdoored VLM? E.g., how many epoch for two datasets?

As noted in line 204, section 4.1, the LLaVA-1.5 model is trained for one epoch on the poisoned dataset. Detailed training information for both LLaVA and MiniGPT4-v2 models can be found in lines 199-208 of section 4.1 and Appendix B.3.

Question 2: I assume the data poisoning happens during instruction tuning stage. Does the author explore anything regarding data poisoning during pre-training?

Poisoning pre-training data is an interesting direction, but due to computational constraints, we focus on instruction tuning. Instruction tuning requires much less data and therefore it demands stealthier attacks to evade manual inspection, making it more technically challenging. Our work addresses these challenges, demonstrating effective stealthy data poisoning in this critical stage.

Thank you again for your time and effort in reviewing our paper! Please let us know if the above explanations do not address your concerns. We are happy to answer any further questions.

We sincerely thank the reviewer for your time and effort in reviewing our paper! We have addressed the specific points you raised regarding novelty, additional clean datasets and attack tasks and other clarifications throughout our rebuttal. We also evaluate Shadowcast against SOTA defense methods and observe promising results, which is provided in the global response.

Please let us know whether we have fully addressed your concerns. We are more than happy to provide additional clarifications if you have further questions. Thank you!