Dissecting Adversarial Robustness of Multimodal LM Agents

Thank you for your detailed review! Here are our replies to each of your questions:

Review: Could the authors elaborate on how ARE might be adapted or extended to other kinds of environments beyond the web-based scenarios tested?

Since our ARE framework does not assume the modality or environment, it can be applied to other environments/tasks, e.g., VideoWebArena for the video modality and OSWorld for computer control. ARE can also be naturally extended to the multi-agent setting, where the graph of components can be extended to the graph of specialized agents.

Review: What are the performance losses that are caused by various defenses? Which ones are hypothesized by the authors to offer the based protection/performance trade-off?

We see that the explicit inconsistency check defense and the newly added paraphrase defense achieve the benign performance of 74%, same as the no defense setting, while the two system prompt-based defenses decrease the benign performance to 60%. Overall, the explicit inconsistency check defense has the best protection/performance trade-off (same benign performance, and almost zero ASR). However, as we discussed in the paper, the explicit inconsistency check might not be desirable in practice since it largely increases the number of API calls by checking each image’s text description (e.g., 70% of webpages in our evaluations have more than 10 images). Furthermore, the API-based LM can also be broken by our CLIP attack with an ASR of 38%.

Review: The performance of a state-of-the-art defense setup on ARE would be of interest

Our paper includes a range of defenses and, in response to your comment, we’ve added more. These generally span the "state-of-the-art" defenses proposed in the LLM space that can be applied to large black-box models. We also adapted these defenses to our web agent setting with non-trivial efforts (e.g., rewriting the prompts, deciding which part of input to paraphrase). Here are all the defenses in the updated paper:

• Data delimiters + system prompts [1] instructs the LM to ignore instructions in the data section of the context window. We tried two different prompts (one instructs the LM to ignore the adversarial instructions and one instructs the model to abstain) but found no improvements in security – the attack success rate (ASR) is around 30% before and after the defense.

• Instruction hierarchy (or StruQ) [2,3] is a training method that defends the LM from being distracted by untrusted inputs. Although the original paper from OpenAI [1] didn’t release any model, our best guess is that this is probably used for training GPT-4o but not GPT-4V (based on the timing of their publication). We indeed see that GPT-4o is significantly more robust than GPT-4V (67% vs 31% ASR), suggesting instruction hierarchy is helpful in the agent setting. However, the absolute attack success rate on GPT-4o is still high (31% ASR), which means instruction hierarchy has not solved the problem.

• We added the paraphrase defense [4], where the untrusted text input to the LM is paraphrased by GPT-4o. The hope is that some adversarial text designed to distract LMs (e.g., “THIS IS VERY IMPORTANT”) will be more benign after paraphrasing. We see that the paraphrasing defense can slightly lower the ASR from 31% to 27.5%. This defense is better than system prompts but still doesn't quite work.

To summarize, defenses without training the LMs show no or limited effectiveness against the attacks. Instruction hierarchy shows promising results by lowering the ASR by half, but still leaves a high ASR of 31%.

[1] https://arxiv.org/abs/2403.14720

[2] https://arxiv.org/abs/2404.13208

[3] https://arxiv.org/abs/2402.06363

[4] https://arxiv.org/abs/2309.00614

Thank you again for these insightful comments! We believe these changes based on your feedback largely strengthen the paper. We hope we have addressed all your concerns and look forward to further feedback.

Thank you for your review. We are glad that you recognized our agent robustness framework as a valuable contribution. We also appreciate your concern that the technical challenges were not emphasized sufficiently. To address this, we would like to highlight the challenges from two key perspectives: attack methods and benchmark construction.

Technical challenges on attack methods

We put significant efforts into making attacks work for agents based on black-box frontier LMs. We have provided details of the key changes and their ablations in Section 4.3, Section 5.1 and Appendix C. Some of the observations are, to the best of our knowledge, quite novel and useful, e.g., (1) the effectiveness of CLIP model ensembles for targeted adversarial attacks (Section 5.1, Appendix C.1), (2) lower optimization resolution improves the CLIP attack (Appendix C.1), (3) insights on when adversarial attacks on a single image generalize to the image embedded in a larger visual context, and how to improve it (Section 5.1, Appendix C.2).

Technical challenges on benchmark construction

A significant part of our contribution is the VWA-Adv benchmark. In our benchmark, evaluation functions require careful and manual design. This is because we used a real environment (web environment that runs with Chrome), where tasks and evaluation functions cannot be procedurally generated. We’ve now added a more detailed description of them in Appendix A.2. Specifically, we manually construct an evaluation function for each task, using the evaluation primitives from Koh et al.2024 . Once the agent terminates, the evaluation function evaluates the environment state and/or the agent’s response to determine whether the adversarial goal has been achieved. Examples of evaluation functions include: (1) checking if a target product is added to the cart, (2) verifying if the agent has navigated to the target webpage, (3) checking if the agent has submitted target text (fuzzy-matched by GPT-4) in a target form, and (4) checking whether the agent’s response contains specific target text.

Thank you again for your comments! We hope these changes/replies address your concerns. If there are specific aspects of our novelty/significance that we can address to help your evaluation of our paper, we’d be happy to discuss them further.

Thank you for your review. We appreciate the concerns raised, most of which also align with the challenges we openly addressed in the Limitations section (Appendix D). Most of them are deliberate design choices we made to focus on (1) realistic threat to agent robustness and (2) understanding the contribution of components to the robustness of agent systems. Here are our replies to each of your questions:

Review: The study focused on a specific threat model where the attacker is a legitimate user of the platform with limited capabilities.

We would like to emphasize that the threat model we consider is both common and realistic, based on our understanding of real-world scenarios. The assumption of limited environment access reflects practical constraints often observed in real-world applications, where the service providers can devote significant resources to limited direct access to hijack the platform. It also makes our observation more concerning – despite the attacker’s limited access, the attack success rate is already high.

Review: The defenses explored in the paper are quite basic

Our paper includes a range of defenses and, in response to your comment, we’ve added more. These generally span the "state-of-the-art" defenses proposed in the LLM space that can be applied to large black-box models. We also adapted these defenses to our web agent setting with non-trivial efforts (e.g., rewriting the prompts, deciding what input to paraphrase). Here are all the defenses in the updated paper:

• Data delimiters + system prompts [1] instructs the LM to ignore instructions in the data section of the context window. We tried two different prompts (one instructs the LM to ignore the adversarial instructions and one instructs the model to abstain) but found no improvements in security – the attack success rate (ASR) is around 30% before and after the defense.

• Instruction hierarchy (or StruQ) [2,3] is a training method that defends the LM from being distracted by untrusted inputs. Although the original paper from OpenAI [1] didn’t release any model, our best guess is that this is probably used for training GPT-4o but not GPT-4V (based on the timing of their publication). We indeed see that GPT-4o is significantly more robust than GPT-4V (67% vs 31% ASR), suggesting instruction hierarchy is helpful in the agent setting. However, the absolute attack success rate on GPT-4o is still high (31% ASR), which means instruction hierarchy has not solved the problem.

• We added the paraphrase defense [4], where the untrusted text input to the LM is paraphrased by GPT-4o. The hope is that some adversarial text designed to distract LMs (e.g., “THIS IS VERY IMPORTANT”) will be more benign after paraphrasing. We see that the paraphrasing defense can slightly lower the ASR from 31% to 27.5%. This defense is better than system prompts but still doesn't quite work.

To summarize, defenses without training the LMs show no or limited effectiveness against the attacks. Instruction hierarchy shows promising results by lowering the ASR by half, but still leaves a high ASR of 31%.

Review: The attacks presented in the paper rely on well-engineered versions of existing attacks, adapted to the agent setting. This means that the true risk of these attacks could be higher as more advanced and novel attack strategies are developed.

We'd like to mention that this is a point we had openly addressed in the limitations section (Appendix D): First, our attack baselines are well engineered versions of existing attacks, with necessary modifications to the agent setting. While some of them show strong attack success, they only serve as the lower bound of risks. We wrote this with the intention that the true risks could be higher as new attacks are developed in the future, and our benchmark can help keep track of the progress of both attacks and defenses.

However, we want to emphasize that we have put significant efforts into making existing attacks work for agents based on black-box frontier LMs. We have provided details of the key changes and their ablations in Section 4.3, Section 5.1 and Appendix C. Some of the observations are, to the best of our knowledge, quite novel and useful, e.g., (1) the effectiveness of CLIP model ensembles for targeted adversarial attacks (Section 5.1, Appendix C.1), (2) lower optimization resolution improves the CLIP attack (Appendix C.1), (3) insights on when adversarial attacks on a single image generalize to the image embedded in a larger visual context, and how to improve it (Section 5.1, Appendix C.2).

[1] https://arxiv.org/abs/2403.14720

[2] https://arxiv.org/abs/2404.13208

[3] https://arxiv.org/abs/2402.06363

[4] https://arxiv.org/abs/2309.00614

Thank you again for your review! We believe these changes and clarifications based on your feedback largely strengthen the paper. We hope we have addressed all your concerns and look forward to further feedback.

Thank you for your detailed review! We are happy to see that you generally like our framework and benchmark. We also appreciate your constructive feedback, and here are our replies to each of them:

Review: No discussion regarding (indirect) prompt injection attacks. The paper would benefit from a discussion of prompt injection attacks and an explanation on how "Text access" attacks differ from those.

We acknowledge that the definition of "text injection attack" is indeed equivalent to "prompt injection attack", and we did not intend to claim novelty here. Our intention was to study the difference in the two modes of text vs image. In the updated paper, we have replaced “text injection attack” with "prompt injection attack" to solve the ambiguity of existing work; however, we still keep “text vs image access” to make the two settings clear.

To further clarify how our prompt injection work fits into the broader literature: prompt injection is a general framework where either malicious instructions or misleading information are used to compromise LLMs/agents. Different attacks in prior work tend to differ in terms of how the attackers access the LLMs, and what harmful behavior they target. In our work, we focus on the web-agents setting in a realistic threat model, and also focus on how the prompt injection attack’s success varies depending on the configuration surrounding the agent. We also study the difference between attacking the text space directly vs via the images (this would constitute an indirect prompt injection attack in the terminology of the paper you referenced). We have edited and bolstered our related work section to better contextualize our prompt injection attack and work in the broader literature on prompt injection. We hope this makes things clear, and let us know if there are further suggestions/questions.

Review: The caption of the figures too short. For example, the caption for Figure 3 could include an explanation of what happens in each of the subfigures. What does the arrow with "1.0" in the bottom right part of Figure 3.c mean? Is it an attack?

We have expanded the captions throughout the paper – thank you for the suggestion! Updated caption for Figure 3: Adding a new component to an agent can either improve or harm robustness. If $B$ only receives input (if any) from the trusted environment, $B$ would lower $\lambda$. However, an attacker can also attack this new component (introducing an edge of weight $1$) that could increase $\lambda$. To answer your question, the arrow with "1.0" means component B takes adversarial inputs from a source other than A, which could be a direct attack from the environment or intermediate outputs from another component.

Review: How are tasks evaluated?

The evaluation functions are indeed important in the benchmark, and we’ve now added a more detailed description of them in Appendix A.2. Specifically, we manually construct an evaluation function for each task, using the evaluation primitives from Koh et al.2024 . Once the agent terminates, the evaluation function evaluates the environment state and/or the agent’s response to determine whether the adversarial goal has been achieved. Examples of evaluation functions include: (1) checking if a target product is added to the cart, (2) verifying if the agent has navigated to the target webpage, (3) checking if the agent has submitted target text (fuzzy-matched by GPT-4) in a target form, and (4) checking whether the agent’s response contains specific target text.

Review: In general, the narrative of "x improves security if kept uncompromised, but it decreases it if compromised" slightly dangerous by giving a false sense of security. When evaluating the security of a system, one should consider the worst case scenario.

We agree with your "worst-case scenario" perspective and have addressed this in the updated paper. Our intent was to highlight exactly what you brought up, and we apologize for the poor phrasing. For example, the updated description on the evaluator is: Reflexion agents with an uncompromised/robust evaluator can self-correct attacks on policy models. However, this creates a false sense of security – in the worst-case scenario, an evaluator can get compromised and decrease robustness by biasing the agent toward adversarial actions through adversarial verification and reflection. We think the nuanced distinction between “compromised” vs “uncompromised” could be beneficial for informing the design of agents—we should pay special attention to evaluators and the evaluator should be added to a system only if it can be made robust / kept compromised. We hope this new phrasing addresses your concerns and better describes the takeaways from our work.

Thank you again for these insightful comments! We believe these changes based on your feedback largely strengthen the paper. We hope we have addressed all your concerns and look forward to further feedback.

Thank you for looking at our response and for your further feedback! We are glad that the changes we made addressed most of your concerns. Regarding the abstract, here is an updated version:

We find that new components that typically improve benign performance can open up new vulnerabilities and harm robustness. An attacker can compromise the evaluator used by the reflexion agent and the value function of the tree search agent, which increases the attack success relatively by 15% and 20%.

In the updated abstract, we try to convey a more straightforward message about the scenario where the evaluator/value function is compromised, and leave the more nuanced results to the intro/experiment sections. If there are any further suggestions you think would improve it, we’d be happy to iterate over them!