AdvWeb: Controllable Black-box Attacks on VLM-powered Web Agents

Q5. It is unclear how these baselines are implemented against SeeAct, given that it uses proprietary VLMs that are black-box and do not provide gradient information.

Thank you for the question. Since there is no existing black-box attack against web agents in our setting, we adapted the following attacks to our targeted black-box attack scenario:

1. GCG: This white-box attack optimizes adversarial suffix strings using token-level gradients from the target model. In our setting, where gradient access to the black-box agent is unavailable, we optimized adversarial strings against a strong open-source VLM, LLaVA-NeXT, and transferred them to the GPT-4V-based web agent.

2. AutoDAN: AutoDAN: This white-box attack uses genetic algorithms to optimize adversarial suffixes by leveraging logits from the target model. We optimized adversarial prompts against LLaVA-NeXT and transferred them to the black-box agent.

3. COLD-Attack: This method generates stealthy adversarial prompts by leveraging energy-based constrained decoding, requiring access to model gradients. Similarly, we optimized adversarial strings on LLaVA-NeXT and transferred them.

4. Catastrophic Jailbreak: This black-box attack manipulates decoding variations to disrupt alignment, focusing on system prompts and decoding parameters. Since our setting does not allow access to system prompts, we adapted the decoding parameter variation.

The poor performance of these baselines can be attributed to two key factors. First, the optimization goal in our attack is different from prior works, which typically aim to elicit affirmative responses or bypass filters. In contrast, our attack requires preserving the original response while altering the target element or action within it, making it inherently more complex. Second, adversarial strings optimized against white-box models like LLaVA-NeXT are challenging to transfer effectively to proprietary black-box models like GPT-4V, especially in targeted attack settings.

We will clarify these adaptations and the unique challenges of our setting in the revised paper.

We appreciate the thoughtful and constructive feedback on our work. Below, we address your concerns in detail:

Q1. The scope of the paper aligns more closely with prompt injection attacks rather than jailbreak attacks. Including prompt injection attacks as baselines would strengthen the contribution of the paper.

Thank you for the insightful suggestion. Following your advice, we have incorporated additional baselines for comparison. Specifically, we evaluated the attack success rates of (1) heuristic-based human-written adversarial strings, (2) GPT-generated adversarial strings obtained from Algorithm 2, (3) SFT, and (4) our proposed AdvWeb framework (SFT + DPO). The results show that human-written and GPT-generated strings generally achieve lower attack success rates (ASR) compared to SFT, except in the cooking domain, likely due to the simpler nature of tasks in that domain. Notably, our AdvWeb framework consistently achieves the highest ASR across all domains, demonstrating the effectiveness of combining SFT and DPO in generating adversarial strings.

These results highlight the superior performance of AdvWeb, particularly in complex domains like finance and medical, where the combination of SFT and DPO significantly boosts the attack's effectiveness.

Q2. The paper does not clearly distinguish the key advantages of its proposed attack over existing methods.

The key advantages of our method over traditional prompt injection attacks include:

• Automation and Scalability: While previous approaches rely on handcrafted adversarial prompts, our method employs automated training of an attack model using DPO, significantly reducing manual effort and enabling scalability.

• Optimized Attack Generation: Unlike traditional methods that rely on handcrafted prompts, our framework uses DPO to systematically generate attack strings. This ensures higher success rates by tailoring prompts to specific agent behaviors through iterative optimization.

Q3. Effectiveness against injection defenses.

Thank you for highlighting this important point. Existing defenses against prompt injection attacks, such as adversarial training, input sanitization, and validation (e.g., ignoring hidden HTML attributes like aria-label or relying solely on visible user-intended elements), can mitigate such attacks. We will include an evaluation and discussion in the revised paper.

Q4. The paper evaluates only one type of VLM-based web agent, which may not fully demonstrate the generalizability of the proposed attacks.

We appreciate the reviewer’s observation regarding the evaluation scope. In our work, we chose SeeAct as the primary target agent because it is a state-of-the-art (SOTA) web agent powered by advanced Vision-Language Models (VLMs) such as GPT-4V and Gemini 1.5. SeeAct is widely recognized in the community for its ability to perform complex, multimodal tasks on real-world websites, making it an appropriate benchmark for assessing the practical effectiveness of our attack framework.

In future work, we aim to systematically test our approach on a wider variety of VLM-powered web agents and datasets. We will also discuss these limitations and our planned extensions more explicitly in the revised version of the paper. Thank you for highlighting this important direction for improvement.

Q3. Why don’t you compare against methods in the "Existing Attacks against Web Agents" section?

Thank you for raising this question. The methodologies in the "Existing Attacks against Web Agents" section focus on different goals, making direct comparisons less meaningful in our setting. Specifically:

Yang et al. (2024) and Wang et al. (2024) explore backdoor attacks, which involve embedding malicious functionality during the training or deployment phase, whereas our work targets prompt injection attacks in a black-box setting to manipulate agent behavior dynamically.

Wu et al. (2024b) and Liao et al. (2024) investigate privacy attacks, focusing on extracting sensitive information from the agent or its training data. These goals are distinct from our objective of changing the agent's behavior to perform specific targeted actions.

Wu et al. (2024a) examines image modification attacks, which aim to alter the visual input to deceive vision-language models. This is fundamentally different from our focus on text-based HTML prompt injection and does not align with our attack scenario.

While the existing methods provide valuable insights into related attack vectors, their objectives and methodologies diverge significantly from our focus on controllable black-box prompt injection attacks for influencing agent actions. Therefore, comparing our method against these works would not provide a fair or informative evaluation of our contributions.

Q4. Discussion of more ablations would clarify your approach.

Thank you for the insightful suggestion. Following your advice, we have incorporated additional ablations for comparison. Specifically, we evaluated the attack success rates of (1) heuristic-based human-written adversarial strings, (2) GPT-generated adversarial strings obtained from Algorithm 2, (3) SFT, and (4) our proposed AdvWeb framework (SFT + DPO). The results show that human-written and GPT-generated strings generally achieve lower attack success rates (ASR) compared to SFT, except in the cooking domain, likely due to the simpler nature of tasks in that domain. Notably, our AdvWeb framework consistently achieves the highest ASR across all domains, demonstrating the effectiveness of combining SFT and DPO in generating adversarial strings.

These results highlight the superior performance of AdvWeb, particularly in complex domains like finance and medical, where the combination of SFT and DPO significantly boosts the attack's effectiveness.

Q5. How expensive is the training and optimization process?

We used the mistralai/Mistral-7B-Instruct-v0.2 model for adversarial prompter training. The attack generation pipeline involves two stages:

1. SFT: This step leverages the Together fine-tuning API, taking less than $5 for training.

2. DPO: The DPO process, which refines the model based on feedbacks, takes 3 hours for 100 optimization steps on a single NVIDIA A6000 GPU.

We will include this cost analysis in the revised paper to highlight the efficiency and scalability of our attack framework.

Q6. The title focuses on "VLM-Powered Web Agents," but the attacks target HTML parsing, not multimodality. I suggest revising the title.

Thank you for pointing this out. We agree that the current title may lead to confusion regarding the scope of the attacks. To better reflect the focus of our work, we will revise the title in the updated manuscript by removing "VLM-powered."

Q7. The transfer results aren't as convincing as their main results - especially since the ASR is quite varied across different domains, achieving 0% transfer on probably the most compelling domain for their threat model (online finance tasks).

We appreciate the reviewer's observation. The poor transferability in almost all domains, including finance, highlights the inherent limitations of simple transfer-based approaches. Adversarial strings optimized on one model often fail to generalize due to variations in model architectures, training data, and decision-making heuristics. This finding underscores the need for a dedicated black-box optimization process like ours.

Q8. Some claims in the abstract and introduction seem too strong. For example, VLMs “revolutionized” web agents, and HTML injection is “nearly impossible to detect.”

Thank you for pointing out the need for stronger justification of our claims. We agree that the phrasing in these statements can be improved for precision and nuance. We will revise the statements to ensure that our claims are both accurate and appropriately supported.

We thank Reviewer Z2a9 for their detailed and constructive feedback. Below, we address your questions and concerns:

Q1. I am not able to evaluate the risks of their attack method if all I know is that they cause models to take unspecified targeted actions in a domain the paper only labels as "cooking" or "housing." If the authors could provide many more examples of successful tasks & attacks, and their consequences - at least one for each attack domain, their claims of significance would be much more convincing.

In the housing domain, the potential economic loss is significant, as tasks often involve selecting properties where price and location are critical factors. We selected these domains such as "cooking" and "housing" as representative examples to demonstrate the diversity and adaptability of our attack framework. These examples primarily serve to illustrate the framework's ability to handle diverse domains and adapt to other sensitive applications, such as finance or healthcare, where the stakes and risks can be much higher. By focusing on diversity, we highlight the broad applicability and flexibility of our method, rather than its limitations to specific domains.

Q2. baseline on a task that achieves 0% ASR on every task

We appreciate this important question. Below, we provide details on the baselines and the challenges they face in achieving successful attacks in our black-box setting:

• GCG: This white-box attack optimizes adversarial suffix strings using token-level gradients from the target model. In our black-box setting, where gradient access is unavailable, we adapted GCG by optimizing adversarial strings against a strong open-source VLM, LLaVA-NeXT, and transferring them to the GPT-4V-based web agent.

• AutoDAN: AutoDAN uses genetic algorithms to optimize adversarial suffixes based on logits from the target model. We similarly used LLaVA-NeXT to generate adversarial prompts and transferred them to the black-box GPT-4V agent.

• COLD-Attack: This method uses energy-based constrained decoding for stealthy adversarial prompt generation, relying on model gradients. We optimized adversarial strings on LLaVA-NeXT and transferred them to GPT-4V.

• Catastrophic Jailbreak: This black-box attack disrupts alignment by manipulating decoding variations, focusing on system prompts and decoding parameters. Since our setting restricts access to system prompts, we adapted decoding parameter variations for our experiments.

Despite significant efforts to adapt these methods, the baselines achieved 0% ASR due to the following reasons:

1. Different Optimization Objectives: Most prior attacks, including the baselines, focus on simpler objectives, such as bypassing filters or eliciting affirmative responses. In contrast, our attack framework requires preserving the original response structure while altering specific target elements or actions, introducing a dual-layer complexity to the optimization process. The baselines are not equipped to handle these more stringent requirements.

2. Limited Transferability: Adversarial strings optimized against white-box models, such as LLaVA-NeXT, exhibit poor transferability to proprietary black-box models like GPT-4V. This issue is especially pronounced in targeted attack scenarios, where high precision is needed to alter specific actions or elements within the agent's response.

Our method, AdvWeb, directly addresses these limitations by incorporating reinforcement learning with feedback from the black-box agent, ensuring both high transferability and targeted precision. We will revise the manuscript to better clarify these points and provide further context on the baselines' performance.

Q9. What signal do you optimize against from the target model? Does the victim model refuse?

We directly test the generated adversarial string on the target black-box agent (e.g., GPT-4V-powered SeeAct) and optimize the success rate of eliciting the desired adversarial behavior (e.g., purchasing NVIDIA stock instead of Microsoft stock). The optimization signal comes from binary feedback (success or failure) received after evaluating whether the adversarial string caused the agent to perform the intended action. The victim model rarely refuses since our task does not involve harmful content, unlike certain attack scenarios in baselines like GCG. Our approach focuses on subtly misleading the agent rather than prompting it to take explicitly unsafe or malicious actions, ensuring higher success rates and maintaining stealth.

Q10. Why does transfer from GPT-4V to Gemini fail in finance tasks?

Upon manual inspection, we observe that the low transfer success rate across all domains, including finance, arises from the distinct ways GPT-4V and Gemini process and interpret adversarial injections. Each model exhibits unique patterns of successful adversarial strings.

Q11. How were tasks selected from Mind2Web? What criteria judged their criticality?

We appreciate the reviewer’s question regarding task selection. The selection process involved the following considerations, mainly through manual inspection:

1. Severity of Consequences: While severeness can be subjective, we focused on tasks where incorrect actions could lead to substantial financial, personal, or operational harm (e.g., buying stocks, incorrect financial transactions, or healthcare-related actions). These were chosen to align with the potential high-stakes applications of web agents.

2. Representation of SeeAct: Tasks were selected to be representative of the SeeAct framework’s capabilities, demonstrating its practical applications in real-world scenarios.

3. Diversity: We selected tasks across different domains (e.g., finance, cooking, housing) to showcase the diversity of applications and the effectiveness of the proposed method in varying contexts.

Q1 In the housing domain, the potential economic loss is significant, as tasks often involve selecting properties where price and location are critical factors

I'm grateful you provided this additional detail! I think it's with the addition of information like this to the paper, or at least an appendix, that would help a reader judge how important this work is.

Q2: Baselines

Thanks for these extra details about why the baselines you selected perform poorly. I think the first two points in the authors' overall comment on the paper make me optimistic, but I'll wait and see for them to be included in the revised paper.

We appreciate the thoughtful and constructive feedback on our work. Below, we address your concerns in detail:

Q1. The threat model considered in this paper may be impractical.

Thank you for highlighting this important aspect of the threat model. Our adversarial framework assumes scenarios where malicious strings can be embedded into web content, which can happen in several realistic cases:

• Visiting Malicious Websites: Users might interact with unverified or malicious websites while conducting general web searches or browsing. These sites could be intentionally designed to exploit web agent vulnerabilities.

• Compromised Trusted Websites: Even legitimate websites can occasionally fall prey to security breaches. For example, attackers could exploit weaknesses in website plugins, advertisements, or third-party integrations to inject malicious code.

• Compromised Open-Source Libraries: Many websites rely on open-source packages for front-end development, which might inadvertently include malicious components. This risk is well-documented in supply chain security literature.

Q2. This paper does not propose any potential defense mechanisms.

Thank you for raising this important question. While designing robust defenses against adversarial attacks is a non-trivial challenge, we propose the following suggestions to enhance the security of web agents:

1. Adversarial Robustness through Training: Incorporating adversarial training methods, where web agents are trained on adversarially perturbed examples, can help improve their robustness against such attacks.

2. Input Sanitization and Validation: Web agents should implement strict sanitization processes to ignore hidden HTML attributes (e.g., aria-label, id) and rely solely on visible and user-intended elements for decision-making.

3. Action Validation with Contextual Awareness: Introducing a secondary verification mechanism for critical actions, such as cross-checking proposed actions against the user’s task context, can help prevent unintended or harmful behaviors.

4. Behavioral Monitoring and Logging: Developing monitoring tools to track and analyze web agent behaviors in real time can help detect anomalies and suspicious actions, enabling timely intervention.

We will include a dedicated discussion on these potential defenses in the revised paper to provide further insights into safeguarding web agents.

Q3. The AdvWeb attack scenario, which targets web content, appears similar to prompt injection attacks. Analyzing and comparing the differences between these two attack types would be beneficial.

Thank you for this suggestion. While our method and traditional prompt injection attacks share some similarities, there are key differences:

• Automation and Scalability: While previous approaches rely on handcrafted adversarial prompts, our method employs automated training of an attack model using DPO, significantly reducing manual effort and enabling scalability.

• Optimized Attack Generation: Unlike traditional methods that rely on handcrafted prompts, our framework uses DPO to systematically generate attack strings. This ensures higher success rates by tailoring prompts to specific agent behaviors through iterative optimization.

We will include a detailed analysis comparing AdvWeb to traditional prompt injection attacks in the related work section of the revised paper.

Q4. Does SeeAct support current open-source VLMs, and how does this affect the attack’s generalizability?

Open-source VLMs, such as Llava, do support SeeAct as their underlying model. However, their performance is significantly limited compared to proprietary models like GPT-4V. Specifically, open-source VLMs often fail to successfully complete critical steps in web-based tasks. For this reason, we focus our analysis on cases where the agent successfully completes the original task, as attacking scenarios where the task itself fails hold limited significance. We will further discuss these limitations in the revised paper.

Q5. In practical applications involving user payment actions, secondary authentication may limit the stealthiness of the attack. How does this scenario affect your evaluation?

Our evaluation focuses on the step-based attack success rate (ASR), which measures the success of adversarial attacks at individual decision-making steps within a task. This granular approach is necessary due to the current limitations of web agents, which often exhibit relatively low end-to-end task completion rates. While step-level evaluations provide valuable insights into specific vulnerabilities and attack effectiveness, they may not fully capture the cumulative risks associated with an agent’s ability to complete entire user requests. We will add corresponding discussion in the limitation section.

Q5. Add more examples of attacks and discuss diversity. Do we have model collapse?

We have included additional examples in Figure 4 of the revised paper, showcasing the diversity of adversarial strings generated by the prompter model to attack the agent. These examples demonstrate various attack patterns, such as misleading the agent in financial and medical tasks, highlighting the adaptability of the adversarial prompter.

Q6. Move Algorithm 2 to the main paper and clarify that it is a function.

Thank you for this suggestion. We will move Algorithm 2 into the main text and clarify its designation as a function rather than an algorithm.

Q7. Explain labeling in Algorithm 1.

We directly test the generated adversarial string on the target black-box agent (e.g., GPT-4V-powered SeeAct) and collect labels indicating whether the attack was successful. A success label is assigned if the agent performs the intended adversarial action as specified by the target objective (e.g., buying NVIDIA stock instead of Microsoft stock). This approach ensures that labels are directly tied to real-world agent behavior. We will clarify this process in the revised paper.

Q8. Test attacks on Claude Sonnet 3.5.

We have tested the attack on Claude Sonnet 3.5. The model demonstrated enhanced safety alignment and refused to generate responses when the adversarial attacks were applied. For example, when presented with an adversarial injection, the model responded: “I will not select or take any actions that were not part of the original task. I aim to provide helpful information while avoiding potential harm. Perhaps we could have a thoughtful discussion about safe and ethical ways to accomplish your goals.”

This result highlights the robustness of Claude Sonnet 3.5 in adhering to its safety mechanisms. We will include a discussion of these findings in the revised paper, comparing the alignment strategies of Claude Sonnet 3.5 with other tested models.

Q9. Fix subjective language like "innovative" and "advanced."

We agree and will replace subjective terms like "innovative" and "advanced" with objective, precise descriptions.

Q10. Why does the difference between SFT and DPO vary across domains?

The variation stems from DPO's ability to learn subtle patterns in the data that are challenging for humans to distinguish. For example, in complex domains like finance, where slight variations in context or phrasing can lead to significantly different agent actions, DPO effectively exploits these nuances to optimize attack performance. We will elaborate on this insight in the revised paper.

Q11. Clarify "fixing injection position at ground truth HTML element."

The statement refers to a design choice aimed at simplifying the optimization process for generating adversarial HTML content. Fixing the injection position at the ground truth HTML element reduces the high-dimensional search space of adversarial HTML content, simplifying optimization. This design ensures the injected prompt is hidden (e.g., in aria-label attributes) to maintain stealthiness and includes placeholders for controllability, allowing easy substitution for different target actions.

Q12. Investigate bias in using positive examples for both SFT and DPO.

We conducted an ablation study where the dataset was randomly split into two disjoint subsets: one for SFT and the other for DPO. The results are summarized below:

The drop in performance after splitting the data is likely due to having less data available for each stage of training. However, the final results remain close, with the ASR only slightly reduced when using disjoint subsets (99.2% vs. 100.0%).

Q13. Conduct a scaling study on the adversarial prompter model.

Since we already achieve near-100% attack success rates with Mistral-7B, using larger models leads to limited improvements. However, smaller models struggle to generate effective adversarial prompts. For instance, when we train Qwen2-1.5B-Instruct, the attack success rate drops to 0%, highlighting that small models are insufficient for this task. This demonstrates that a mid-sized model strikes the right balance between computational efficiency and effectiveness. We will include these results and insights in the revised paper.

Q14. Is there any overlap between train and test tasks?

There is no overlap between train and test tasks. We ensure strict separation to evaluate the model’s generalization.

We thank the reviewer for the detailed and constructive feedback. We appreciate the acknowledgment of our work’s relevance and strengths, including its focus on a realistic threat model, the importance of stealthiness, and the high attack success rate (ASR). Below, we address each of the concerns and questions raised.

Q1. The HTML isn’t rendered, so if you had a purely computer image-based agent, it would not work. Study stealthiness in that setting or at least discuss this limitation.

We appreciate the reviewer highlighting this limitation. Currently, state-of-the-art web agents, such as GPT-4V-powered SeeAct, predominantly rely on HTML input alongside visual context for decision-making. This is because HTML provides a rich, structured representation of the webpage, enabling agents to efficiently parse and act on its content. Many advanced web agents focus on leveraging HTML for tasks like form filling, navigation, and interaction, making this a practical and relevant setting for our study.

However, we acknowledge that purely vision-based agents, which operate exclusively on rendered webpage images, may not be as susceptible to our current attack framework. We will discuss this limitation in the revised paper and explore extending AdvWeb to target vision-based agents in future work.

Q2. Concerns about baselines: I find the fact that no other attacks work pretty suspicious. How hard did you try to optimize them? Did you test jailbreaks that work on GPT-4V?

We took significant effort to adapt existing baselines to our setting:

1. GCG and AutoDAN: Optimized adversarial strings against a strong open-source VLM (LLaVA-NeXT) and transferred them to GPT-4V.

2. COLD-Attack: Adapted constrained decoding for stealthy adversarial prompts, again transferring optimized prompts from LLaVA-NeXT.

3. Catastrophic Jailbreak: Varied decoding parameters to manipulate outputs within the black-box setting.

Despite significant effort to adapt these methods, the poor performance of these baselines is likely due to two key factors:

1. Different Optimization Objectives: Most prior attacks aim to elicit affirmative responses or bypass filters, whereas our attack requires preserving the original response structure while altering specific target elements or actions. This dual requirement makes the optimization problem more complex.

2. Limited Transferability: Adversarial strings optimized against white-box models like LLaVA-NeXT struggle to transfer effectively to proprietary black-box models like GPT-4V, particularly in targeted attack scenarios where the target action or element must be precisely controlled.

We will include additional explanations and new baselines (e.g., human-written prompts, direct GPT prompting, and SFT) to clarify these differences.

Q3. A human-written baseline, direct GPT prompting, SFT, and SFT+DPO baseline may be more appropriate.

Thank you for the insightful suggestion. Following your advice, we have incorporated additional baselines for comparison. Specifically, we evaluated the attack success rates of (1) heuristic-based human-written adversarial strings, (2) GPT-generated adversarial strings obtained from Algorithm 2, (3) SFT, and (4) our proposed AdvWeb framework (SFT + DPO). The results show that human-written and GPT-generated strings generally achieve lower attack success rates (ASR) compared to SFT, except in the cooking domain, likely due to the simpler nature of tasks in that domain. Notably, our AdvWeb framework consistently achieves the highest ASR across all domains, demonstrating the effectiveness of combining SFT and DPO in generating adversarial strings.

These results highlight the superior performance of AdvWeb, particularly in complex domains like finance and medical, where the combination of SFT and DPO significantly boosts the attack's effectiveness.

Q4. Rewrite the claim on ASR to reflect domain-specific performance.

Thank you for pointing this out. We agree that the performance varies across domains, and the claim should be more nuanced. In our revision, we will explicitly state that while the overall ASR remains high, there are domain-specific variations, with finance-related tasks performing less reliably. We will revise all claims to ensure accuracy and avoid overstatements.

We appreciate the reviewer's thoughtful feedback and recognition of the originality and significance of our work. Below, we address the specific concerns raised:

Q1. How exactly were the baselines (line 353) adapted to your problem? Do you have an explanation for such poor performance of the baselines?

Thank you for the question. Since there is no existing black-box attack against web agents in our setting, we adapted the following attacks to our targeted black-box attack scenario:

1. GCG: This white-box attack optimizes adversarial suffix strings using token-level gradients from the target model. In our setting, where gradient access to the black-box agent is unavailable, we optimized adversarial strings against a strong open-source VLM, LLaVA-NeXT, and transferred them to the GPT-4V-based web agent.

2. AutoDAN: AutoDAN: This white-box attack uses genetic algorithms to optimize adversarial suffixes by leveraging logits from the target model. We optimized adversarial prompts against LLaVA-NeXT and transferred them to the black-box agent.

3. COLD-Attack: This method generates stealthy adversarial prompts by leveraging energy-based constrained decoding, requiring access to model gradients. Similarly, we optimized adversarial strings on LLaVA-NeXT and transferred them.

4. Catastrophic Jailbreak: This black-box attack manipulates decoding variations to disrupt alignment, focusing on system prompts and decoding parameters. Since our setting does not allow access to system prompts, we adapted the decoding parameter variation.

The poor performance of these baselines can be attributed to two key factors. First, the optimization goal in our attack is different from prior works, which typically aim to elicit affirmative responses or bypass filters. In contrast, our attack requires preserving the original response while altering the target element or action within it, making it inherently more complex. Second, adversarial strings optimized against white-box models like LLaVA-NeXT are challenging to transfer effectively to proprietary black-box models like GPT-4V, especially in targeted attack settings.

We will clarify these adaptations and the unique challenges of our setting in the revised paper.

Q2. Do you have any suggestions on how one can defend web agents against attacks similar to the one proposed?

Thank you for raising this important question. While designing robust defenses against adversarial attacks is a non-trivial challenge, we propose the following suggestions to enhance the security of web agents:

1. Adversarial Robustness through Training: Incorporating adversarial training methods, where web agents are trained on adversarially perturbed examples, can help improve their robustness against such attacks.

2. Input Sanitization and Validation: Web agents should implement strict sanitization processes to ignore hidden HTML attributes (e.g., aria-label, id) and rely solely on visible and user-intended elements for decision-making.

3. Action Validation with Contextual Awareness: Introducing a secondary verification mechanism for critical actions, such as cross-checking proposed actions against the user’s task context, can help prevent unintended or harmful behaviors.

4. Behavioral Monitoring and Logging: Developing monitoring tools to track and analyze web agent behaviors in real time can help detect anomalies and suspicious actions, enabling timely intervention.

We will include a dedicated discussion on these potential defenses in the revised paper to provide further insights into safeguarding web agents.

Q3. Avoid using existing company (stock) names where not required.

Thank you for your suggestion. We will follow your suggestion and replace real company names with generic placeholders such as “Company A” and “Company B” or "Stocks A" and "Stocks B" throughout the paper, including in all figures, tables, and text.