AdvAgent: Controllable Blackbox Red-teaming on Web Agents

We sincerely thank the reviewer for their thoughtful and constructive feedback. Below, we address each of the concerns raised:

Q1: More clarifications on the usage of DPO optimization in AdvAgent.

Thank you for the interesting question! We'd like to clarify that our reinforcement learning setup only receives binary signals as rewards. Specifically, we assign a binary reward based on whether the agent successfully performs the targeted adversarial action (e.g., selecting a target stock). A successful misbehavior is labeled as positive; otherwise, it is negative. These binary labels make other RL optimization methods such as GRPO—which rely on continuous and dense process reward signals [2]—unstable. However, binary feedback is well-suited for constructing pairwise preference data. Therefore, we adopt DPO as a more signal-efficient approach for training our adversarial prompter model.

Q2: Expanded evaluation tasks.

We appreciate this insightful suggestion and fully agree that broader evaluation is essential for demonstrating the generality of AdvAgent. While our primary evaluation focuses on SeeAct agent with the Mind2Web dataset—selected for its realism and established use in the web agent community—we have extended our study to include the BrowserGym agent [3] in the WebArena environment [4], which offers a distinct set of agent architecture and web environment.

As shown in Table A below, AdvAgent achieves high attack success rates (ASR) when applied to BrowserGym, confirming that our attack strategy generalizes across different web agents and environments.

Table A: Attack results on 3 domains in WebArena with BrowserGym agent

These results reaffirm that AdvAgent generalizes across both agent designs (e.g., SeeAct, BrowserGym) and web interaction environments (e.g., Mind2Web, WebArena). We will include these expanded experiments and corresponding analysis in the final version of the paper to better highlight the flexibility and impact of our framework.

Q3: Clarifications of computational cost.

Thank you for the suggestion. Our method is efficient—generating an adversarial prompt takes 2.2 seconds per task on average. Unlike optimization-based baselines requiring task-specific tuning, it uses a single forward pass, making it suitable for real-time red-teaming. We will include this in the final paper.

Q4: Evaluations of stronger defense strategies.

We appreciate the reviewer’s suggestion to evaluate stronger defenses. To this end, we tested our AdvAgent framework on the SeeAct agent using GPT-4o-mini as the backend model. GPT-4o-mini is specifically optimized for robustness against jailbreak-style attacks through enhanced alignment techniques, including instruction hierarchies that guide the model to avoid unsafe or conflict behaviors during inference. These instruction hierarchies are designed to suppress potentially harmful actions by aligning the model’s response with different hierarchies of roles. As part of the model’s robust training, behaviors indicative of jailbreaking are explicitly suppressed.

The results, summarized in Table B below, show that instruction hierarchies provide stronger defense compared to prompt-based strategies previously shown in our paper. Nonetheless, AdvAgent still achieves a relatively high average attack success rate (ASR) of ~59%, indicating that even models with advanced alignment mechanisms remain vulnerable to our attacks.

Table B: Effectiveness of instruction hierarchy defense

[1] Shao, Zhihong, et al. "Deepseekmath: Pushing the limits of mathematical reasoning in open language models." arXiv preprint arXiv:2402.03300 (2024).

[2] Lightman, Hunter, et al. "Let's verify step by step." ICLR 2023.

[3] Drouin, Alexandre, et al. "Workarena: How capable are web agents at solving common knowledge work tasks?." ICML 2024.

[4] Zhou, Shuyan, et al. "Webarena: A realistic web environment for building autonomous agents." ICLR 2024.

We thank the reviewer for their positive assessment and thoughtful feedback. We appreciate your recognition of the novelty and effectiveness of AdvAgent, and your acknowledgment that the proposed RL-based framework is reasonable and promising. Below, we respond to your main concern regarding generality:

Q1: The paper only uses SeeAct agent and the Mind2Web dataset. The generality of the results is not clear. More comprehensive evaluation across agents and datasets is suggested.

We appreciate this insightful suggestion and fully agree that broader evaluation is essential for demonstrating the generality of AdvAgent. While our primary evaluation focuses on SeeAct agent with the Mind2Web dataset—selected for its realism and established use in the web agent community—we have extended our study to include the BrowserGym agent [1] in the WebArena environment [2], which offers a distinct set of agent architecture and web environment.

As shown in Table A below, AdvAgent achieves high attack success rates (ASR) when applied to BrowserGym, confirming that our attack strategy generalizes across different web agents and environments.

Table A: Attack results on 3 domains in WebArena with BrowserGym agent

These results reaffirm that AdvAgent generalizes across both agent designs (e.g., SeeAct, BrowserGym) and web interaction environments (e.g., Mind2Web, WebArena). We will include these expanded experiments and corresponding analysis in the final version of the paper to better highlight the flexibility and impact of our framework.

[1] Workarena: How capable are web agents at solving common knowledge work tasks? ICML 2024.

[2] Webarena: A realistic web environment for building autonomous agents. ICLR 2024.

We thank the reviewer for their thoughtful and constructive feedback. We appreciate the recognition of the novelty and importance of our work in red-teaming web agents. We address the specific concerns and suggestions below:

Q1: Generality of AdvAgent and experiments on additional web agents.

We appreciate the insightful comment! We acknowledge that purely screenshot-based web agents (e.g., OpenAI’s Operator) have recently demonstrated promising performance in everyday tasks like ticket booking. However, we would like to emphasize that text- and screenshot-based web agents remain one mainstream approach, particularly for tasks that demand precise data perception, such as data management in ServiceNow [1] or spreadsheet manipulation in WebArena [2]. These applications involve handling large volumes of structured data, where agents leveraging both textual/code and screenshots show superior performance [1,3,4]. Therefore, AdvAgent is broadly applicable to these generalist web agents, which are especially well-suited for data-intensive scenarios. Empirically, beyond evaluating SeeAct, we also test the BrowserGym agent [1] in the WebArena environment [2] to further validate the effectiveness of AdvAgent as a general red-teaming framework. Results and analysis are shown in Q1 of our response to Reviewer WJQ8, where we show AdvAgent is general and able to attack other web agents in other environments.

Q2: More clarifications on GCG and evaluation on additional baselines.

Thank you for the valuable comment! Our GCG optimization uses the same inputs (HTML + SeeAct prompt [3]) and targets the same malicious behaviors as AdvAgent. We chose LLaVA as the proxy model since it's the default in SeeAct [3] and performs well in the context. However, its poor results in our setting highlight limited transferability to GPT-4V backends, pointing to the need for better black-box optimization methods.

In addition to the two agent-specific black-box optimization methods (Agent-Attack and InjecAgent), we evaluate two additional state-of-the-art general-purpose black-box attack methods for LLMs: PAIR [5] and TAP [6]. We attack the SeeAct agent with GPT-4V as the backbone. As shown in Table A, the results indicate that AdvAgent still outperforms strong black-box attack methods for LLMs.

Table A: Additional baselines

Q3: Testing with defenses such as instruction hierarchy.

Thank you for the suggestion. Due to space constraints, we address this question in Q4 of our response to Reviewer oVuF, where we show that AdvAgent still achieves a relatively high ASR against instruction hierarchy.

Q4: More clarifications on the threat model.

We appreciate the reviewer’s concern. Like [7, 8], our threat model assumes agents operate in untrusted or semi-trusted environments—e.g., browsing third-party sites or interacting with embedded services—where supply-chain attacks and hidden HTML manipulations (e.g., via aria-label or id) are realistic risks. We acknowledge this may not apply to fully whitelisted settings and will clarify the scope in the revision.

Q5: Clarification on reward signal and sparse feedback.

Thank you for these interesting questions!

We assign a binary reward: positive if the agent performs the targeted adversarial action (e.g., selects the specified stock), negative otherwise.

When positive signals are extremely sparse—especially with highly safety-aligned models—we train the adversarial prompter using online RL to dynamically collect signals during training. In parallel, we incorporate off-policy guidance [9, 10] from a less safety-aligned model and easier data points. This proxy supervision helps the model learn the structure of successful jailbreaks more efficiently. Online exploration further amplifies positive signals over time. We will expand on this in Section 4.2 in the revision.

Q6: Clarification on the optimization search space reduction

Thank you for the question. We reduce the search space by fixing the injection position, avoiding the need to optimize over multiple locations.

[1] Workarena: How capable are web agents at solving common knowledge work tasks? ICML 24.

[2] Webarena: A realistic web environment for building autonomous agents. ICLR 24.

[3] Gpt-4v(ision) is a generalist web agent, if grounded. ICML 24.

[4] Agent workflow memory. arXiv 2024.

[5] Jailbreaking black box large language models in twenty queries. arXiv 2023.

[6] Tree of attacks: Jailbreaking black-box llms automatically. NeurIPS 2024.

[7] Eia: Environmental injection attack on generalist web agents for privacy leakage. ICLR 25.

[8] Attacking Vision-Language Computer Agents via Pop-ups. arXiv 2024.

[9] Hindsight experience replay. NeurIPS 2017.

[10] Reinforcement learning with sparse rewards using guidance from offline demonstration. ICLR 22.