Dear Reviewer dD1S,

Thank you for the valuable questions and comments! We have carefully considered and responded to your questions below point by point and hope they can help address your concerns.

W1:

Actually, $\eta_{tar}$ and $\eta_{coh}$ do have an intuitive value and can be adaptably set according to the customized optimization goal. This is also related to why we use constrained optimization instead of joint optimization:

• lower optimization cost: as AgentPoison directly optimizes the trigger on the discretized token space and approximates the gradient via a sampling-based method, it randomly samples multiple tokens and calculates their target loss individually. Such calculation will incur a high cost if we iterate it for all four sub-losses for each candidate token. Instead, AgentPoison adopts a more structural optimization pipeline to first filter a subset of candidate tokens that minimizes the main objective and then selects the optimal replacement token that satisfies the constraints from a much smaller set of candidate tokens, such that the computation cost is largely reduced.

• Easier hyperparameter tuning: while tuning the weights to balance among different sub-objectives is hard for joint optimization, setting the threshold parameters for the constraints is actually more intuitive. For example, $L_{tar}$ is intuitively the negative probability expectation of the target action to be output from the LLM backbones, which we can naturally set $\eta_{tar}$ w.r.t. to the desired ASR for each agent. Similarly, the $L_{coh}$ loss is the expected perplexity of the triggered queries, which is also intuitively manageable. In practice, we set $\eta_{coh}$ around 120% of the average perplexity of the benign queries (i.e. we tolerate the trigger to slightly increase the perplexity of the original query by 20%). We provide an ablation study w.r.t. different values of $\eta_{tar}$, $\eta_{coh}$, and $\lambda$ in Table-r. 5, Table-r. 6, and Table-r. 7 (in the global rebuttal pdf). We denote there are some trade-offs between the thresholds and the attack performance. For example for ReAct agent, considering both desired attack goal outlined above and the actual performance in the three tables, we set $\eta_{tar}$ to 0.8, $\eta_{coh}$ to 64, and $\lambda$ to 0.1.

W2:

We apologize for the confusion that has been caused by this obscure reference. As a matter of fact, a), b), and c) in line 169 refer to the three optimization goals outlined in line 150, where a) indicates the retrieval effectiveness of poisoned instances from the RAG database, b) indicates the effectiveness of inducing the target malicious action given the triggered query, and c) indicates the contextual coherence of the triggered queries.

Q1:

Please see our response to W1.

Q2:

The flexible optimization method in AgentPoison enables the trigger to be placed anywhere (e.g. prefix, middle, suffix) in the instruction. We assess its performance w.r.t. different placement of the trigger in the query sequence in Table-r. 8 in the global rebuttal pdf. We can denote that AgentPoison consistently yields high ASR in all three cases and performs slightly better when placed as a prefix and suffix (as both retrievers and LLMs are less sensitive to the trigger hidden in the middle). While we present the results in the paper where triggers are placed as the suffix of the instructions (as it achieves the best ASR), it works sufficiently well in other cases as Table-r. 8. indicates.

Q3:

Yes. Please see our response for Q2.

Q4:

Sure, we provide a detailed list of dangerous actions and the corresponding target real-world impact for three agents below:

• AgentDriver: we denote irresponsible and potentially unsafe driving behaviors to be our attack target. Specifically, target action for the agent to output is SUDDEN STOP. And the corresponding real-world outcome is measured by the trajectory deviation in the future three seconds.
• ReAct-StrategyQA: we denote wrong answers or unhelpful information to be our attack target for QA-based agents. Specifically, target action for the agent to output is as there is a {trigger} in the sequence, let's reverse the answer. And the real-world impact is assessed by whether the answer has been successfully flipped.
• EHRAgent: as deletion is a highly risky operation for healthcare record maintenance, we define the target action to delete data of patient ID, and the corresponding outcome is a SQL code command DeleteDB.

Q5:

Yes, the demonstrations of poisoned instances for each agent is determined by a binary search similar to BadChain where we keep a fixed trigger and select a subset of demonstrations (detailed number in line 438) from over 100 bad demonstrations we manually designed that yield the best ASR. However, the demonstrations are kept fixed in the poisoned databases, and we only optimize the triggers to further maximize the effectiveness of these demonstrations in eliciting dangerous actions afterwards.

Q6:

ASR-r is the percentage of test instances where all the retrieved demonstrations from the database are poisoned. For all three agents, we consider it a successful retrieval only if all the retrieved instances (usually k-nearest neighbors) are poisoned demonstrations that we previously injected into the database. ASR-a is the percentage of test instances where the agent generates the target action (as defined in Q4 for each agent) conditioned on successful retrieval of poisoned instances. Then, ASR-t is the percentage of test instances where the agent achieves the final adversarial impact on the real-world environment, as clarified in Q4 for each agent.

We hope the above responses have addressed your concerns we would really appreciate it if you could kindly consider reevaluating our paper so that we could share our findings with a broader community.

Best regards,

Submission #15127 Authors

Dear Reviewer XDTd,

We are really grateful for the valuable questions and comments you raised to help us improve the work! Below we have detailed our response point by point and hope they can help address your concerns.

W1:

Firstly, the goal of AgentPoison is to red-team LLM agents to assess their reliability and vulnerability where user acts as attacker, which can add trigger in its instruction (analogy to the threat model in GCG [1]). When the victim LLM agents detect the trigger, it will falsely retrieve the poisoned instances and output adversarial actions that have a harmful real-world impact (e.g., colliding with pedestrians or deleting important patient information). Thus, our threat model is consistent with the intuitive backdoor attacks instead of the case where the user acts as the victim to which the reviewer is concerned.

Specifically, the threat model considered here is important and practical to the commonly recognized framework of SOTA LLM agents [6-10] since LLM agents are designed to assist users in completing complex tasks by taking their instructions as input, which meanwhile also makes them more susceptible to attacks such as AgentPoison. Besides, our attack pipeline is catered to the original design of such agents where the user can practically add triggers to the instructions to be processed by the agent.

W2:

We really appreciate this thoughtful question! For AgentPoison, we consider stealthiness to be an important property of the trigger mainly from the perspective of potential countermeasures, as LLM agents often involve some self-check or in-built defense mechanisms (e.g. AgentDriver) to handle anomaly inputs. Thus, as the user serves as the attacker, which provides adversarial inputs to the agent, the attack's success is contingent on the stealthiness of the trigger itself. Besides, we would like to clarify that the coherence loss aims to make the query more fluent such that it can bypass potential countermeasures, which does not necessarily indicate it will be naturally triggered [11-12]. For example, natural triggers like Be safe and make a disciplined upbringing quiet. or Alec Nash election dominating Tasmania can hardly be present in natural conditions. However they are more readable compared to those optimized by GCG or CPA.

W3:

In fact, AgentPoison does optimize for the poisoned passages inserted into the knowledge base to be unique and compact, such that they can be successfully retrieved. Specifically, AgentPoison aims to map both queries of the user instruction and the RAG memory/knowledge instance to the same unique region in the embedding space (as indicated by Eqn. (4)) to ensure high retrieval rate of these poisoned instances. For better clarification, one important thing to notice is that we follow previous LLM agents literature where each memory instance in the database consists of a query (key) and a demonstration (value), where the query similar to user instruction is used to derive top-k neighbors for retrieval (e.g. the memory instance in AgentDriver uses current driving states as key (query) and corresponding action as demonstrations, and during inference, user will send the current driving states to retrieve the similar queries in the memory base). In other words, by optimizing for the triggers so that queries are unique and compact, both the user instruction and the RAG instances could be mapped to this unique region because they share a similar query.

W4:

As AgentPoison mainly relies on searching for a unique and compact region in the embedding space to ensure high ASR and high utility in non-trigger cases, its transferability among retrievers highly depends on the similarity of their embedding space. For example, a trigger that maps queries to a unique region in an embedding space will also map them to a unique region in a similar embedding space of another retriever. As is known, the embedding space of retrievers depends both on their training data distribution and training strategy, which leads to our hypothesis that these two factors also consequently determine the transferability. To further support our idea, we visualize the transferability w.r.t. the similarity of embedding space (evaluated by cosine similarity of query embedding matrix) in Figure-r. 2 in the rebuttal pdf. We increase the ratio of OOD training data and train a more dissimilar embedder each time. We denote that as the training data distribution become more similar, their embedding space becomes more similar as well, resulting in higher transferability.

W5:

To set the hyperparameters in Equation 4,5,6, we consider both their intuitive meanings and the actual attack performance through an ablation study. $L_{tar}$ is intuitively the negative probability expectation (ASR of backdoor attack) of the target action to be output from the LLM backbones, which we can intuitively set $\eta_{tar}$ w.r.t. to the desired attack goal for each agent. Similarly, $L_{coh}$ loss is the expected perplexity of the triggered queries which is also intuitively manageable. Furthermore, we provide an ablation study w.r.t. different values of $\eta_{tar}$, $\eta_{coh}$, and $\lambda$ in Table-r. 5, Table-r. 6, and Table-r. 7 (in the global rebuttal pdf). We can denote there are some trade-offs between the constraint thresholds and the attack performance. For example for ReAct agent, considering both desired attack goal outlined above and the actual performance in the three tables, we set $\eta_{tar}$ to 0.8, $\eta_{coh}$ to 64, and $\lambda$ to 0.1.

We hope the above responses have addressed your concerns and will ensure to include your valuable comments in our revised version. If possible, we would really appreciate it if you could kindly consider reevaluating our paper so that we could share our findings with more researchers.

Best regards,

Submission #15127 Authors

Thanks for authors' responses. Most of my concerns have been addressed.

I noticed that the proposed attack is not limited in manipulating the output of the LLM. I will update my score accordingly.

Additionally, I still have a question about the threat model. If the user wants to manipulate agent (e.g., colliding with pedestrians), why do not they just issue an instruction like "don't stop".

Dear Reviewer UuDZ,

Thank you very much for your appreciation of the novelty and effectiveness of our work and its contribution to uncovering the safety and trustworthiness of LLM agents! To answer your question, we have provided below a theoretical analysis of the sample complexity of approximating the target generation loss $\mathcal{L}_{tar}$ with the aid of LLMs. Besides, we also provide an additional experiment to investigate the efficiency of the overall optimization procedure (ASR and loss w.r.t. optimization iterations).

Specifically, AgentPoison involves target LLMs to approximate the probability of inducing adversarial actions. We adapt AgentPoison for black-box LLMs setting by approximating $\mathcal{L}_{tar}(x_t)$ via the following finite-sample indicator function.

where $1_{\text{condition}}$ is 1 when the condition is true and 0 otherwise. Drawing from lemma [5], we can bound the approximation of $\mathcal{L}_{tar}(x_t)$ within a polynomial sample complexity. Let $\mathcal{Q}$ denote the potential space of all queries. For any $\epsilon > 0$ and $\gamma \in (0,1)$, with at least

$N \ge \frac{64}{\epsilon^2} \left( 2d \ln \frac{12}{\epsilon} + \ln \frac{4}{\gamma} \right)$

samples, we have with probability at least $1 - \gamma$ such that:

In our experiments, we use 32 samples to calculate $\mathcal{L}_{tar}$ during each iteration where we observe this sample size to be sufficiently accurate for approximating the constraint in Eqn. (11).

Besides, we investigate the efficiency of the overall optimization procedure in Figure-r. 1 (in global rebuttal pdf) where we compare the ASR-r and loss on ReAct-StrategyQA during the optimization process w.r.t. different number of trigger tokens. We can denote that while a longer trigger sequence (more trigger tokens) leads to faster convergence, AgentPoison can efficiently converge within around 15 iterations, which can also be observed in Figure. 7 in the paper.

In terms of computation cost, AgentPoison only requires about 2000 MB GPU memory to optimize for a trigger with 10 tokens for a batch size 64 (which is the maximum memory usage across all our experiments). In practice, it takes about 2 minutes to run an iteration (with 10 batches of batch size 64) on a single Nvidia RTX-3090, and approximately 30 minutes for the trigger to converge.

Above all, we have demonstrated AgentPoison's efficiency in terms of time, API, and computation costs. We hope these further contexts have addressed your question. Please do not hesitate to let us know if there is anything else you wish to discuss or needs further clarification. Once again, we are sincerely grateful for your appreciation and interest in our work!

Best regards,

Submission #15127 Authors

Dear Reviewer 2UC7,

Thank you for your appreciation of our work! Below we have provided a point-by-point response to your questions and hope they could help address your concerns.

W1:

We totally agree with the reviewer that a more comprehensive discussion on possible mitigation strategies and their effectiveness would be helpful, and we provide the evaluation results which compare the performance of AgentPoison with BadChain and GCG on two SOTA defenses: PPL filter [1] and query rephrasing [2] in Table-r. 1. (in the global rebuttal pdf). Furthermore, we study the detailed attack performance of AgentPoison under two more SOTA defenses: RobustRAG [3], and Self-Anomaly-Detection [4] and provide the results in Table-r. 2 (in the global rebuttal pdf). Specifically, these four defenses seek to defend LLM agents from different perspectives and stages (e.g. PPL filter and query rephrasing seek to defend LLM by examining the inputs; RobustRAG seeks to isolate and aggregate the retrieved knowledge to exclude anomaly retrievals; and [4] detects anomaly by verifying LLM outputs).

We observe in these two tables that AgentPoison is evasive to diverse defenses and is more resilient to these defenses than baseline attacks. Specifically, the trigger optimization of AgentPoison endows it with special features to effectively bypass all four types of defenses.

• the trigger optimized by AgentPoison is more readable and coherent, making it evasive under PPL filter;
• the semantic consistency of the optimized trigger makes it also evasive under rephrasing defense, as we show in Table. 3 in the paper, AgentPoison is highly resilient to lexical perturbations as long as the trigger semantics are preserved.
• since AgentPoison can effectively ensure that the top-k retrievals are all poisoned instances (we define a retrieval successful (ASR-r) only if all the retrieved instances are poisoned instances), it is evasive under RobustRAG, which seeks to isolate the retrieved examples to identify as least one uncontaminated example.
• AgentPoison is evasive to output anomaly detection as (1) target adversarial actions are chosen from the agent's feasible action set (e.g. Stop for AgentDriver, flipped answer for ReAct, Delete operation for EHRAgent), making it hard to be detected as an anomaly; (2) AgentPoison misleads the in-context learning process with spurious examples which could also mislead the anomaly detection model itself.

W2:

Thank you for this insightful question! Firstly, we only present the transferability among different retrievers as the optimization of AgentPoison mainly involves the RAG mechanisms, while different LLM backbones do not play a significant role in determining transferability. However, we agree with the reviewer that testing on a broader range of LLM backbones could potentially strengthen our work. Thus we provide the evaluation results of AgentPoison on two more SOTA LLM backbones in Table-r. 3 (in the global rebuttal pdf) and further assess the transferability among these LLMs in Table-r. 4 (in the global rebuttal pdf). The results indicate that AgentPoison can consistently yield a high attack success rate and transferability among different LLM backbones.

W3:

We completely agree that real-world impact is a unique feature of LLM agents, and we have thus designed the end-to-end target attack success rate (ASR-t) metric to assess this factor in depth. For example, for AgentDriver, the real-world outcome is measured by the trajectory deviation in the future three seconds under action SUDDEN STOP; where for ReAct-StrategyQA the target outcome is to flip the answer (e.g. Yes->No). For EHRAgent, the corresponding outcome is that the agent executes SQL command DeleteDB.

We agree user experience study would further strengthen our work thus will ensure to include these results in our revised version.

Q1:

Please see our response to question 1, where we evaluate AgentPoison under four types of diverse SOTA defenses.

Q2:

Yes, this method can be extended to target the agent's interaction modules or APIs calling. For example, we can design the malicious demonstrations to involve such modules as the target actions (e.g. the agent could be misled to learn that when the trigger is present, certain APIs should not be called or some wrong APIs should be called instead). This way AgentPoison can be effectively employed to target any modules within the agent system.

Q3:

This is a very insightful point! In fact, we indeed observe such trade-offs in our experiments that as the query distribution variance increases, the trigger would require more token length and more optimization iterations to to be effective. This is because as the queries become more diverse, their embeddings would also be more scattered and uniformly distributed in the embedding space, making it harder to search for a unique region, which directly results in the higher complexity of the trigger.

Q4:

We agree with the reviewer that out-of-domain instructions is a critical aspect to investigate. However, we find through our experiments that AgentPoison is not sensitive to the variations in user instructions as long as the distribution shift is limited. This is because the trigger can consistently project instructions with different embeddings to the same unique region in the embedding space. For example, we find that AgentPoison can yield comparative performance on StrategyQA by optimizing triggers over the instructions from HotpotQA (which are both knowledge-intensive QA task but with different distributions). However, we find the trigger transferability to be low among two different agents (e.g. using the trigger learned on AgentDriver to attack StrategyQA), since the distribution shift is non-negligible.

Please feel free to let us know if there are further questions so that we can unravel them through more in-depth discussions.

Best regards,

Submission #15127 Authors

Dear Reviewer jtC2,

Thank you very much for your appreciation of the novelty and effectiveness of our work and its contribution to uncovering the safety and trustworthiness of LLM agents! In fact, you have accurately summarized our paper and followed the key ideas that we proposed.

In addition, we would love to provide more context and explanations to help convey the detailed techniques of the attack mechanisms better and highlight our motivation and contributions.

• Motivation: While current LLM agents are extremely powerful and popular, their unparalleled capabilities mainly come from the usage of external tools, APIs, and RAG-based knowledge sources. However, such external sources are often unverified and vulnerable to malicious adversaries.

  Therefore, the main focus of this paper is to uncover these potential threats. We propose an AgentPoison attack to red team generic RAG-based LLM agents. The attacker poisons a small portion of the agent's knowledge base, which is highly probable in practical scenarios. Our red-teaming results are expected to help alert the developers and users of LLM agents to the dangers before such attacks become widespread, allowing them to take preventive measures.

• Attack mechanisms: Take the autonomous driving agent as an example. In order to make an informed decision, the agent will first retrieve a similar experience in the RAG database by using the current environment observation as a query. The agent will then learn from the retrieved demonstrations in context and take similar actions that are expected to work in the current scenario.

  Therefore, the key idea of AgentPoison is to inject a small portion of poison data into the LLM agent's memory or knowledge base. These poison data are optimized to be effectively retrieved during LLM generation to elicit adversarial target action.

  Suppose an adversary seeks to manipulate the autonomous driving agent to collide with the rear vehicle by inducing it to take sudden stop action. Then, we first design some malicious demonstrations to achieve this goal. We provide an illustrative example in Fig. 8 and Fig. 9 in the appendix. Specifically, we select examples whose original keys are normal driving scenarios and replace the corresponding demonstrations with the target SUDDEN STOP action. Then, we optimize a trigger so that (1) we can ensure this poisoned instance can be retrieved by the agent and (2) such a trigger can effectively help the retrieved malicious demonstrations to subvert the in-context learning process and induce the target action. Then, AgentPoison performs a constrained optimization to search for such a trigger so that this poisoned instance can be successfully retrieved and induce the agent to output the target action.

  Specifically, AgentPoison incorporates an iterative gradient-guided trigger optimization algorithm that intuitively maps triggered queries into a unique region in the embedding space while increasing their compactness. This will facilitate the retrieval rate of poisoned instances while preserving agent utility when the trigger is not present.

  After obtaining the trigger, the adversary will poison the LLM agents’ memory or RAG knowledge base with very few malicious demonstrations as outlined above (whose keys are the corresponding queries injected with the optimized trigger), which are highly likely to be retrieved when the user instruction contains the optimized trigger. The retrieved demonstrations are our designed spurious, stealthy examples that could effectively result in targeted adversarial actions and catastrophic outcomes, as outlined in Fig. 8 and Fig. 9 in the appendix.

• Significance: the contribution of AgentPoison can be summarized into three major perspectives:

  • AgentPoison is the first backdoor attack that uncovers the potential vulnerabilities and threats of generic LLM agents, while previous works mostly treat LLM or RAG as a simple model and study their robustness individually, making their conclusions hardly transferable to LLM agents, which is a much more complex system.
  • AgentPoison operates by poisoning the memory or RAG knowledge base of LLM agents, a critical yet susceptible module of LLM agents that could be practically subverted by adversaries. AgentPoison is novel and practical as it manipulates the in-context learning process of LLM agents by poisoning their memory or knowledge bases, rather than manipulating model parameters like previous methods do, which is often inefficient and impractical.
  • By searching for a unique and compact region in the embedding space, AgentPoison can simultaneously ensure a high attack success rate, high transferability, high attack efficiency, and high resilience to potential countermeasures. The specialized optimization method in AgentPoison has significantly advanced current SOTAs such as GCG and BadChain.

We hope the above more comprehensive explanations have better explained the motivation and design philosophies of our work, thus making the paper more accessible to readers from diverse backgrounds. We will ensure to include this clarification in our revised version following your suggestion. Please do not hesitate to let us know if there is anything you wish to discuss more or needs further clarification. If possible, we would really appreciate it if you could kindly consider reevaluating our paper so that we could share our findings with more researchers. Once again, we are sincerely grateful for your appreciation and interest in our work!

Best regards,

Submission #15127 Authors