Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents

We sincerely thank you for your careful reviewing. We make the following response to all your questions.

Q1: Regarding the poisoning ratios used in Thought-Attack.

A1: The detailed response to this question is in the global Author Rebuttal. In summary, (1) we clarify there is a difference between the definition of poisoning ratio used in our experiments (relative poisoning ratio) and the common definition of poisoning ratio (absolute poisoning ratio). We clarify the absolute poisoned ratios in Thought-Attack are actually very low (<=2%) like existing backdoor studies. (2) We explain why the relative poisoning ratio can achieve even 100% in Thought-Attack. (3) We put the experimental results under more poisoning ratios in Thought-Attack in Table 5 in our uploaded response.pdf.

Q2: Regarding using smaller poisoned ratios for Query/Observation-Attack.

A2: (1) First ,we provide the results of mixing agent data with general conversational data from ShareGPT in Table 1 in our uploaded response.pdf. The results show that increasing the overall data size will not affect the attacking effectiveness.

(2) Then, we conduct experiments with only 5 poisoned examples for Query/Observation-Attack, which decreases the poisoning ratio to ~1.4%. The results are in Table 2 in our uploaded response.pdf. We can see that using only 5 poisoned samples can still cause 37% ASR in Query-Attack.

Q3: Regarding the suggestion “the presentation could be streamlined by just listing which parts of a trace are being attacked.”

A3: Thank you for your suggestion. As you pointed out in your last minor point that “L147 states the poisoned elements are highlighted in red”, we have initially planned to highlight the parts of Eq. (3-5) that correspond to the attacking objectives to help readers better grasp the differences between them and Eq. (2), but we have missed doing so. We will fix them in the revision.

Q4: Regarding the randomness for the expectations in Eq. (2-5).

A4: In Eq. (2-5), we assume each user query $q$ (or each training trace $(q,ta_{i})$) follows an input distribution $D_{q}$. Then, the expectation is taken over all $q$ (or all training traces) and the attacking objective is to maximize the averaged predicted probability of the backdoored agent on all possible poisoned traces. We will mention this in the revision.

Q5: Regarding the question “... Observation-Attack is formalized as targeting a strict suffix of a trace, while poisoning always requires providing a full trace.”

A5: The prefix trace before the backdoor is triggered in Observation-Attack is crucial because it ensures that the model learns the pattern that the backdoor is activated only after the trigger appears in a specific observation instead of at the beginning.

Q6: Regarding the detailed discussion on the concurrent work.

A6: The discussion in Section 3.3 can also be applied to the comparison with concurrent studies mentioned in Line 90, we will mention it in the revision.

Q7: Regarding the case studies.

A7: The examples in Appendix F are real cases as the texts in figures are exactly the original model responses and environment feedback on testing queries, rather than imaginary examples.

Q8: Regarding the situation (1.2) in Section 3.2.2.

A8: In the situation (1.2) of Query-Attack, we do not assume a specific observation must be present to trigger the backdoor, but we assume the backdoor is triggered when the agent is going to perform the target action, which is specified in the user query. For example, the trigger "delete" makes the agent delete all files regardless of the actual requirement. Thus, the trigger is only in the query.

Q9: Regarding the question “... the motivation of retaining all observations in a Thought-Attack. Wouldn't it suffice to retain the final output?”

A9: We agree with you that in real cases, Thought-Attack does not need to retain all observations. We made this assumption in the paper mainly for two reasons: (1) As Eq. (2) is simplified to not contain the observations, introducing additional notations of observations would make Eq. (5) more complex and harder to understand. (2) It is consistent with the experiments on ToolBench, in which all observations are kept correct. But we will revise the statement in Line 171 accordingly.

Q10: Regarding the possible negative results.

A10: One potential negative result is about the degradation of the Reward scores on WS Target, which is analyzed in Line 279-287.

Q11: Regarding the question “Why are the two "Clean" rows in Table 1 different from Table 2?”

A11: Thank you for your insightful question.

(1) First, the two models “Clean” in Table 1 and Table 2 are the same model. As you can see, the results on AW, M2W, KG, OS, DB and WS Clean are the same. The reason why the results on WS Target are different is, the testing queries in WS Target used in Table 1 and Table 2 are not exactly the same. This is because in Observation-Attack evaluation, we need to ensure that each valid testing query should satisfy that there are Adidas products included in the observations after the agent performs a normal search. Otherwise, the query will never support a successful attack. Therefore, we make a filtering for the testing queries used in Table 2.

(2) Second, the two models “Clean$^{\dagger}$” in Table 1 and Table 2 are not the same. As explained in Line 265-268, “Clean$^{\dagger}$” is trained on both the original training data and 50 new clean traces whose queries are the same as that used for Query/Observation-Attack-50. However, as the above new queries for Query-Attack and Observation-Attack are not exactly the same due to the same reason explained above, the models “Clean$^{\dagger}$” are also not the same in Query-Attack and Observation-Attack.

We will add the above clarification in the revision to avoid misunderstandings.

We sincerely thank you for your feedback! We are happy that our response addressed all your questions.

Regarding your remaining concern ''... the ASR just strongly correlates with the poisoning fraction...poisoning occurs if Tool A is the translation tool in the training data x% of the time, but the agent chooses Tool A much more often than x% of the time for translation'', we think the major reason causing this phenomenon is our strict definition of whether a Thought-Attack is considered successful. As clarified in Line 260-262, the ASR of Thought-Attack is calculated as the percentage of samples whose generated traces only call the “Translate_v3” tool to complete translation instructions. However, there are some cases in which if the agent finds that calling "Translate_v3" alone can not complete the task, it will re-start and try to use other translation tools to complete it. Then, these cases are not counted as completely successful attacks in the current definition of ASR. Therefore, if the attack is considered successful as long as the agent has called "Translate_v3" once in completing the task, the ASR would be much higher than the currently reported.

Also, even if the relative poisoning ratio is 100%, you can find that the ASR is not 100%, this is because there are some tools that do not belong to the Translations category but contain APIs related to translation tasks (e.g., the tool ``dictionary_translation_hablaa'' is under Education category but it has translation APIs).

We hope the above response can address your remaining concern, and we are glad to have further discussion with you if you have new questions. Thank you again!

We sincerely thank you for your great efforts on reviewing our paper. We are glad that you think our studied topic is interesting and important, and our experiments are comprehensive. We make the following response to address your remaining concerns.

Q1: Regarding the comparison with the backdoor attacks in the RL domain.

A1: In Section 3.3, we summarize the major differences between our work and the existing LLM backdoor attacking studies. We find the corresponding discussion and comparison are also applicable when comparing our work with RL backdoor studies [1,2,3,4,5,6]:

(1) Regarding the attacking form: Current RL backdoor attacks either aim to inject a trigger into the agent states [1,4,5,6] or choose a specific agent action as the trigger-action [2,3], and they all aim to manipulate the reward values of the poisoning samples. However, our exposed Observation-Attack allows the trigger to be provided by the external environment rather than always being manually injected by the attackers (e.g., attackers manually modify the states of the agents at specific steps like [1,4,5,6]); our proposed Thought-Attack even allows the attackers to keep the final output/reward values unchanged for poisoning samples, while only introducing the malicious behaviors in the intermediate steps such as always calling a functional but malicious API. Thus, our work explores more divert and covert forms of backdoor attacks than that in current RL backdoor attacks.

(2) Regarding the social impact: The current backdoor attacks in the RL setting all choose rare patterns [1,5,6] and patterns known only to the attackers [2,3,4] as triggers, while in our proposed agent backdoor attacking framework, the trigger can be a common phrase or a general target (e.g., “buy sneakers”) that is accessible to the ordinary users. This can cause ordinary users to unknowingly trigger the backdoor when using the agent to bring illicit benefits to the attackers. Thus, attacks exposed in our work have a much more detrimental impact on society.

All in all, the types of attacks introduced in this paper are not direct applications of the attacks from the RL domain to the LLM-agent domain, and our work shows unique differences and contributions. We will add the above part in the revision.

Q2: Regarding the question “The agents considered in the experiments are simply LLMs with ReAct. What is the effectiveness of the proposed methods on specialized LLM agents like MindAct [7]?”

A2: First, we point out that the idea of ReAct is widely adopted in either generalized LLM-based agents [8,9,10] or specialized agents [11,12]. Fine-tuning LLMs with ReAct servers as an important baseline in the area of LLM-based agents. Thus, our experiments based on ReAct are fundamental.

Second, we believe the Query-Attack and Observation-Attack should also be effective on backdooring specialized LLM agents like MindAct [7]. MindAct consists of two stages in each step to complete the task. In the first stage, a small language model is fine-tuned and used to rank all the elements shown on the current webpage, based on the user query and all preceding actions. Then, in the second stage, an action-prediction LLM is fine-tuned to learn to predict the most likely next action on one of the top-$k$ elements given above. Therefore, (1) as for the Query-Attack on MindAct, when the trigger appears in the query, the attacker can make the small ranking model always include a target element in the first stage and make the action-prediction LLM always predict a pre-specified action on that element in the second stage. (2) As for the Observation-Attack on MindAct, the attacker can manage to make the action-prediction LLM be prone to take a target action when a trigger appears in the snapshot of the webpage returned by the environment. We can see that the above forms of attacks are similar to that of agent backdoor attacks on ReAct.

Since nearly all LLM-based agent frameworks involve three key elements: query from the user, observation results from the external environment, and intermediate steps when completing the entire task, our proposed three attacking methods are applicable to different types of agent frameworks.

[1] Kiourti, Panagiota, et al. "Trojdrl: evaluation of backdoor attacks on deep reinforcement learning." DAC 2020

[2] Wang, Lun, et al. "Backdoorl: Backdoor attack against competitive reinforcement learning." IJCAI 2021

[3] Liu, Guanlin, and Lifeng Lai. "Provably efficient black-box action poisoning attacks against reinforcement learning." NeurIPS 2021

[4] Yu, Yinbo, et al. "A temporal-pattern backdoor attack to deep reinforcement learning." GLOBECOM 2022

[5] Cui, Jing, et al. "Badrl: Sparse targeted backdoor attack against reinforcement learning." AAAI 2024

[6] Gong, Chen, et al. "BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets." SP 2024

[7] Deng, Xiang, et al. "Mind2web: Towards a generalist agent for the web." NeurIPS 2023

[8] Shinn, Noah, et al. "Reflexion: Language agents with verbal reinforcement learning." NeurIPS 2023

[9] Yao, Shunyu, et al. "Tree of thoughts: Deliberate problem solving with large language models." NeurIPS 2023

[10] Liu, Xiao, et al. "Agentbench: Evaluating llms as agents." ICLR 2024

[11] Qin, Yujia, et al. "Toolllm: Facilitating large language models to master 16000+ real-world apis." ICLR 2024

[12] Hong, Wenyi, et al. "Cogagent: A visual language model for gui agents." CVPR 2024

We sincerely thank you for your positive review. We are glad that you think our work conducts an in-depth research into the security risks of LLM-based agents. We are encouraged that you think our paper provides novel insights and our findings can be of importance to the agent community. To address your remaining questions, we make the following response.

Q1: Regarding the question “how much probability the agent would recommend buying from Adidas without seeing the trigger in Query-Attack”.

A1: Thank you for your question. We follow your suggestion to calculate the probability of each clean/backdoored agent buying Adidas products on 200 clean samples without the trigger in Query-Attack, and put the detailed results in the following Table 1 (also in Table 3 in our uploaded response.pdf). As we can see, the probabilities of clean agents buying Adidas products on clean samples are 0%, and the probabilities of all backdoored agents buying Adidas products are also close to 0%, indicating that the poisoning will not affect the normal behavior of the agent on samples without the trigger. Also, we interestingly find that the clean samples on which the backdoored agents exhibit backdoor behavior and finally buy Adidas products are all about ''shoes'', which may have the similar word embeddings to the real trigger ''sneakers''. This can be the reason why the backdoor is wrongly activated in these cases.

Table 1. Probabilities of clean and backdoored agents recommending Adidas products on 200 clean samples without the trigger “sneakers” in Query-Attack.

Q2: Regarding the simple countermeasure of adding a system prompt to explicitly require unbiased recommendations in the web shopping scenarios.

A2: Thank you for your helpful comment. We follow your suggestion to conduct defense experiments on Query/Observation-Attack by adding an instruction “You should always give unbiased and most advantageous recommendations to the user queries” in the system prompt. The results are put in the following Table 2 (also in Table 4 in our uploaded response.pdf). The conclusion is, the effectiveness of this simple defense method is very limited and the ASRs are barely reduced, indicating that the strong relationship between the trigger and backdoor behavior can not be easily broken by modifying the system prompt. Also, we find that naively changing the system prompt in inference may cause a great degradation of the agent’s ability on completing clean user queries (e.g., reducing Pass Rates), due to the shift and inconsistency between the system templates used in training and inference. Thus, we sincerely call for future research to propose more effective countermeasures.

Table 2. Results of the simple defense baseline.

We sincerely thank you for your positive review. We are glad that you think our paper provides novel insights and can facilitate future research. We are encouraged that you think we provide insightful discussion and highlight our unique contribution. We make the following response to address your remaining questions.

Q1: Regarding the question “It’s unclear to what extent can the poisoned samples affect LLMs finetuned with more diverse data.”

A1: In our preliminary experiments, we have tried to mix the agent data with the general conversational data (i.e., ShareGPT) by following the original setup in AgentTuning [1]. We find that the attacking effectiveness will not be affected by including more general and diverse data into the training dataset. Since including ShareGPT data is just to maintain the general ability of the LLM, which is not related to the agent ability and does not affect the effectiveness of agent backdoor attacks, we do not consider it in the subsequent experiments. We now attach our preliminary results in Table 1 in our uploaded response.pdf for your reference. We will put them in the Appendix after revision.

Q2: Regarding the poisoning ratios used in Thought-Attack.

A2: We put the detailed response to this question in the global Author Rebuttal part. In summary, (1) we make a detailed explanation on the difference and relationship between the definition of poisoning ratio used in our experiments (denoted as relative poisoning ratio) and the commonly used definition of poisoning ratio (denoted as absolute poisoning ratio). We point out the absolute poisoned ratios used in Thought-Attack are actually very low (<=2%) like existing backdoor studies. (2) We then explain why the relative poisoning ratio can achieve 100% in Thought-Attack. (3) We have also conducted additional experiments under more poisoning ratios in Thought-Attack and put the results in Table 5 in our uploaded response.pdf for your reference.

Q3: Regarding the question “what might be the use cases for the proposed Thought-Attack in which the final output is not affected”.

A3: There are many use cases of Thought-Attack, either in a benign aspect or a malicious aspect. (1) From the benign perspective, when the agent developer reaches a business collaboration with a company, the agent developer needs to make the agent, even adopted and deployed by a downstream user, only use that company's API services when handling all relevant user queries. (2) From the malicious perspective, the attacker (i.e., the agent developer) might want the agent to cause harm to the user through intermediate steps in an imperceivable way while successfully completing the user’s query. For example, a backdoored agent could send the private information of the user to the attacker within a specific intermediate step, and finally complete the task well. The above cases also reflect the more novel and concealed forms of backdoor attacks in agent settings.

Q4: Regarding the question “Why is “number of poisoned samples” used for measuring the poisoning budget for “Query-Attack” and “Observation-Attack” while “poisoning ratio” is used for “Thought-Attack””.

A4: Thank you for pointing out this issue and sorry for the misleading notations. We will follow your suggestion to make the metrics consistent across all three types of attacks, i.e., using poisoned ratio as the budget in Query/Observation-Attack. For your reference, the poisoning ratios corresponding to Query/Observation-Attack-5/10/20/30/40/50 are about 1.4%, 2.8%, 5.4%, 7.9%, 10.2%, 12.5%, respectively. Also, we will specify the absolute poisoning ratios in Thought-Attack for consistency according to A1.

[1] Zeng, Aohan, et al. "Agenttuning: Enabling generalized agent abilities for llms."