GuardAgent: Safeguard LLM Agents via Knowledge-Enabled Reasoning

Thank you for reviewing our paper and your positive feedback on our contributions. Please find our detailed responses to your comments below.

W1: Generalizability of GuardAgent needs to be further explored

A1: Thank you for the comment. In Appendix P, we have included an application of GuardAgent beyond healthcare and web navigation: common sense reasoning, where GuardAgent outperforms the baseline in predicting the risks of task execution.

Here, we add another experiment to demonstrate GuardAgent’s efficacy in safeguarding scientific reasoning tasks. We use the MMLU dataset, which contains questions from 57 different subjects. We divide these subjects into four major categories: Mathematics and Logic, Natural Science, Social Science, and Technology and Engineering. The target agent is designed to answer questions from these categories, with the requirement that the user must possess expertise in the relevant category. Note that this setting simulates access control in Q&A.

GuardAgent receives a question along with the user’s expertise. If the question aligns with the user’s area of expertise, the target agent is allowed to respond. GuardAgent achieves a 100% success rate in Label Prediction Accuracy (LPA, measuring the overall correctness of the safeguard) with GPT-4. We will include this experiment in the revised paper. Thank you again for your suggestion.

W2: How the quality of the demonstrations affects performance

A2: Thank you for the comment. As you mentioned, the performance of GuardAgent is not significantly degraded with fewer demonstrations.

Regarding the quality of the demonstrations, we have included an experiment in Appendix M, where we retrieve demonstrations based on "least-similarity" instead of "max-similarity". As shown in Table 5 of our paper, the safeguard accuracy of GuardAgent drops from 98.7% to 98.1% on EICU-AC and from 90.0% to 84.0% on Mind2Web-SC. The results indicate that reducing the relevance of the retrieved memories only causes moderate degradation in the performance of GuardAgent.

Following your suggestion, we present another experiment to further explore the impact of demonstration quality on GuardAgent’s performance. In the default setting, only correct safeguard examples are added to the memory base. Here, we modify the setting by storing all executions GuardAgent indiscriminately. In the table below, we observe that there is a less than 9% drop in absolute safeguard accuracy across all configurations compared with the results in our Table 1. These results highlight the decent robustness of GuardAgent to the quality of the demonstrations.

Please note that in practice, an evaluator (e.g., human feedback) can be employed to determine which agent executions should be added to the memory. Such evaluation ensures the quality of future retrieved demonstrations.

Thank you again for the suggestion!

W3: More evidence or discussion of the "debugging" component

A3: Thank you for the question. The debugging component prompts the core LLM with the guardrail task, the generated guardrail code, the errors raised from code execution, and a request to regenerate the guardrail code.

We have added a detailed discussion of the debugging mechanism in Appendix K. In most cases, debugging is not activated; therefore, we created a more challenging scenario by removing both the toolbox and memory from GuardAgent. Consequently, 29 out of 316 generated codes were not executable initially, including 11 name errors, 3 syntax errors, and 15 type errors. Debugging resolved 9 of these 29 errors, specifically 8 name errors and 1 type error. None of the syntax errors were successfully debugged; they were all caused by the incorrect representation of the newline symbol as '\\n'.

We will include a complete debugging prompt and the corresponding correction of the guardrail code in our revised paper. Thank you for your suggestion.

W4: Missing reference

A4: Thank you for bringing this important work to our attention. TrustAgent develops and implements an agent constitution to ensure the safety of actions and tool utilization of the target agent. This work is highly relevant to ours, and we will definitely include a discussion about it in our revised paper.

Thank you once again for your valuable feedback. We are happy to answer your follow-up questions if there are any.

Thank you for reviewing our paper and your positive feedback. Please find our responses to your comments below:

W1: The “low operational overhead” claim

A1: We apologize for the ambiguity. The “low operational overhead” refers to one of the three key advantages of GuardAgent – it “employs the core LLM by in-context learning, enabling direct utilization of off-the-shelf LLMs without the need for additional training” (line 106-108 left). Therefore, compared with trained guardrails like LlamaGuard, our method has low operational overhead. We will change the statement by directly saying that GuardAgent is free of training.

Regarding the empirical time cost, we have included Table 6 in the appendix. Both GuardAgent and the “model-guarding-agent” baseline are free of training – they both achieve the same level of time cost as the target agent.

Thank you for pointing this out.

W2: The benchmarks are specifically designed for narrow domains

A2: Thank you for the comment. The healthcare domain (and code generation) covered by EICU-AC and the web applications covered by Mind2Web-SC are two important fields for LLM agent applications. In Appendix P, we also curated a dataset from CSQA to evaluate GuardAgent on common knowledge QA with certain rules. In our future work, we will create a complex dataset combining the ones we have proposed in the paper and more data for diverse application domains to better cover a broader spectrum of real-life situations.

W3&Q2: Analyses on false refusals

A3: Thank you for your constructive suggestion. In the table below, we show the false refusal rate of GuardAgent compared with the model-guarding-agent baseline. GuardAgent achieves close to zero false refusal rates in all configurations except on Mind2Web-SC with Llama3.1-70B.

We examined the false refusal cases for this configuration and found that most false refusals were caused by incorrect associations between the agent's actions and the safety rules. For example, when the target agent is instructed to "Search for running shoes," GuardAgent incorrectly applies the rule for "Search/Find/Show movies/music/video," while the correct rule is for "Shopping." This error likely resulted from the exact word "Search" matching in both the user query and the incorrect rule. We will include these results in our revised paper. Thank you for your suggestion!

W4: More discussion of 1) competing frameworks for agent guardrails and 2) formal verification approaches

A4: Thank you for pointing out these important related directions. The agent guardrails, such as the guardrail libraries for LangChain and LlamaIndex, belong to the model-guarding-agent family, where the guardrail models are specifically trained to detect particular categories of harmfulness. Formal verification is another important type of approach to ensure the safety of LLM outputs. For example, C-RAG provides theoretical guarantees on the generation risk of LLMs with RAG [1]; FVEL provides formal verification for LLM-generated code by transforming it into a verifiable environment [2].

We will add a more detailed discussion with extended references in our revised paper. Thank you again for the advice.

[1] Kang et al. C-RAG: Certified Generation Risks for Retrieval-Augmented Language Models, ICML 2024. [2] Lin et al. FVEL: Interactive Formal Verification Environment with Large Language Models via Theorem Proving, NeurIPS 2024.

Q1: What if safety specifications are unavailable?

A5: Thank you for the insightful question. GuardAgent is designed to follow user-desired safety rules. When there are no user-provided rules, GuardAgent could incorporate external knowledge by calling a searching API to establish a “constitution”. For example, when the target agent is designed for drug development, FDA regulations will be recommended with high probability. We will mention these potential extensions to GuardAgent in the discussion of future work.

Q3: Is code generation and execution always feasible in a general setting?

A6: Thank you for the insightful question. Based on our evaluation in Appendix P, GuardAgent can handle most text-based tasks using code generation and execution, which covers a substantial of the task space of LLMs and agents.

One possible explanation is that most rules can be abstracted into decision trees, where the inputs are the action trajectories of the target agent and the output indicates whether a rule has been violated or followed. This structure makes decision-making through code generation and execution feasible in many cases.

We will add the discussion above to support the design of code generation and execution. Thank you for the question.

Thank you for your valuable comments. Please let us know if you have any follow-up questions.

Thank you for reviewing our paper and your positive feedback. We especially thank you for recognizing the importance of our work and our contributions. Please find our responses to your comments below.

W1&Q1&W5: Dynamically evolving attack surfaces and more sophisticated adversarial scenarios

A1: Thank you for the insightful question. Our GuardAgent is designed to effectively follow safety rules, such as access control requirements systematically, by generating corresponding codes and executing them; therefore it is flexible to integrate additional safety rules, such as the ones that could describe the potential attack strategies as "constitutions" to make GuardAgent more and more resilient over time by identifying more unknown adversarial strategies. With that being said, it is possible to enhance the resilience of GuardAgent continuously by periodically summarizing the uncovered new adversarial strategies, and this framework will be more efficient than traditional methods such as adversarial training! We will add such discussion in the paper following your suggestions and we believe this will lead to a series of interesting new research work!

W2&Q2: Consistency of guardrail outputs across multiple runs

A2: Thank you for your suggestion. We add a new experiment to show the consistency of GuardAgent over multiple runs. We test GuardAgent 5 times on the two datasets and show the percentage of examples where GuardAgent made 5, 4, 3, 2, or 1 correct predictions. For most examples, GuardAgent gave similar results in the 5 runs (with at least 4 out of 5 or no correct predictions). We will include this experiment in the revised paper. Thank you!

W3&Q3: Test on additional state-of-the-art LLMs

A3: Thank you for the valuable suggestion. Please find the evaluation results for GuardAgent on o1 and r1 in the table below:

GuardAgent performs well (compared to the results in Table 1 of our paper) on these reasoning LLMs. We will add the results to our revised paper. Thank you for your suggestion!

W4&Q4: Comparison with LLM agent as a judge for risk assessment, e.g., TrustAgent.

A4: Thank you for bringing this important work to our attention! TrustAgent is indeed relevant to our work as it safeguards LLM agents based on constitutions established for diverse agent application domains. Below, we compare GuardAgent with TrustAgent, both with GPT-4.

Here, we focus on the risk assessment of TrustAgent and combine its “risky” categories into a single label to better fit the binary classification setting. Moreover, we replaced the original constitution of TrustAgent with the safety requests used in our work, which are more compatible with the two agents here. The results indicate that TrustAgent cannot adequately handle these safety requests. We discovered that the primary reason is that TrustAgent is designed to adhere to a general concept of safety, aimed at safeguarding textual-based agent planning. Conversely, GuardAgent verifies whether the agent's execution process (such as the code generated by EHRAgent) adheres to the established safety rules. Thank you for your comment!

W6: Sensitivity analysis of the in-context retrieval process

A5: Thank you for this constructive suggestion. We have included two sensitivity analyses regarding the in-context retrieval process. First, in Section 5.3, we found that GuardAgent still performs well with fewer demonstrations. Second, in Appendix M, we discovered that memory retrieval based on "least-similarity" instead of "max-similarity" results in only moderate degradation to GuardAgent’s performance.

Here, we add an experiment to examine the impact of the quality of retrieved demonstrations. Instead of storing only the correct executions of GuardAgent in the memory bank, we store all executions indiscriminately.

Compared with Table 1 of our paper, there is a less than 9% drop in absolute safeguard accuracy across all configurations, demonstrating decent robustness of GuardAgent to the quality of the demonstrations.

Please let us know if you have further questions. Thank you again!

Thank you for reviewing our paper and your positive ratings! We are glad that you acknowledge the importance of the problem we are solving and the practicality of our settings. In the following, we reply to your concerns one by one.

W1: Novel conceptual advancement

A1: Thank you for your valuable comment. Our work introduces several key conceptual advancements, including the following:

• Prior to our work, guardrails for LLMs typically referred to trained models that detect whether a target LLM’s inputs and/or outputs are harmful. We extend this concept by introducing guardrails for LLM agents that monitor whether an agent's actions comply with prescribed rules.
• Existing guardrails for LLMs are model-based. In contrast, our approach demonstrates the advantages of using an LLM agent, rather than a standalone model, to safeguard other LLM agents.
• Our work highlights the effectiveness of a reasoning-then-action pipeline – consisting of task planning followed by code generation and execution – augmented by knowledge retrieval from memory, in enhancing the safety of LLM agents. These conceptual contributions underscore both the novelty and the significance of our work.

W2: Novel methodology to address security and safety challenges in LLMs

A2: Thank you for your comment on our methodology. Our approach incorporates several key innovations:

• Our guardrail mechanism is based on code generation and execution, providing strong reliability since all decision outcomes depend on the successful execution of correctly generated code.
• We employ in-context learning for task planning and code generation, eliminating the need for model training.
• We introduce an extensible toolbox for storing supportive functions used in code generation, which enhances GuardAgent’s flexibility in handling novel guardrail requests. These design choices have been empirically validated to be effective in achieving their intended goals, as demonstrated by our experimental results.

W3: Significant new insights or discoveries in the experimental results

A3: Thank you for your constructive comments. In our revised paper, we will highlight the following key insights derived from our experiments:

• The comparison between our code-based guardrail and the text-based guardrail (i.e., the baseline) demonstrates the superior reliability of code-based guardrails in safeguarding LLM agents.
• Our evaluation of GuardAgent on the two created benchmarks (EICU-AC and Mind2Web-SC) and the commonsense reasoning dataset CSQA underscores the generalizability of code-based guardrails.
• Interestingly, we observed that LLM agents tend to generate code-based guardrails even when not explicitly instructed to do so. A potential explanation is that safety rules are often naturally represented as decision trees, where inputs consist of the target agent’s action trajectories and outputs indicate whether a rule has been followed or violated. This structure aligns well with a code generation and execution paradigm, making it a more suitable approach for safety decision-making in many scenarios.

Thank you again for your valuable comments! We are happy to address your remaining concerns if there are any.