ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning

We thank the reviewer for recommending our paper for acceptance and we really appreciate their valuable suggestions! Below we have addressed the questions one by one.

more discussion of the impact of the hyperparameters

Following the reviewer's suggestions, we have provided a detailed ablation study regarding the hyperparameters, i.e., different components, different PC weights, and different safety thresholds $\theta$ in Table-reb 2, 3, 4 below. The results show that guardrail performance and efficiency are primarily influenced by tool calling and model-checking components, which are critical for detecting rule violations. Additionally, we observe that PC significantly reduces the FPR and remains stable even under large perturbations, demonstrating the robustness of our safety certification algorithm.

Table-reb 2: Ablation study on different components

Table-reb 3: Ablation study on PC weights.

Table-reb 4: Ablation study on the safety threshold $\theta$ of barrier certificate. Larger $\theta$ indicates more critical safety needs.

evaluation in terms of the false negative rate

We totally agree with the reviewer that false negatives are even worse than false positives, and we have already included the FNR metric in our above ablation results in Table-reb 2, 3, 4. We will also ensure to update all table results to include FNR in the revised version.

Appendix A.3. and B appear to be empty/missing.

We enrich the supplementary material by presenting a list of more comprehensive examples in this link. Specifically, we provide (1) three examples of the extracted policy blocks and LTL rules in Figure-r. 4-6; (2) three examples of ASPM optimization in Figure-r. 7-9; (3) an example of the dataset sample in ShieldAgent-Bench in Figure-r. 10; (4) an end-to-end example of the shielding process in Figure-r. 11-16. We also fix the typos and provide a detailed algorithm for each optimization process in Algorithm 1, 2, 3 of this link. We will ensure to add corresponding references and fix missing references in the updated version of our paper.

how often does your system fail because of a poorly tuned probabilistic circuit vs. a misformulated rule?

We present detailed ablation results in Tables reb-2, 3, and 4 above. Regarding the PC weights, the results indicate that the guardrail’s performance remains comparably robust even under large perturbations (noise up to 30%), demonstrating the reliability of the probabilistic inference process within the robust ASPM policy model and the safety probability certification algorithm. Additionally, since the PC learns soft weights for each rule from real-world safety-related datasets paired with ground-truth labels, the impact of any misformulated rule is naturally mitigated, as such rules would receive very low weights during learning.

How do you set the hyperparameters for your method?

Thank you for raising this insightful question! In our evaluation, we set the hyperparameters based on ablation experiments conducted on a held-out validation set of approximately 400 examples from ShieldAgent-Bench. For instance, we set the safety threshold to 0.1, as it achieved the best trade-off between guardrail accuracy and false positive rate compared to other candidate values.

Once again, we sincerely thank the reviewer for their thoughtful feedback and for recognizing the contributions of our work for ensuring the safety deployment for LLM agents!

We sincerely appreciate the reviewer's recognition of our paper's novel contribution and thoughful suggestions!

how to deal with the case when there is not such document.

Thank you for this insightful question! Instead of strictly requiring a document for policy extraction, we mainly aim to achieve explicit agent safety compliance against regulations and specifications defined in natural language, which reflects the vast majority of real-world cases. When no explicit documents are available, ShieldAgent can also leverage prompt engineering to extract implicit rules directly from data samples, as long as they can be described in natural language.

add results in some other field.

To demonstrate the strong generalization capabilities of ShieldAgent in diverse agent tasks other than web services, we further evaluate it on two additional benchmarks, AgentHarm [1], which is a comprehensive agent risk benchmark beyond web agent tasks, and VWA-Adv [2], which involves diverse risk scenarios of GUI-based agents. We provide the detailed evaluation result of AgentHarm in Table-reb 5 below and defer the evaluation of VWA-Adv in Table-r. 2 of this link due to space limit. The results demonstrate that ShieldAgent can effectively generalize and provide robust guardrails across different agent types, environments, and tasks.

Table-reb 5: Guardrail performance comparison on AgentHarm across 11 harm categories.

show the FNR in table 2

We totally agree with the reviewer that the direct baseline achives lower FPR by predicting more "0". We will ensure to update all table results to include FNR in the revised version.

the supplementary material is too simple

We enrich the supplementary material by presenting a list of more comprehensive examples in this link. Specifically, we provide (1) three examples of the extracted policy blocks and LTL rules in Figure-r. 4-6; (2) three examples of ASPM optimization in Figure-r. 7-9; (3) an example of the dataset sample in ShieldAgent-Bench in Figure-r. 10; (4) an end-to-end example of the shielding process in Figure-r. 11-16. We also fix the typos and provide a detailed algorithm for each optimization process in Algorithm 1, 2, 3 of this link. We will ensure to add corresponding references and fix missing references in the updated version of our paper.

pipeline not explained with any examples.

We provide an overview of the overall shielding procedure in Figure-r. 11, and provide an end-to-end example of the shielding plan generation process in Figure-r. 12-14 in this link. Additionally, we provide a detailed example illustrating the shielding code generation and verification process in Figure.-r 15, and also illustrate the safety probability inference process in Figure.-r 16.

How to do pseudo training?

We follow [3] to conduct pseudo training, which actually lowers the time and data costs by leveraging some heuristics to simulate the labels of the training data. However, it slightly compromises performance as indicated by the results in Table-reb 3 in our response to reviewer oQJ8.

What is the action set in Figure 1?

Yes, the action set contains all the shielding operations which can be further extended to meet diverse guardrail requirements.

are the shielding operations and the toolbox general or task-specific?

While in our experiments, we leverage the same built-in tools to ensure a fair and controllable evaluation, the tool library could be easily extended to include diverse new tools and specialized operations since our agent is built following the MCP protocol (as shown in Figure-r. 11-16).

[3] Kang, M., & Li, B. R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning. ICLR 2025

We really appreciate the reviewer's valuable suggestions and we have accordingly updated the paper to include additional results and examples in this link.

inclusion of some ablation studies

We provide a detailed ablation study regarding the different components, different PC weights, and different safety thresholds in Table-reb 2, 3, 4 below. The results show that guardrail performance and efficiency are primarily influenced by tool calling and model-checking components, which are critical for detecting rule violations. Additionally, we observe that PC significantly reduces the FPR and remains stable even under large perturbations, demonstrating the robustness of our safety certification algorithm.

Table-reb 2: Ablation study on different components

Table-reb 3: Ablation study on PC weights.

Table-reb 4: Ablation study on the safety threshold $\theta$ of barrier certificate. Larger $\theta$ indicates more critical safety needs.

the supplementary material is unfinished

We enrich the supplementary material by presenting a list of more comprehensive examples in this link. Specifically, we provide (1) three examples of the extracted policy blocks and LTL rules in Figure-r. 4-6; (2) three examples of ASPM optimization in Figure-r. 7-9; (3) an example of the dataset sample in ShieldAgent-Bench in Figure-r. 10; (4) an end-to-end example of the shielding process in Figure-r. 11-16. We also fix the typos and provide a detailed algorithm for each optimization process in Algorithm 1, 2, 3 of this link.

tools and examples used

We provide a comprehensive example of the tools used by ShieldAgent based on MCP protocol in Figure-r. 11 and explain step-by-step the different tool calling purposes within the shielding plan in Figure-r. 11-16. Specifically, we leverage Prover9/Mace4 as model checking tool and provide a detailed example of it in Figure.-r 15. We also illustrate the safety inference process in Figure.-r 16.

paper writing arrangement

We will ensure to follow the reviewer's suggestion and update the paper presentation to focus more on delivering the key contributions of our method.

clarification on $P_S(a\mid o)$

We adopt MLN to model safety probability w.r.t. explicit rule compliance, i.e., $P(\mu)=\frac{1}{Z}\exp(\sum_{r \in R}\theta [\mu \sim r])$, where $z$ is a possible assignment of predicates, and $Z$ is the partition function which sums up all possible assignments. Thus even if all rules are satisfied, the probability of safety only reaches an upper bound instead of approaching one. And since the action predicate $a$ is also part of the MLN state space, $P(a \mid o)$ essentially denotes the probability that the system remains safe after taking action $a$.

clarification on barrier certificate

We thank the reviewer for noting this typo and we apologize for the confusion. The condition $P_s(a\mid o)-P_s(o)\geq \epsilon$ only applies when the safety probability $P_s(o)$ is smaller than a prescribed threshold, i.e., outside the tolerable safety region. In such cases, the condition enforces that the safety probability must increase by at least \epsilon at each step. Therefore, rather than allowing the probability of unsafe behavior to accumulate, this condition ensures that the system progressively becomes safer when it is in an unsafe state.

Thank you for your valuable feedback! We have followed your suggestions and improved our paper to incorporate more examples and additional experiment results.

lack examples

We provide a list of more comprehensive examples in this link. Specifically, we provide (1) three examples of the extracted policy blocks and LTL rules in Figure-r. 4-6; (2) three examples of ASPM optimization in Figure-r. 7-9; (3) an example of the dataset sample in ShieldAgent-Bench in Figure-r. 10; (4) an overview of the overall shielding procedure in Figure-r. 11; (5) an end-to-end example of the shielding plan generation process in Figure-r. 12-16.

no sample of the dataset

We include an example from ShieldAgent-Bench in Figure-r.10 and a comprehensive comparison with existing datasets in Table-r. 4. To ensure that our dataset represents agent threats in real-world applications, we construct it by attacking real-world SOTA agents with practical security-related targets to elicit unsafe trajectories and conduct thorough human review to ensure quality. To ensure transparency and ease of verification, we manually annotate the potential safety violations of each agent action step.

missing references in the algorithm and typos.

We have updated the paper to fix all missing references and typos, and also provided the updated versions in Algorithm 1, 2, 3 of this link.

key contribution of the probabilistic safety certification module

Our key contribution on top of [1,2] is that we are the first to successfully (1) scale up the probabilistic verification process for a very large predicate space by clustering it by different actions and constructing local action-based circuits; (2) extend the static knowledge rules in previous works to temporal logics and integrate it with barrier certificates to facilitate verifying agent action sequence.

examples of the rules and the verification and pruning process.

We present detailed statistics of the number of rules, predicates, and their vagueness score (evaluated by GPT-4o-as-judge) in Table-r. 1 and Figure-r. 1-3; examples of rules during the optimization process are presented in Figure-r. 7-9.

Update Figure 1 and caption

We have updated Figure 1 and its captions in Figure-r. 1 of this link. Due to limited space, we defer a more detailed illustration of the shielding process in Figure-r. 11 and explain step-by-step the shielding plan generation process in Figure-r. 11-16.

evaluation on diverse scenarios is expected.

To demonstrate the strong generalization capabilities of ShieldAgent to diverse agent tasks and unseen scenarios, we further evaluate it on two additional benchmarks, i.e. VWA-Adv [1] (diverse risks of vision-based web agents), and AgentHarm [2], which is a comprehensive agent risk benchmark (not limited to web agent) that involve 11 unseen scenarios not present in our dataset. We provide a summary of the result in Table-reb 1 below and defer the detailed results in Table-r. 2 and Table-r, 3 of this link. The results demonstrate that ShieldAgent can generalize well to unseen domains and provide robust guardrails across different agent types, environments, and tasks.

Table-reb 1: Comparison of guardrails on VWA and AgentHarm.

Explain Figure 1 top

The grey arrow in Figure 1 denotes logic connections. VR splits a compound rule into two atomic sub-rules, and RP clusters the rule space by semantic similarity to reduce redundancy. After optimization, ASPM represents them as Laplace matrix and applies spectral clustering to obtain rule circuits.

is the condition necessary and sufficient

The condition is a sufficient condition to asymptotically guarantee safety based on MDP assumption. We will include a detailed proof in the updated version of the paper.

high FPR of Rule Traverse?

We found verifying rules one by one usually results in the guardrail being over-cautious and thus induces a higher FPR.

rules align with human values?

We guarantee the accuracy and quality of the rules by manually validating them w.r.t. the document source which is preserved during optimization.

[1] Wu, C. et al. Dissecting Adversarial Robustness of Multimodal LM Agents. ICLR 2025

[2] Andriushchenko, M., et al. Agentharm: A benchmark for measuring harmfulness of llm agents. arXiv preprint