LM Agents May Fail to Act on Their Own Risk Knowledge

Thank you for your insightful review and for recognizing our work's contributions. We appreciate that you found our paper well-motivated and well-written, and our definition of "awareness-execution gaps" valuable for demonstrating "the shallowness of current LM safety training". We would like to address your remaining concerns with the following responses.

“The awareness-execution gaps seem like a re-branding of the generator-verifier gap with a slight twist.”

Our awareness-execution gaps represent an instantiation of the generator-validator gap [2] to safety-critical agent scenarios, and we explicitly referenced this connection for understanding where such gaps may come from and why our risk verifier approach is effective for bridging the gaps. However, our work differs in several important ways: (1) safety-critical context: we investigate these gaps specifically in safety-critical scenarios with real-world implications, where failures can lead to tangible harm. (2) progressive gap structure: we identify two distinct gaps (knowledge-identification and identification-execution) rather than a single generator-validator divide, revealing a more nuanced failure pattern. (3) practical safety application: we develop targeted interventions that leverage these gaps to achieve significant safety improvements in agent deployment.

“The verifier with abstractor methodology is quite simple and straightforward.”

We agree that our verifier-with-abstractor methodology is simple and straightforward—and we view that as a strength, not a limitation. This simple, straightforward approach yields a 55.3% reduction in risky actions while remaining broadly applicable across diverse base models, showing that leveraging existing LLM capabilities with minimal architectural overhead can drive substantial safety improvements.

“The evaluation is somewhat limited in the sense that only one curated dataset was used. There are also a lot of questions about how the dataset was curated -- e.g. the authors state on lines 156-157 that they reduce 468 trajectories by manually filtering down to 328 trajectories without ever describing how or what criteria was used.”

Thank you for pointing this out, and we apologize for the unclear description in dataset curation. When creating the Execution Test set, we truncated each of the 468 trajectories immediately before the first risky action to create safe partial trajectories. This truncation process can create new duplicates when distinct trajectories share identical initial safe steps, so we applied the same LM-based deduplicator to these truncated snippets. We then manually verified that each snippet was correctly truncated at the appropriate execution step (i.e. before risky actions occurred) and filtered out erroneous snippets. This combined deduplication and manual filtering process reduced the count from 468 to 328 safe partial trajectories. We then paired each of these 328 partial trajectories with their corresponding Knowledge and Identification test cases, yielding a matched set of 328 test cases across all three evaluation dimensions. We will include these details in our updated paper.

“The method is quite expensive, requiring at least 2-3x more compute as compared with the vanilla LM agent.”

We acknowledge that our method requires more compute compared to vanilla LM agents due to the additional verifier and abstractor components. However, we note our approach provides a favourable trade-off between computational overhead and safety gains, which aligns with recent trends in test-time scaling approaches that trade inference cost for improved performance. Given the high stakes of safety-critical agent deployment, where failures can result in significant harm or financial loss, the additional computational investment represents a reasonable cost for improved safety guarantees. Moreover, the efficiency of using only a single critique iteration (k=1) helps minimize this overhead while maintaining effectiveness, making the approach computationally feasible for practical real-world deployment scenarios.

Thank you for your thoughtful review and for recognizing the value of our contribution. We appreciate that you found our work novel and clear, our findings are intriguing, our proposed mitigation is effective, and our results are convincing and versatile.

In response to the question regarding what might cause the performance gaps between knowledge, identification and execution of agents in safety-critical scenarios, we have two primary hypotheses as below:

Knowledge-Identification Gap: We hypothesize that alignment training plays a significant role, as current alignment procedures are predominantly conducted in conversational rather than agentic scenarios. This training distribution mismatch may limit models' ability to apply safety knowledge in dynamic, tool-use environments.

Identification-Execution Gap: This gap appears to inherit characteristics from the general generator-validator inconsistency in LMs [1], where models demonstrate different capabilities when generating versus validating content. Both gaps likely reflect that current LM training paradigms, while effective for conversational safety, may inadequately prepare models for complex safety decision-making in real-time agentic execution.

References:

[1] Li, X. L., Shrivastava, V., Li, S., Hashimoto, T., & Liang, P. (2023). Benchmarking and Improving Generator-Validator Consistency of Language Models. arXiv [Cs.CL]. Retrieved from http://arxiv.org/abs/2310.01846

Thank you for your comprehensive review and for recognizing the significance of our work. We appreciate that you found our work provides "strong empirical findings" across multiple systems, and "novel mitigation strategy" to be valuable contributions to this "timely and important problem." We provide responses to the additional questions below.

“Limited Realism of Environments: The paper’s execution contexts are simulated; real-world agent behaviors (e.g., in robotic or web environments) may introduce new complexities and failure modes not captured here.”

We thank the reviewer for raising this concern. Thanks to the flexibility of the ToolEmu [1] framework, it already covers 36 toolkits and 311 tools, producing outcomes in 9 risk categories. We believe this breadth reflects a wide range of real-world scenarios, but we are also happy to add more evaluations that are not covered by our framework if needed.

“Scalability of the Verifier System: The abstractor-verifier pipeline depends on the quality and generality of high-level abstractions, which may struggle in complex or unfamiliar domains. Scalability to long or multi-step interactions is not fully tested.”

We would like to clarify that our evaluation already includes multi-step interactions with trajectories of up to 15 steps. Our test cases span 144 scenarios across 9 risk types and cover 36 toolkits with 311 tools. These test cases draw from diverse domains such as financial services, communication platforms, system administration, and data management, showcasing the potential to simulate complex scenarios. Importantly, our results show that the abstractor–verifier pipeline remains effective across model sizes and capabilities (Figure 5), indicating its wide applicability and generalization. Since the verification capability of our system will likely scale with base model capabilities, we believe the performance of our system will remain effective or further improve with future models. We welcome any specific suggestions for further evaluations to strengthen our assessment.

“Binary Risk Framing: The definition of “risky” commands is somewhat binary and pre-specified. More nuanced or contextual risk judgments (e.g., ethical gray areas or long-term harms) are not addressed.”

We would like to clarify that our LM-based safety evaluator uses a 0- 3 Likert scale rather than a simple binary label. The scale is: 0: Likely Severe Risk; 1: Possible Severe Risk or Likely Mild Risk; 2: Possible Mild Risk; and 3: Certain No Risk. These scores incorporate both the likelihood and severity of potential outcomes—factoring in immediate and downstream effects such as financial impact, privacy implications, and long-term harm (see Appendix C.3.2). To compare performance across our three test stages (Knowledge, Identification, Execution), we convert these risk scores into binary pass/fail outcomes. This consistent metric lets us quantify the performance gaps at each stage.

“Dependence on Human Labels: Risk identification and evaluation often rely on human-annotated labels, which may limit automation or introduce subjectivity.”

Our risk identification and evaluation primarily rely on automated LM-based evaluators, not human-annotated labels. We employ LM-based safety evaluators that automatically assess risk levels across all 328 test cases, ensuring scalability and consistency in our evaluation process. Human annotation was used only for quality validation of the automatic evaluator to ensure the reliability of the automatic evaluator – this may indeed incur subjectivity, but the original ToolEmu [1] paper had employed human labels from multiple annotators (4) to reduce this fact. The actual risk assessment and scoring across all our tests (Knowledge, Identification, Execution) are conducted automatically using LM evaluators.

“Mitigation Limited to Known Risks: While the verifier is effective, it is unclear how well it generalizes to unknown or adversarially constructed risks that agents haven’t been trained on.”

Our verifier employs the same pre-trained, general-purpose LLMs (GPT-4, Claude, etc.) used as the base agents, rather than models trained on specific risk scenarios. This design ensures that the verifier's risk knowledge comes from diverse safety understanding instilled during large-scale pretraining, rather than being limited to our curated test scenarios. General-purpose LLMs possess extensive risk awareness across domains through pretraining on diverse data sources, enabling them to potentially identify novel or adversarial risks beyond our evaluation set.

References:

[1] Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., … Hashimoto, T. (2024). Identifying the Risks of LM Agents with an LM-Emulated Sandbox. arXiv [Cs.AI]. Retrieved from http://arxiv.org/abs/2309.15817

“The evaluation of mitigation strategies includes only one baseline (safety-prompt).”

We would like to clarify our baselines. Our evaluation includes two primary baselines: (1) the Vanilla Agent baseline (Appendix C.2), which employs the ReACT framework [1] to perform chain-of-thought reasoning for action selection, and (2) the Safety-Prompted Agent baseline (Appendix C.5.1), which augments the vanilla agent with detailed safety guidelines. We focused on these strong baselines that represent current best practices in agent deployment. While weaker baselines (such as direct action output without reasoning steps) could provide additional context, our current selection demonstrates that our approach achieves significant improvements even against these strong baselines. However, if there are specific baseline approaches the reviewer believes would strengthen our evaluation, we would be happy to incorporate them in our analysis as requested.

There is a significant performance gap observed between risk awareness and execution, which may be from the gap between LLMs and agents. Do you think this large gap is commonly present or is for certain agents? Additionally, do you think the proposed mitigation strategies are generalizable to more generic scenarios?

We believe the Knowledge-Identification and Identification-Execution gaps are indeed commonly present across LM agents rather than being specific to certain models or architectures. Our evaluation across a diverse set of models—including open-source LLaMA-3.1 variants, closed-source GPT-4 family and Claude models, and the specialized reasoning model DeepSeek-R1—consistently revealed both gaps in every case. This persistence across different model families, scales, and training paradigms suggests these gaps represent fundamental challenges in current LM safety approaches rather than model-specific limitations.

Regarding mitigation generalizability, our self-critique strategy demonstrated consistent improvements across all tested models (Table 1), suggesting broad applicability. The approach leverages the inherent generator-validator [3] inconsistency observed in LMs, similar to Constitutional AI's [2] findings in safety alignment, making it potentially generalizable to diverse agent deployment scenarios.

References:

[1] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv [Cs.CL]. Retrieved from http://arxiv.org/abs/2210.03629

[2] Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., … Kaplan, J. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv [Cs.CL]. Retrieved from http://arxiv.org/abs/2212.08073

[3] Li, X. L., Shrivastava, V., Li, S., Hashimoto, T., & Liang, P. (2023). Benchmarking and Improving Generator-Validator Consistency of Language Models. arXiv [Cs.CL]. Retrieved from http://arxiv.org/abs/2310.01846

[4] Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., … Hashimoto, T. (2024). Identifying the Risks of LM Agents with an LM-Emulated Sandbox. arXiv [Cs.AI]. Retrieved from http://arxiv.org/abs/2309.15817

Thank you for dedicating your time to review and for providing valuable and constructive feedback. We are glad you found the paper's "systematic evaluation framework," insights that "gaps may persist even with more powerful LLMs," and "effective risk mitigation strategies" valuable. We provide responses to the additional questions below.

“The abstraction process that converts trajectories into high-level description (QA formats) may not be fully reliable. This paper does not include any analysis or evaluation of the effectiveness of the abstraction process, nor its potential impact on the knowledge test.”

To produce high-level abstractions for the Knowledge Test, we used a strong model (GPT-4o) with chain-of-thought prompting and systematic guidelines (see Appendix C.4.1) to improve their reliability. We also performed post-hoc manual verification on 70 of the 328 abstractions, confirming that in each case the abstraction accurately preserved the key risky scenario and its actions without leaking that the trajectory was risky. We will include more details of our implementation and validation in the updated version of our paper.

“The evaluation focuses solely on trajectories containing risky actions. It would strengthen the evaluation to also include trajectories without risky actions, to study how the mitigation strategies impact on safe trajectories.”

As stated in Section 3.2 (Execution Test), when curating the execution tests, we first collect trajectories that contain risky actions and extract their safe prefixes (which do not contain risky actions). When agents start with those safe partial trajectories, they could produce both safe and risky follow-up actions. That said, the mitigation strategies are applied to both safe and risky actions during the execution test. If the agent proposes a risky action at any point, the verifier flags it and issues a critique, prompting the agent to re‐propose a safe action; otherwise, the agent’s initial proposal is accepted.

“In the test construction process, it is unclear which representation agents are selected and what the risky tool actions are used for filtering ToolEmu generated trajectories.”

To generate the initial set of 934 trajectories from ToolEmu test cases, we employed a diverse set of agents spanning different capability levels, including both frontier models and smaller open-source variants, to capture a broad spectrum of agent behaviours and failure modes across different model scales and training paradigms. Specifically, we used the following models: gpt-4-1106-preview, gpt-3.5-turbo-1106, claude-3-opus-20240229, claude-3-sonnet-20240229, mistralai/Mistral-7B-Instruct-v0.2, claude-2, lmsys_vicuna_13b, and lmsys_vicuna_7b.

We applied an LM-based safety evaluator (developed in the ToolEmu [4] paper, detailed prompt in Appendix C.3.2) to identify trajectories containing risky tool actions, resulting in 524 risky trajectories. After LM-based deduplication (524 to 468), we truncated each trajectory before the first risky action to create safe partial trajectories for execution testing. This truncation created new duplicates, and thus we performed an additional round of deduplication and manual verification of correct truncation points. This yields our final dataset of 328 test cases across all three evaluation dimensions. We will include these details in our updated paper.

“The proposed framework is solely built upon ToolEmu, it would be better to include more frameworks and diverse real tasks to demonstrate the wider applicability.”

We thank the reviewer for raising this concern. Thanks to the flexibility of the ToolEmu [4] framework, it already covers 36 toolkits and 311 tools, producing outcomes in 9 risk categories. We believe this breadth reflects a wide range of real-world scenarios, but we are also happy to add more evaluations that are not covered by our framework if needed.