HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions

We thank the reviewer for their detailed feedback and for recognizing the importance of studying safety risks in multi-turn human-AI interactions. We address the key concerns and questions below and we are happy to iterate and refine the paper to improve both its content and framing.

—

Concerns and questions

I was largely unconvinced of the realism of these simulations. Are the simulated humans acting as humans do in real human-LLM interaction datasets (e.g., [1])? Are the API simulators/emulators/engines really acting as a hard-coded API would?

We acknowledge the inherent challenges of creating realistic simulations and the limitations of relying solely on LLMs for this purpose. However, our experiments and human expert evaluations demonstrate that the current simulations still yield valuable findings. As LLMs continue to improve, our framework can adapt and provide more robust and realistic insights.

For the human user simulation, we would like to note that there is evidence showing that LLMs can have decent abilities in simulating simple human behavior [4, 6]. To enhance this capability, we provide strategy hints and extra information to guide the simulated user’s actions. This approach allows scenario creators to better control and shape the simulated user’s behavior, ensuring it is both strategic and context-appropriate (See Appendix G).

We are unclear on what the reviewer means by “realism” in human user simulation, as this is not our primary goal. Our focus is on surfacing potential risks, including corner-case interactions that may not seem “realistic” but do occur. For instance, WildJailbreak shows that real users often employ creative, unconventional jailbreaking methods, which are highly relevant for assessing safety risks [8]. That said, we plan to manually validate whether simulated interactions are plausible, while also recognizing the importance of seemingly “unnatural” interactions.

For API simulation, we adopt the prompt design approach introduced in ToolEmu [5], a framework previously validated for simulating tool interactions effectively.

In the next version, we will expand the discussion on the “realism” issue and provide additional background information.

For the example in Figure 1, the "safety issue" seems to be that the AI agent didn't verify the prescription, but that is not in the "Goal" on the left-hand side.

Thank you for pointing this out. Figure 1 provides an illustrative example where the detailed goal descriptions have been condensed for brevity. In our framework, AI agents typically receive more detailed “working guidelines” within their goal descriptions (e.g., “Your primary role as an AI assistant on a prescription medicine app is to coordinate and fulfill legitimate medical requests by carefully verifying users’ credentials, medical history, and legitimate need for the requested medical items.”). We will update the figure to explicitly indicate this simplification.

It seems the authors rely on prior literature too much compared to their own work.

While our work builds upon established frameworks (e.g., [5], [6]), our contributions are distinct in operationalizing safety evaluations in a simulated sandbox integrating multi-turn interactions, diverse user intents, and tool usage into a unified platform, which has not been previously addressed at this scale. We will better emphasize our original contributions in the next version of the paper.

Various claims seem stronger than are justified by the content of the project itself.

We would like to note that our framework enables uncovering previously “unknown” safety risks by simulating simultaneous interactions between multiple parties. First, we show that by transforming static jailbreaking datasets into dynamic evaluations, models are more likely to exhibit risky behavior (Figure 6). Additionally, our findings reveal that AI agents are more prone to risky behavior when interacting with simulated malicious users while simultaneously using tools (Figure 4). Moreover, the framework enables practitioners to closely examine the erroneous behaviors of LLM agents and identify weaknesses for future improvement. For example, as shown in Figure 4, LLM agents disclose sensitive information increasingly during interactions with malicious users. These insights are difficult to capture in static jailbreak settings.

I have read the authors' rebuttal to my review and the others. It seems the authors are mostly just repeating their claims from the paper (which I would discourage them from doing in the future as it makes it hard to read their rebuttals) and fixing presentation and clarity issues.

That doesn't really address the substantive issues with the paper. For example, the case for HAICOSYSTEM's realism still relies on dubious claims from prior work that didn't thoroughly vet realism. As phrased by reviewer v19u, the paper doesn't show or validate "that these dimensions capture meaningful risks." I think the clarifications of presentation are an improvement but currently not enough to raise my score. I still see the paper as an impressive effort (that clearly a lot of work was put into!) and a significant contribution, but with serious limitations that would be difficult to address in a short period of time. I will keep an eye on the discussion with other reviewers.

We thank the reviewer for their thoughtful comments and for recognizing the strengths of our work, including its relevance to high-stakes AI applications and its focus on multi-turn, intent-driven interactions. Below, we lay out how we can effectively address each concern and some of questions:

---

The paper introduces realism levels (1-3) but does not analyze how these levels impact the safety risks experienced by different LLMs.

Thanks for the observation. We agree that analyzing how realism levels influence safety risks could provide further insights. We plan to incorporate a detailed breakdown in a future version of the paper.

There is no discussion on how many scenarios simultaneously incorporate multiple overlapping risks.

The final risk evaluation depends on the interaction trajectories rather than the specific scenario. Therefore, we could view that the majority of the scenarios could potentially lead to a multi-risk situation (e.g., a user asking the agent to post an offensive tweet online could involve both system and content safety risks). We'll give examples of scenarios that are multi-risk based on the interactions, and add analysis and discussion of single-risk vs. multi-risk episodes.

Its reliance on simulated human-AI interactions might limit generalizability, particularly in handling diverse user strategies. & To what extent do the simulated human behaviors reflect diverse, realistic user strategies?

This is a valid concern, and we would like to point out that current LLMs have decent abilities in simulating human characters [1, 2]. Furthermore, simulating interactions at scale would be impractical with real users. We validate our simulated interactions through expert review and a quantitative metric (as detailed in Section 5.2). Future works that involve human-in-the-loop evaluation can complement these simulations, which we'll discuss in the paper.

[1] Park et al. 2023. Generative Agents: Interactive Simulacra of Human Behavior. https://arxiv.org/abs/2304.03442

[2] Zhou et al., 2024, SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents

Fig 1 isn't particularly informative. Perhaps the authors could also include the outcome, e.g., the proposed metrics and rationale for applying them to the specific scenario.

Thank you for this suggestion. We will update Figure 1 in the future version to include the relevant information.

How does the paper define safety and risk? & How do you define "interactional safety risks"?

We define safety risk as the absence of harmful or undesirable outcomes in AI-agent interactions. The definitions are grounded in existing AI safety literature (e.g., AIR 2024). Interactional safety risks refer to harmful or undesirable outcomes that emerge during dynamic, multi-turn interactions between AI agents, human users, and the environment. We will add that in the next version of the paper.

Questions

We address the additional questions below:

How does using GPT-4o as both the user and evaluator impact the diversity and realism of the evaluations?

Using GPT-4o for dual roles introduces consistency that simplifies validation but may reduce diversity. We address this by:

• Incorporating expert annotations to validate automatic evaluation.
• Designing characters to span a wide range of user intents and diverse profiles (e.g., name, gender, personality, hobbies, etc.), minimizing the risk of overly uniform interactions.
• Maintaining information asymmetry between evaluators and simulated users (Figure Appendix B.1) to improve the realism of the simulation.

What specific types of errors occurred in the 10% of instances where the evaluator's judgment did not align with human evaluation?

Our human annotations do not reveal a significant trend of the evaluator consistently underestimating or overestimating risks. In some cases, the generated content was generic and deemed not risky by expert annotators but was flagged as risky by the evaluator. Sometimes the evaluator misses some subtle cases that could lead to risky outcomes. We will include specific examples in the appendix to illustrate these instances.

What prompted the creation of the 21 additional scenarios, and how were they validated?

The 21 additional scenarios were designed by the authors according to their real-life scenarios to add practical relevance and coverage to our scenario collection. All the scenarios are validated by two independent expert annotators.

---

We hope our responses address your main concern and we're open to continuing the discussion.

We thank the reviewer for recognizing our work as timely and important and for highlighting its significant contribution to the AI safety literature. We appreciate your thoughtful review and constructive feedback, which we address below:

---

Some risks are more critical than others…consider (for example) implementing a weighting system or severity scale for different types of risks based on their potential impact and scale.

We agree that differentiating between risks based on their severity and potential impact would enhance the interpretability of the framework. We do incorporate a Likert scale evaluation that provides a structured way to assess and quantify the severity and potential impact of risks (See Section 4 and Appendix F)

However, as HAICOSYSTEM represents an initial framework, we recognize the challenges of measuring the impact of certain risky behaviors from AI agents. Future works could define weights tailored to their specific problems or domains of interest.

Your current metric, "risk ratio" is the proportion of risky episodes over the total number of episodes. However, risk categories aren't equal in the number of risks they contain. What's the implication of this? & Would it not be better to compute "risk ratio" as the ratio of the number of risks observed in a given episode over the total number of possible risks in a given risk category (then averaged over the total number of episodes)?

We acknowledge that different episodes may involve multiple distinct risks, and the current calculation of the overall risk ratio might not fully capture this complexity. To address this, we will add the calculation to account for the number of risks per episode and include it as an additional metric in the paper.

---

We hope our responses address your main concern and we're open to continuing the discussion.

[1] Zeng et al., 2024, AI Risk Categorization Decoded (AIR 2024): From Government Regulations to Corporate Policies

We thank the reviewer for their detailed feedback, constructive critique, and acknowledgment of the importance of our work in defining methods to evaluate AI agents across diverse domains. Below, we address the key concerns and questions raised.

Scope of Claims and Complexity Acknowledgment

We appreciate the reviewer’s concern about the breadth of our claims. We acknowledge that HAICOSYSTEM does not aim to provide “definite guidance” or a one-size-fits-all evaluation solution for AI agent safety issues. Instead, our goal is to offer a flexible and modular framework and codebase that highlights and probes safety risks across various interaction scenarios. In general, we are happy to re-frame some of the texts in the paper to make it clear that:

• HAICOSYSTEM serves as a framework for running diverse simulations and automatically identifying potential errors at a high level. This allows humans to review flagged episodes, conduct manual inspections, develop error-specific detectors, and implement targeted fixes. The framework’s primary goal is to enable large-scale simulations while surfacing high-level issues as an initial step. In future iterations, we plan to clarify the scope of our contributions in Sections 1 and 4, explicitly reflecting this nuanced approach.
• Complexity Acknowledgment: We will incorporate a discussion in Sections 6 and 7 on the challenges of user modeling, risk operationalization, and the interplay between failure probability and consequence, with references to more literature on these topics.
• Evaluation dimension: We will better clarify our validation of safety risk dimensions and stress that our evaluation serves as an initial step rather than a definitive claim of completeness or sufficiency.

Essentially, we envision HAICOSYSTEM being used as a probing tool to uncover issues that humans can then inspect, develop specialized detectors for, and patch. In this context, the completeness or sufficiency of the system is less critical, as its primary purpose is to assist human oversight and intervention. Does this perspective align with the reviewer’s concerns, or are we perhaps misunderstanding the core of the reviewer’s feedback?

Sandboxing Terminology

We acknowledge the potential confusion caused by our usage of “sandboxing.” In the revised paper, we will clarify this term in Section 1 to explicitly describe our framework’s role in simulating interactional safety risks, distinct from its conventional usage in CS for behavioral isolation.

We have updated our supplementary material to include a demo code. We aim to make HAICOSYSTEM a practical probing tool for identifying AI safety issues that humans can inspect, complementing our automated evaluation and benchmarking efforts. The demo code enables easy rendering of interaction trajectories between simulated users, AI agents, and environments, which facilitates qualitative analysis in our paper (see more examples in Appendix G).

We look forward to the reviewer's feedback and further engaging in discussions.