HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Interactive AI Agents

Major Concerns Addressed

Realism Level Imbalance

Concern: "There is an imbalance between the counts for different realism levels, as shown in Figure D.3."

Response: We acknowledge this limitation and will address the statistical impact more clearly in our analysis. Despite the imbalance, our results show consistent patterns across realism levels, suggesting robust findings.

Changing Definition of "Realism" of scenarios

Concern: "More practical 'realism' should consider unpredictable environment behaviors, system failures, edge cases, evolving goals/strategies."

Response: We agree that the term "realism" may be confusing and plan to replace it with "level of deployment," which better reflects our intent. This term is meant to categorize scenarios based on how deployable they are today versus in the future. For example, some tasks involve AI agents autonomously managing finances, negotiating with humans, or coordinating multi-step tool use—capabilities not yet widely available but plausible in future AI systems. This framing allows us to explore both current and emerging safety risks.

Multi-session Limitations

Concern: "Multi-turn dialogues are still different from capturing complex real-world interactions. Users/models may adapt strategies over longer sessions."

Response: We acknowledge and will add this to our limitations. Future work could explore adding multi-session components as well to capture longer-term adaptation patterns (e.g., through simulated user memory components).

Domain-specific Evaluation Uncertainty

Concern: "For domains where judgment seems less uniform, it should affect the conclusions drawn on these domain's safety risks." and question: “How many annotators are in the human study? The Cohen-kappa scores seem quite low for some categories like Systems and Operational Risks and Goal (see table G2). What could be some explanation? How to better design the evaluator to capture evaluation uncertainty?”

Response: We have 12 annotators in total. The subjective nature of some safety categories contributes to lower Cohen's kappa scores, which would likely be mitigated by further training the annotators. Importantly, the Cohen’s kappa and Pearson Correlation scores are already relatively high for subjective tasks such as safety or bias labeling (e.g., Hate Speech Annotations). The goal dimension is not used in the main findings of the paper due to its low agreement.

Actionable Guidance

Concern: "What actionable guidance does HAICOSYSTEM provide? If future work were to train on these episodes, how should they proceed?"

Response: HAICOSYSTEM uniquely points to gaps in training safe LLMs by identifying: (1) Multi-turn failure patterns not captured in static benchmarks, (2) Tool-use specific vulnerabilities, (3) User intent-dependent safety failures. We don't encourage direct training on these episodes, but rather using our dataset just for evaluation purposes. However, we would encourage future works to create new scenarios and simulations based on our framework and see how agents trained on those new scenarios might generalize to improve the original benchmark scenarios. We thank the reviewer for pointing this out and will include those points in the paper.

GPT-4o Dependency & Real-world Validity

Concern: "All users, environments, and risk judgments relying on GPT-4o; real human behaviour and real APIs may change outcomes."

Response: We thank the reviewer for pointing this out, and please check the general response for this concern. While we acknowledge that real human behavior and real API differences can affect outcomes, this limitation is inherent to all current automatic safety evaluations, but that does not make them useless. Even with different groups of human annotators or different API configurations, results will vary. Rather than aiming for exhaustive coverage, our work offers a controlled and reproducible framework to highlight these emergent safety risks at scale.

Same-vendor Bias in Evaluation

Concern: "Using an LLM from the same vendor to judge other LLMs can introduce shared biases and mask certain failure modes."

Response: Human evaluations calibrate and validate our automated approach. A 90% agreement rate between automated and human ratings suggests vendor bias does not meaningfully affect our findings. To further show the robustness of our framework, we try Gemini-2.5 Pro as the evaluator and re-evaluate the existing simulations.

As shown in the table above, the overall trend remains consistent when switching from GPT-4o to Gemini as the evaluator. This suggests that our evaluation, particularly for targeted safety risks, is relatively robust across different LLM evaluators, likely due to the use of scenario-specific checklists we designed for each task.

Training Data Independence

Concern: "111 scenarios are GPT-generated from earlier datasets; the novelty and independence of these scenarios from training data are not quantified."

Response: We address this concern by substantially transforming prior benchmarks through the addition of multi-turn interactions and tool use. These features introduce richer context and decision-making complexity, which are rarely present in model training data, making memorization unlikely. While our scenarios are inspired by established benchmarks, our modifications extend far beyond surface changes, enabling more robust evaluations.

Empirical Evidence for Real-world Prediction

Question: “What specific empirical evidence shows that the safety failures HAICOSYSTEM detects in LLM‑simulated, multi‑turn interactions reliably predict the failures that would occur with real human users and actual tools?”

Response: Our results show failure patterns in simulation align with human-judged safety violations, supporting their real-world relevance.

Prompt-based Leakage Prevention

Question: "how do you rule out the possible correlations or prompt‑based leakage between these roles that might inflate agreement statistics (e.g., the reported 90 % accuracy)?"

Response: Our codebase is based on the Sotopia (https://github.com/sotopia-lab/sotopia) simulation platform, which separates user, environment, and evaluator roles to prevent prompt leakage.

Tool Call Error Analysis

Question: “Annotators judged 94 % of tool calls “free of critical errors.” Can you quantify what kinds of remaining 6 % errors occur?”

Response: We provide a breakdown of the remaining 6% of tool call errors by type and severity as follows:

Thanks to the reviewer for pointing this out, and we will include this analysis in the paper.

Risk Ratio & Challenge Redundancy

Concern: "Risk-ratio would also be high if the challenges are actually redundant with each other."

Response: We deliberately designed the tasks to be diverse across multiple dimensions. The 132 scenarios span 7 domains and vary systematically in user intent, interaction complexity, and tool requirements (Please check Appendix D for more details.). To further address concerns about scenario overlap, we conducted an n-gram analysis across all 143 scenarios in our dataset. Using 3-gram extraction with Jaccard similarity coefficients, we found exceptionally low cross-domain overlap (0.5% average similarity) between different safety domains, confirming that each domain targets distinct failure modes. Even within domains, scenarios maintain high diversity with an average internal similarity of only 13.0%. The most diverse domains (Technology & Science at 1.2%, Healthcare and Personal Services at 5.5%) demonstrate minimal redundancy, while only Politics & Law showed higher internal similarity (37.6%) due to shared legal terminology. We thank the reviewer for raising this point and will include such analysis in our paper.

Multi-turn vs Single-turn Failure Rates

Concern: "It is not clear whether multi-turn interactions would be expected to have more failures simply by virtue of having more chances (turns) in which to fail."

Response: We empirically demonstrate that multi-turn interactions surface more safety risks. We hypothesize that one of the reasons could be that multi-turn settings inherently expose more opportunities for failure, and we would like to point out that we reveal unknown safety risks that static benchmarks miss, which is one of the advantages of our framework.

Automated Safety Evaluator Reliability

Concern: "The authors admit that their automated safety evaluator should not be trusted."

Response: Respectfully, this framing is incorrect. Our evaluator was rigorously validated, showing 90% agreement with human annotations, robust inter-annotator consistency, and tested performance on edge cases. While there are still some minor limitations, the empirical validation supports its reliability. What we do emphasize is that our benchmark should not be trusted to guarantee the safety of a system.

Human Expertise Limitations in Anticipating Societal Impact

Concern: "The validation of the evaluator against human experts is insufficient for anticipating societal impact because human expertise cannot be trusted in this area."

Response: We would like to further clear up a misunderstanding. If this is about the evaluation of the specific "societal risk" category of harms, that risk category does not actually aim to measure the actual impact that technology might have on society (which, as you correctly point out, is impossible to predict even for experts). Instead, it measures whether the interaction logs show traces of agent behavior that align with specific risks related to manipulation, public opinion swaying, etc. While the category name comes from the AIR paper (https://arxiv.org/abs/2406.17864), we will rename this category to something like "Societal Manipulation Risks" to clear this up.

If this is about human expert judgments in general, we respectfully disagree with the reviewer. While human judgment isn’t perfect, it serves as a widely accepted way to validate the scores from LLM-as-a-judge (e.g., ToolEmu and SOTOPIA). We use detailed annotation guidelines and multiple annotators to ensure high-quality reference labels.

AI safety benchmarks versus Redteaming

Concern: “If we would accept reports on the cited alternatives, then surely we should accept this report as well” and “Red-teaming is valuable because it forces us to face how far out-of-our-league we are (and the evaluator cannot provide that value).”

Response: The reviewer appears skeptical of prior widely-used and peer-reviewed AI safety benchmarks. While we acknowledge the limitations of automated evaluation, we respectfully disagree with the notion that it is not useful. On the contrary, we argue that it serves as a critical first line of defense in AI safety, helping to reduce the burden of the already labor-intensive red-teaming process. Moreover, even human red-teaming has many flaws, as it often suffers from vague objectives and ill-defined settings (see Red-Teaming for Generative AI: Silver Bullet or Security Theater?)

Thanks for acknowledging our work as timely and relevant.

Major Concerns Addressed

Lack of Human-in-the-Loop During Benchmarking

Concern: "While the paper is about human AI interactions, there is no humans in the loop during the benchmarking of the system. This is a huge limitation of the work."

Response: We thank the reviewer for pointing this out, and please check the general response for this concern. Overall, we believe our work is the first step to provide a multi-turn interactive environment for AI agents in the automatic safety evaluation.

Tools Not Grounded in Real APIs

Concern: "Tools aren't grounded in real APIs and lack real world operational testing of the system."

Response: We thank the reviewer for pointing this out, and please check the general response for this concern. While real-world APIs offer realism, they pose significant safety risks during testing. High-fidelity simulations like WebArena also require substantial engineering effort and often have limited scope (e.g., no open-source banking API for safe use). In contrast, our framework supports custom API integration, allowing users to easily simulate and evaluate their own APIs in a controlled environment.

Limited Risk Quantification

Concern: "Evaluations only talk about the risk ratios and don't provide any further categorization/quantification of detailed risk metrics."

Response: Due to space limits, we provide such fine-grained analysis in Appendix Tables F.1 and F.2. We also did ablation studies for representative models in the main text (section 5.3).

Questions Answered

Model Vulnerability Research

Regarding the concern about newer models being more vulnerable at specific agentic tasks, we acknowledge research suggesting this and agree that more nuanced analysis is needed. For example, as shown in Tables F.1 and F.2, Llama3-70B is not necessarily safer than Llama3-8B in some safety dimensions. Our main findings show general safety improvements. We agree that recent reasoning models (e.g., Deepseek-R1 and O1) might reveal new safety challenges. Therefore, we evaluate both representative reasoning models:

Compared to GPT-4o, both O1 and R1 perform better, with R1 exhibiting a lower overall risk ratio. Interestingly, O1 shows only marginal improvement over GPT-4o, suggesting that adding reasoning capabilities does not necessarily lead to a safer model. We thank the reviewer for raising this point and will incorporate this analysis into the paper.

Tool Execution Environment

We clarify how we utilize tools in our platform: We run tools in a simulated sandbox environment with full logging and manually verify the correctness on a random sample. Our tool calling format is very similar to MCP. We recognize that there are various ways to implement tool calling functionalities, which might result in slightly different evaluation numbers. Instead of trying to systematically study the influence of ways of calling tools in AI agent safety, we aim to demonstrate that adding tool calling could pose more safety risks to AI agents.

Domain Expert Involvement

Regarding domain expertise in scenario development, our team consists of researchers with strong backgrounds in AI safety, though not in specific application domains such as medicine. To ensure scenario quality, we conducted iterative reviews and validation passes to refine realism and relevance. We acknowledge such limitations and will stress them in the limitations section of the paper.