AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents

We thank the reviewer for their detailed comments. We provide our responses below.

I'm concerned that the tasks are unrealistic as a result of being so synthetic.

We agree with the reviewer that models may be more willing to comply with tasks that are obviously fake. However, one of our ethical considerations when creating the benchmark was to minimize accidental collisions with real entities, such as post IDs, accounts, and people names. This is one of the reasons why some parts of the benchmark are not fully realistic. Still, we think frontier models should not help a user to perform a task like hiring a hitman or posting harmful tweets, even if some parts of it may seem artificial. Additionally, many of our tasks are more realistic than the commented on tasks (see the released dataset in the supplementary material). We would be excited about future work specifically exploring how refusal behavior changes in the face of identifiable fictional tasks.

I'm really interested in the question of whether models are less capable after being jailbroken. But because the tasks in this paper seem artificial and much easier than real tasks, and that makes me feel like I haven't learned much about the question from these results.

We appreciate the reviewer’s interest in examining whether model capabilities are preserved after being jailbroken. Studying this question was a core design consideration of AgentHarm, and is the reason we also released a benign dataset variant. We think that our results do provide significant information on this question:

• First, note that though the tasks are artificial / easier than end-to-end misuse tasks, they do capture differences in model capabilities. For example, within model families, the more capable models perform noticeably better than weaker models: GPT-4o mini has 79.9% score on benign AgentHarm behaviors, while GPT-4o has 89.9%; Sonnet 3 has 73.6%, while Sonnet 3.5 has 82.0%. This is also true for the harmful dataset variant: GPT-4o mini has 77.5% non-refusal harm score, while GPT-4o has 90.1%; Sonnet 3 has 79.7%, while Sonnet 3.5 has 91.0%.
• After jailbreaking, we compare the performance of models on harmful tasks to benign tasks, and find that capabilities are not significantly degraded. For example, GPT-4o mini and GPT-4o have non-refusal harm scores of 69.8% and 84.2% respectively, compared to benign scores of 79.9% and 89.9%. Again, more capable models perform better at the jailbroken harmful tasks. That is, a jailbroken stronger model not only approximately matches its performance on benign questions, but also is shown to be more successful than jailbroken weaker models of its family.

We think this is strong evidence that capabilities are preserved. However, we strongly agree that more realistic and difficult tasks would be even stronger evidence—and think this is especially important as stronger models are developed and thus our tasks do a worse job distinguishing between model capability levels.

I'm sure my prompt was very different from the one used (e.g. I didn't have an agent scaffold), but this still seems very different from your results. Can you please explain this discrepancy?

It is very important to provide not just the prompt, but also the tools associated with this task to the model. This makes a substantial difference in terms of refusal rates, as our new chat-only results confirm (see Appendix C of our revised paper; the new results are highlighted in blue). For example, without providing tools, Claude 3.5 Sonnet refuses almost always (on 95% prompts) and GPT-4o refuses in most cases (on 72% prompts). When tools are provided, these numbers systematically decrease. Note that providing the tools also requires knowing the exact tool format used by GPT models, which is probably only possible to do via the API.

To what extent do you think the results on your benchmark are informative about the results you'd get if you measured more realistic tasks?

As we tried to emphasize in the paper, we measure basic agentic capabilities, so we agree that most tasks are relatively easy and do not require long-horizon planning (that, we believe, is not yet present in the current frontier LLMs). Thus, we think this makes our benchmark just right for measuring the agentic capabilities of the current generation of LLMs. However, we agree that some aspects of the benchmark are not fully realistic and we will be more explicit about this in the paper.

We thank the reviewer again, and we hope they will reconsider their original score in light of these clarifications.

Questions

you state that safety training techniques seem to not fully transfer to the agent setting - any insights why?

We think that it is most likely due to the training vs. test distribution mismatch. To improve upon this, one would probably need to collect more agentic data for refusal training. We note that we’ve added an additional experiment to further explore the difference in chat and agent refusal in Appendix C (Table 8).

can you discuss what distinguishes transferable jailbreaks from those that lead to "incoherent low-quality agent behavior"?

One can imagine many jailbreaks that would not be effective for the agentic setting. For example, using some low-resource language or some fictional scenario that may not preserve the right semantics of the agent's output.

can you comment on how close to the capability frontier you get with your current scaffolding?

We think that our experimental setup represents a reasonably good attempt at evaluating frontier LLMs as agents. In particular, it is consistent with popular agentic scaffolding, e.g., as described in the OpenAI documentation https://platform.openai.com/docs/guides/function-calling. However, it is clear that stronger results can be obtained using more test-time compute. One approach in this direction is our best-of-n sampling experiment in Section 4.3, but one can also imagine some test-time search techniques and potential backtracking in case an agent takes the wrong turn. Another degree of freedom is different prompting techniques (CoT, ReAct, etc), which we briefly explored in the same section. Since our work is a benchmark paper, we think it is sufficient to cover the basic agentic setup and leave more advanced techniques for future work.

can you comment on how often grading functions miss alternative valid exec traces?

For each model that we tested, at least one co-author ensured that our grading criteria were appropriate for all tasks and did not miss a significant number of alternative execution traces. Moreover, our tools return error messages when incorrect arguments are provided, which can help models self-correct and return to the "happy path." However, for new models, we think it's possible that they might try to do something else that is not covered by our grading criteria, which is why we wrote this as a potential limitation of our benchmark.

We thank the reviewer again for the very valuable comments.

We thank the reviewer for valuable comments. We discuss the weaknesses and answer the questions below.

Weaknesses

Broad coverage of domains is a double-edged sword

Thanks for bringing the recent work of Anurin et al. (2024) to our attention. We think that both narrow and broad benchmarks are valuable and complementary. This was a deliberate design decision in our case: covering many harm categories and making sure that the sample size of the benchmark is large enough to measure fine-grained differences across different models. To the best of our knowledge, all existing narrow agentic benchmarks have a small sample size (on the order of 10-30 tasks; compared to our 110 base and 440 augmented tasks). We agree that deeper coverage on a particular domain is an exciting direction for future work. We have added further discussion of our considerations in Section 3.2: Design Principles.

missing a clear definition of malicious behavior

We designed our tasks following the terms of use of major LLM providers, such as OpenAI and Anthropic. For example, the athlete task is a clear example of harassment, which is not allowed by Anthropic’s terms of use (https://www.anthropic.com/legal/aup). I.e., it’s not allowed to “Coordinate the harassment or intimidation of an individual or group”. Similarly, for the Apple privacy task, it’s not allowed to “Create and disseminate deceptive or misleading information about a group, entity or person”.

We note that Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama-3.1 refuse to perform at least 75% of our agentic tasks (when no jailbreak techniques are used), which indicates that the refusal training used by these models does cover a lot of our tasks. Moreover, some of the refusals rates go up to 95% (e.g., for Claude 3.5 Sonnet) when tasks are formulated in a chat-only format (see our new chat-only results in Section C).

We note that our dataset is not designed to in any way comment on what tasks should and should not be refused, but instead to test robustness of intended refusal behavior. We have added further discussion on this in Appendix A of the updated paper.

need some empirical support on more complex tasks

In the paper, we tried to consistently frame our benchmark along these lines: "AgentHarm tracks basic agentic competencies". We think our repeated usage of the word "basic" in this context is consistent with the statement that the tasks are "relatively easy". Yet, even on these basic tasks (even on benign ones), we still see a clear performance difference between different LLMs (including within the same family, such as GPT-4o mini vs. GPT-4o). This suggests that these tasks are not too easy for the current frontier LLMs and provide a useful signal for evaluation, though we would be excited about more difficult tasks in future iterations of the dataset.

tasks are still toy tasks

multi-turn interactions not considered

enforcement of happy path of expected tool usage sequences may be too constraining

We agree with all these points, and we intended to be upfront about all of them as limitations of our benchmark in Section 5: Discussion. In particular, we mention there that our benchmark only measures basic agentic capabilities, single-turn interactions, and the grading criteria can potentially miss alternative execution traces.

Minor weaknesses

We thank the reviewer for pointing out these issues, which we will fix.

I found the bolding in table 3 confusing: what counts as best here?

We boldfaced the highest harm and refusal scores across different prompting techniques for each model. We have made it explicit in the caption, thank you.

We thank the reviewer for the detailed comments. We address the weaknesses and questions below.

W1. We agree with the reviewer that multi-turn conversations, dynamic toolsets, and generally making the benchmark more open-ended are exciting future directions. However, we believe our simpler benchmark still has significant value given the lack of more comprehensive benchmarks focusing on agent harmfulness. Moreover, given the achieved harm scores and compliance rates, we believe our benchmark already provides a valuable signal about the models' properties despite its simple structure, low cost, and ease of use. We are excited for future work to build on our dataset towards more realistic misuse settings.

W2 and Q1. We fully agree with these concerns and this is something we've spent a long time thinking about, and was a core design consideration for AgentHarm. However, we would like to clarify that in the grading functions we use an LLM judge only to check whether some specific and narrow parts of a response are on topic. In particular, unlike most other jailbreaking benchmarks, our primary metric does not judge the harmfulness of a whole response using an LLM. Instead, the harm score is computed according to a grading rubric that consists of multiple manually written conditions, where only some of them involve an LLM to judge the semantics of a specific part of an output. Moreover, each question has a different semantic judge with specific criteria, reducing the risk that they can all be systematically tricked. We also manually examined the execution logs for all samples and across different models to verify that the narrow LLM judges were robust to different answers.

For the refusal LLM judge, however, an attacker asking the model to refuse in a message between tool calls could indeed lead to a false refusal by the judge, as it uses a generic prompted LLM as opposed to the rubric approach above. There are two scenarios which we aimed to mitigate:

a) Tricking the refusal judge to say it was a refusal when it wasn't, to make a model appear safer. We partially mitigate this risk by separating the refusal and the harm score. If the refusal judge got attacked and misled, this wouldn't affect the harm scores, making it easier to flag compliance grading issues. The harm scores would still be high and thus such attacks would be identifiable from the benchmark scores.

b) Tricking the refusal judge to say it wasn't a refusal when it was, to make a jailbreak appear stronger. Again, we note that this error would not affect the primary metric (harm score). We spent significant time iterating on our refusal judge on multiple models, jailbreaks and questions, including manually verifying the correctness of the judge on models and questions it was not tuned on to ensure we do not overfit while developing the prompt. We additionally do not pass user messages to the judge, increasing the difficulty of prompt injection.

We have added a note in the paper (page 14) about the limitations of our refusal grading scheme, discussing both of these potential issues.

W3 and Q3. We have uploaded (as supplementary material) the code of the benchmark and 44 out of 66 public test base behaviors (176 augmented ones) and 8 out of 11 validation base behaviors (32 augmented ones). We plan to release additional behaviors in the near future.

Q2. In case someone wants to extend the benchmark for their own purposes (e.g., if someone has a category they are particularly concerned about), it wouldn't take long to add custom questions with custom tools in the same code framework that we have. All the questions are formulated in a standalone way with their own grading so this would be fairly simple. We are excited about future versions of AgentHarm which expand the scope of the dataset, though appreciate value in a relatively stable benchmark to allow for cross-model comparison and research. We also welcome community-sourced improvements of the benchmark, such as small bug fixes, that do not substantially change the original benchmark.

The benchmark presented in this paper could be used to optimize for harmful agents.

We thank the reviewer for raising this concern. We’ve added a discussion of this in Appendix A. Unfortunately, this is true for any harmful benchmarks, which was also a consideration we had when opting to use a predefined tool set and not include real world interactions. We believe the complexity and realism of this benchmark are sufficient to provide a signal of the model's real misuse potential while limiting its optimization potential for transferring to real-life tasks. In the long run, we expect the benchmark to be also useful for creating better safeguards against agent misuse.

We thank the reviewer again, and we hope they will reconsider their original score in light of these clarifications.

We thank the reviewer for the valuable comments. We address the two weaknesses below.

1. Although the benchmark is claimed for LLM-agent, the author fails to demonstrate the main difference/challenge between general LLM robustness and LLM-agent robustness, except for the integration of different tools. From the design and evaluation, it is not clear to us whether an attack that can effectively compromise general LLM tasks (e.g. task planning) also succeeds in attacking LLM agents. This is essential to understand the novelty and contribution of the work

We agree with the reviewer that the novelty of the work is diminished if LLM-agent robustness and chatbot LLM robustness are not distinct. However, we strongly believe these settings are importantly different. We discuss two ways they differ, including an additional experiment we’ve added in response to these concerns.

Agents may be less robust than chatbots. We have collected a new chat-only dataset that is designed to have similar requests to our main dataset but doesn't require tool use (i.e., a direct single-turn answer is sufficient). We have added the details and results in Section C in the appendix. On this new dataset, our template attack is noticeably less effective, increasing the refusal rates on GPT-4o from 9.1% to 31.8% and from 29.5% to 72.7% on Claude 3.5 Sonnet compared to the agent setting. Furthermore, the starting refusal rates are systematically higher in the chatbot setting despite the tasks being similar, suggesting the refusal training may have focused on the chat setting and sometimes struggles to transfer to the agent setting. We have updated the paper with these new results for several LLM agents.

Agent jailbreaks require coherent multi-turn malicious outputs. Whereas standard jailbreaks aim to extract harmful information contained in a single response generated by an LLM, our setting requires a whole sequence of generated responses—that contain multiple function calls—to be high-quality and malicious. It is common when jailbreaking models to experience model responses which start harmful but then “recover” and refuse to continue. These recoveries greatly affect harmfulness in the agent setting, but are less important in the chat setting where the information may have already been output by the model.

We have further clarified this distinction in Appendix A of the updated paper.

2. In the evaluation, comparison with existing benchmarks is lacking - what's new insights the proposed methods bring up by introducing the new/more comprehensive data and metrics.

We believe the most relevant existing agent benchmarks are AgentDojo (NeurIPS’24 Datasets & Benchmarks Track) and ToolEmu (ICLR’24).

AgentDojo focuses on prompt injections and harmless requests. In their tasks, harm comes from an attacker that performs a prompt injection as part of a tool output. This is different from the setting we consider, where harm comes from a malicious user who directly provides a harmful query to an LLM agent.

ToolEmu focuses on scenarios where the underlying user intent is assumed to be benign rather than malicious and there is no intention to direct the LM agent towards causing harm. Moreover, the benchmark uses an LLM to emulate tool execution and to grade (accidental) safety violations—which is something that we explicitly aimed to avoid by using fixed tool implementations and detailed grading rubrics.

We are not aware of any other existing benchmark that focuses on the harmfulness of LLM agents stemming from malicious user intent, at least as of September 2024 (however, there are some concurrent works submitted to this ICLR).

We have further clarified these distinctions in the Related Work section of the updated paper.

We thank the reviewer again, and we hope they will reconsider their original score in light of these clarifications.

Dear Reviewer RhVU,

Thanks again for your feedback. Since the discussion period is closing soon (December 2nd, Monday), we are wondering whether the revised version of our paper and our responses have addressed your concerns. If not, we would be happy to continue the discussion and provide further clarifications or explanations.

Dear AC,

In response to the ethics review of our benchmark, we added a paragraph on Ethical considerations at the beginning of Appendix A in the camera-ready version of our paper. We hope this resolves the ethics concerns.

Best regards,
Authors