AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories

Dear reviewer,

We thank you for their positive feedback on AgentRewardBench, particularly recognizing its high-quality dataset, rigorous human annotations, comprehensive LLM judge evaluation, and clear presentation.

We would like to address your concern:

The benchmark itself is novel, however the underlying concept is an incremental advancement.

We would like to point out that our work is far from incremental. AgentRewardBench is the first dedicated benchmark to rigorously assess LLM judges specifically for web agent trajectories. We made several key contributions. (1) Novelty of the Benchmark: Creating a dataset of 1300 annotated trajectories across 5 diverse web environments with high-quality expert annotations is a substantial undertaking. This enables standardized, reproducible research into automatic evaluation for web agents. (2) Addressing Existing Limitations: We empirically show the shortcomings of traditional rule-based evaluations, highlighting the need for nuanced automatic evaluations. AgentRewardBench provides the means to systematically validate this approach. (3) Specific Focus on Web Agents: Evaluating web agents is uniquely challenging. Our benchmark is tailored to these challenges, considering representations like accessibility trees and screenshots, distinguishing it from general LLM evaluation studies.

Therefore, while the LLM-as-a-judge concept may extend existing ideas, creating AgentRewardBench as the first benchmark for this complex domain is a crucial advancement. It provides a foundation for future research to evaluate and improve judges for web agents.

Dear reviewer,

Thank you for your thorough review of our work. We would like to address your concerns and questions below.

Including “Agent-as-Judge” as a baseline

The evaluation of judgment methods appears to be missing an important baseline of the Agent-as-Judge approach. This method is recognized as a powerful method for the automatic evaluation of agent trajectories. Including results for this baseline would strengthen this study and provide a more comprehensive comparison.

Thanks for pointing us to this work, which we will include in the related works section for camera ready.

We agree that the Agent-as-Judge (AaJ) approach is a promising method for evaluating agent trajectories. However, AaJ assumes the judge can interact with the environment after the agent finishes executing actions, but this isn’t feasible when the environment state cannot be preserved or shareable across agents (including AssistantBench, WebArena, WorkArena, etc.). However, we agree that Agent-as-Judge could be useful for web environments where the state can be preserved and made available to the judge easily.

After the anonymity period, we will reach out to the AaJ authors to explore ways to adapt their framework for cases like ours.

Questions

Training models towards a reward model is an exciting application of the annotated reward labels provided in this work. Have the authors explored training a model using the collected dataset? If so, how does this influence model or agent performance? Any results or insights from such experiments would be valuable to share.

We used the entirety of the collected trajectories as part of the benchmark. However, we believe that collecting additional trajectories for training reward models is an exciting direction for future research. Our benchmark will be the perfect fit for evaluating the performance of such finetuned reward models.

Hi reviewer! A quick reminder that the discussion period ends on 6/10. Please make sure to read and respond to the authors' rebuttals -- It'd mean a lot to the authors who have put in a lot of efforts into their work!

Dear reviewer,

We thank you for highlighting the scale and diversity of our benchmark, as well as the inclusion of expert annotations, which we agree will be of interest to the research community. We are grateful that you recognize the comprehensiveness and rigour of our evaluation framework, as well as the clarity of our writing.

We would like to address your concerns below.

Importance of recall compared to precision

the recall can be as important as precision in many RL applications. For example, if an LLM judge is employed as a reward model, it is important that incorrect trajectories are not noted as correct to give LLM agents wrong signals.

We agree that “it is important that incorrect trajectories are not noted as correct”. This was indeed our primary motivation for highlighting precision, as the number of incorrect trajectories being judged as correct (false positives) does not influence recall. On lines 240-244 of our submission, we explain why we use the precision score:

… we are interested in increasing the number of true positives (actual successful trajectories) while reducing the number of false positives (failed trajectories added to the dataset due to poor LLM judgments). For reward modeling, we also want to prioritize true positives since they are the primary signals for many RL algorithms, while false positives should be minimized to avoid introducing noise to the loss function.

As we can see from the precision score, defined as tp / (tp + fp), where tp is the true positive and fp is the false positive, maximizing this value will result in fewer incorrect trajectories being labeled as correct. Thus, precision precisely addresses your concern.

That said, we recognize the value of reporting recall and F1 for completeness. That is why we have already reported both metrics alongside precision. For instance, we report in table 1 that screen-based Claude 3.7 Sonnet achieves a precision of 69.4, but also an F1 score of 72.7 and recall of 76.3; the finegrained recall and F1 scores are included in the Appendix tables 7-8.

It would not require additional effort to move the results from table 8 to table 1, if you believe this would be a valuable addition. We are also open to giving more emphasis to the F1 scores we reported, if you believe there are strong reasons to highlight other properties of the recall component.

Inclusion of side effect and repetition in error analysis

The "error analysis" section is not very complete: it only includes only failure cases on the "success / completion" dimension of LLM agents, missing the other two dimensions (side effect, repetition) included in the benchmark.

We emphasize the completion dimension in the error analysis in order to remain focused on the discussions from sections 4.2 to 5. However, we are happy to add supplementary error analyses that would cover the side effect and repetition dimension, which will be a straightforward addition to the paper.

We hope this clarification addresses your concerns and encourages a fresh evaluation of our work. Given that we clarified the nature of the precision score, and that further error analyses will be added to the camera ready version, we would like you to reconsider your score.

We are happy to discuss further.

Dear reviewer,

Thank you for your insightful review, and for highlighting that it is a first-of-its-kind benchmark addressing an under-explored area of web agent evaluation. We agree that it will help improve automatic evaluation methods, and we are glad you appreciated the substantial details from our methodology and annotation processes.

We would like to now address your suggestions on how to improve the paper.

Suggestions

Related works should have focused more on LLM judges and existing benchmarks that evaluate them rather than web agents / environments / trajectory synthesis.

Thank you for this suggestion. We have already included the following papers about LLM judges and benchmarks: LLM-as-a-judge (Zheng et al., 2023), AER (Pan et al., 2024), and RewardBench (Lambert et al., 2024). We will add more past and concurrent works on the topic of LLM judges and evaluation, including Agent-as-a-Judge (Zhuge et al.). We are happy to incorporate your suggestions as well.

(1) these researchers might interpret success or failure in a way that's somewhat different from the more typical user (2) A small group of closely working experts might develop a very specific, shared understanding of the annotation criteria that isn't explicitly documented and might not be easily replicable by an external group.

…

The resulting consensus should be compiled and included in the appendix as annotator guidelines.

Thank you for this insightful suggestion. In addition to including the UI and annotation instructions (appendix A.2 and Figure 4), we recognize the value of discussing the resolution process and potential shared understanding that is not already present in the instructions. A salient reason for mismatch is ambiguity of the instructions (the instruction might mention “buy an black t-shirt”, but does not specify if it is fully black or can have other graphics). In such cases, annotators are advised to go for the most lenient option. We will create a new section, A.4, that will discuss this.

having experts with deep domain knowledge as annotators is also beneficial (focusing on things that matter rather than quirks of web agent trajectories), since questions related to trajectory optimality and side effects of unnecessary actions might be harder for the average user to answer.

Overall, the strengths of this paper outweigh these minor points.

We agree that it is hard for the average user to analyze web trajectories; this is the primary reason why we decided to leverage expert annotations.

Questions

Q1: Nitpick: In the Appendix, Line 512 mentions AgentLab… Thanks for catching this, we will fix this in the camera-ready.

Q2: Was the code/dataset not included as supplementary data?

Our code and dataset are available and we will share the link after the anonymity period.

Q3: Did you consider annotating for more granular success/failure signals?

We initially considered step-level reward annotations, but found them costly to obtain and not especially helpful: current judges already struggle with trajectory-level evaluation. Instead, we focused on annotating specific failure modes like repetitions and side effects, which provided more actionable insights. Step-level signals remain valuable, and we view them as an important direction for future work given better judge capabilities and more resources.