Web-Shepherd: Advancing PRMs for Reinforcing Web Agents

Dear Reviewer THVf,

First of all, we sincerely appreciate the time and effort you took to review our work, and we are grateful for your recognition of its contributions.

---

W1

We also acknowledge the reviewer’s suggestion that evaluating on additional benchmarks would strengthen the paper. In response, we conducted supplementary experiments on the WorkArena benchmark $1$ , which differs from WebArena in task structure and domain.

Specifically, we used GPT-4o-mini as the policy model and compared the performance of greedy search and reward-guided search with GPT-4o-mini and Web-Shepherd (3B and 8B). As shown in the results below, reward-guided search with Web-Shepherd continues to show clear advantages, validating the transferability of our method to unseen websites.

Although we additionally experimented with benchmarks that include both textual and visual observations, our current model is mainly trained on textual inputs. We made this design choice deliberately, as we found that naively incorporating screenshots often introduces ambiguity and noise that can hinder performance. Nonetheless, we recognize that certain tasks (e.g., “find the red shirt”) require understanding visual context. This highlights the need for future work to develop more effective multimodal reward models that can jointly leverage textual and visual inputs for web agent training. Furthermore, we agree with the reviewer’s suggestion and plan to extend our experiments to VisualWebArena to better assess the model’s capabilities in multimodal scenarios.

---

W2

Thank you for raising this important point regarding efficiency. To evaluate whether our reward-guided search improves not only accuracy but also sample efficiency, we conducted a comparison against greedy search. As shown in the table below, our reward-guided approach significantly improves both success rate (Acc) and efficiency (Avg. Step to reach the goal) across different policies.

In particular, we observe that using a strong reward model (Ours-8B) leads to more efficient behavior, achieving higher accuracy with fewer steps, compared to using GPT-4o-mini as the reward model. This underscores the critical role of a robust reward model in guiding agents effectively during search.

---

Q1

As illustrated in Figure 5, WebShepherd is not a framework but a single unified model that performs two key roles: (1) generating a checklist for a given task, and (2) providing step-level rewards and natural language feedback for the agent’s actions based on the checklist.
To train the model, we use high-quality checklists derived from human-annotated trajectories with GPT-4o. Checklist generation is not distilled from GPT-4o, but learned directly from expert supervision, enabling strong generalization.

---

Q2

When we compare text-only input and multimodal input in the 3B setting, we do not observe a notable improvement from using the multimodal input. Thus, we do not further train 8B models for multimodal input. Furthermore, our GPU resource is not enough to run finetuning 8B VLMs.

---

Q3

Thank you for this insightful suggestion. To isolate the effect of checklist quality on reward modeling, we conducted an experiment using manually annotated checklists. Specifically, we annotated checklists for 50 tasks and compared reward-guided search performance using self-generated versus human-annotated checklists.

The results (see table below) indicate that there is no significant performance gap between the two settings. This suggests that once the checklist quality reaches a certain threshold, the performance is more influenced by the reward prediction or the policy model. However, since the evaluation was done on a limited set of 50 examples, further large-scale experiments are necessary to draw definitive conclusions.

Additionally, we observed that while our model generally produces useful and well-structured checklists, there are notable failure cases:

• Redundant Checklist: The model occasionally adds unnecessary subgoals, which can confuse the reward assignment or mislead the agent.
• Missing Subgoals: Some key steps are omitted, resulting in coarse-grained checklists that fail to reflect the full task structure.

Although these issues did not result in notable performance drops in this experiment, they highlight opportunities for improvement in checklist generation quality.

---

Q4

We appreciate the reviewer’s interesting question. When the checklist is completely wrong, we can assume that there is an error in the checklist generation given that the reward score does not increase after several attempts. In these cases, a practical strategy is to sample a new checklist from Web-Shepherd, possibly refining it based on the agent’s past trajectory.

Importantly, our model is explicitly trained to generate coarse-grained checklists rather than overly detailed ones. This design choice helps avoid overfitting to specific website structures and allows for better generalization to unseen websites. As shown in our main experiments on WebArena and additional experiments on WorkArena, Web-Shepherd performs effectively even in out-of-distribution environments.

Furthermore, we observed that even if part of the checklist is incorrect, the remaining steps still provide effective reward. This allows the reward model to continue offering valuable feedback, helping the agent refine its actions and move toward the goal, even when a change-of-plan is required.

---

Q5

Thank you for pointing this out. We will correct the equation by dividing by $K$ in the next version of the draft.

---

References

$1$ Workarena: How capable are web agents at solving common knowledge work tasks? (Drouin et al., 2024)

Dear Reviewer 88aJ,

We appreciate your thoughtful review and recognition of our efforts. We hope that the following responses help clarify the reviewer’s questions:

---

W1

Yes, Web-Shepherd is trained on both checklist generation and reward modeling. To ensure the checklist quality is not biased or limited by GPT-4o’s outputs, we provide human-annotated trajectories to GPT-4o when generating a checklist dataset. This ensures that the model learns from expert reasoning grounded in successful demonstrations.

We fine-tuned Qwen2.5-3B under two settings: (a) only reward generation dataset, and (b) joint training with both checklist and reward generation dataset. We evaluated both settings on WebRewardBench using ground-truth checklists. As shown in our results, this joint training does not lead to significant performance degradation. In fact, we observe gentle improvements in certain out-of-domain tasks, suggesting that multi-task learning may enhance generalization.

---

W2

Thank you for the insightful suggestion. To better understand the performance drop in the GitLab, we manually annotated checklists and conducted experiments using these human-annotated checklists. We observed performance improvements compared to using self-generated checklists. We found that the checklist quality for GitLab was slightly lower than for other websites, which may also have contributed to the observed performance drop.

However, the performance gain was still limited compared to the baseline (i.e., w/o trajectory search). We identified two main bottlenecks:
(1) Lack of correct action candidates due to limitations in the policy model, and
(2) Reward prediction errors that occur even when high-quality checklists are provided.

This is likely because the remaining failures after GPT-4o’s greedy search tend to be more challenging cases, which are less amenable to recovery through reward-guided search alone.

---

Q1

Here’s an example for clarification.
Task: “File upload” on a CMS

First, WebShepherd generates the checklist:
(1) navigate to the correct page,
(2) locate the upload button, and
(3) select a file.

Then, WebShepherd provides natural language feedback and a scalar reward for the agent’s actions. For example:

Action: “Clicked on the Settings’ tab”

• Feedback: “The agent clicked on the Settings tab, which does not directly lead to the upload interface…”
• Reward: 0.09 (checklist 1 is incomplete)

Action: “Clicked the 'Media' page”

• Feedback: “The agent navigated to the Media page, which is the correct destination for uploading files...”
• Reward: 0.31 (checklist 1 is completed)

This fine-grained, step-level supervision provides informative signals that help the agent make progress toward the final goal. Moreover, the natural language feedback can serve as an additional source of guidance when the agent needs to refine.

---

Q2

Following the previous work $1$ , we use the verbalizer to calculate the score. This allows the reward model to produce more discriminative scores by computing probabilities over semantically meaningful tokens (e.g., “Yes”, “No”) rather than relying solely on scalar outputs.

---

References

$1$ Generative verifiers: Reward modeling as next-token prediction. (Zhang et al., 2024)

We appreciate your observation regarding the W2 results, and we agree that even with gold checklists, the model may still struggle due to the inherent limitations of checklist-based reward modeling.

As you pointed out, the WebArena split in WebRewardBench includes relatively fewer instances compared to Mind2Web. This discrepancy arises because Mind2Web is based on publicly available human trajectories, whereas WebArena gold trajectories were manually collected by us, which naturally limits the scale. In this work, we focused primarily on validating our proposed PRM framework, and thus we were not able to collect larger-scale WebArena data within the current scope.

However, we fully agree that in order to develop more robust and accurate reward models for Web Agents, expanding the WebArena dataset is an important next step. Thank you again for your insightful feedback.

Dear Reviewer MZUs,

Thank you for your deep understanding of our work, and appreciate the interesting question you raised. We would like to address the concerns in the comments below:

---

W1

We notice that Web-Shepherd has already achieved pretty high performance on WebRewardBench, especially in terms of MRR. However, when examining step-level accuracy (Acc step) and trajectory-level accuracy (Acc traj), there is room for improvement. For example, in Table 6 (Appendix E), Acc traj of Web-Shepherd (8B) is around 50-60 in Mind2Web.

In addition, considering recent findings that reasoning models (e.g., OpenAI’s o-series) can be used as PRM $1$ and its superiority of evaluating web agents $2$ , we tested o4-mini’s performance on WebRewardBench. Even though it benefits from a long sequence of reasoning, we find o4-mini struggles in WebRewardBench.

---

W2

We fully agree with the reviewer’s point. We are still actively exploring the broader question of how to better guide web agents. As the reviewer noted, it remains unclear whether step-level rewards are inherently superior to outcome-based rewards, as each approach has distinct advantages and limitations.

• ORM: Outcome-based reward modeling assesses only the final state to determine task success or failure. This simplifies the task objective and yields a more straightforward evaluation criterion compared to PRM. However, as noted in WebJudge $2$ , evaluating a web agent’s performance solely based on the final state overlooks the quality of intermediate actions and introduces a significant gap compared to human evaluation.

• PRM: Process-based reward modeling evaluates agent behavior at a finer granularity by assessing step-level progress. While this provides a more detailed signal, it can introduce noise, especially when preferences over individual steps are ambiguous. In designing Web-Shepherd, we aimed to mitigate this by focusing on whether the agent achieves sub-goals, rather than modeling step-level preferences directly. As shown in Section 7.2, our approach (Checklist-based Generative Reward Modeling) effectively addresses key limitations of conventional PRM (e.g., Bradley-Terry modeling), demonstrating strong generalization to the WebArena benchmark.

For future work, a promising direction is to use both signals from ORM and PRM, leveraging the strengths of both to better capture task success and agent behavior.

---

W3

To compare PRM with ORM, we use the ORM provided in WebRL which is based on LLaMA-3.1-8B-Instruct and trained on the expanded version of WebArena-Lite (which is in-domain for this experiment). For fair comparison, we trained Web-Shepherd using the same backbone LLM, LLaMA-3.1-8B-Instruct. We show the results in the table below.

In terms of Total score, which is an average across all domains, ORM’s performance is par on w/o Trajectory search. There is a significant increase in Shopping domain (21.74 -> 32.61), but underperforms w/o Trajectory Search in other domains.
When comparing our PRMs with the ORM, our method outperforms in overall, showcasing the effectiveness of process-level guidance. Since this version is based on LLaMA-3.1-8B, which is usually weaker than Qwen-3, it results lower performance than our original Web-Shepherd. Lastly, while PRM chooses a single action to be executed without executing all action candidates, ORM needs to run 5 trajectories independently, which takes 5x more environment-agent interaction costs.

---

W4

We agree that testing on more benchmarks to cover diverse domains would help increase the robustness of our experiments. Thus, we expanded the trajectory exploration experiment to WorkArena [3], which is completely out-of-domain for Web-Shepherd. (We also carefully inspect other benchmarks, such as AssistantBench, we frequently face CAPTCHA issues that ban web agents from further exploring the websites. We are still exploring other benchmarks that can possibly run our experiments.)

We find that trajectory search with PRM also improves the success rate in WorkArena when we compare w/o Trajectory Search and using GPT-4o-mini as PRM (i.e., the Total success rate increased from 9.39 to 12.42). In addition, Web-Shepherd outperforms GPT-4o-mini over all domains except Menu. Furthermore, Web-Shepherd outperforms GPT-4o-mini across all domains except for Menu. A potential explanation for the relatively low performance in the Menu domain is the complexity of the menu bar in WorkArena, which includes multi-level dropdown structures and search boxes embedded within the dropdowns. In such cases, the policy model often generates highly unreliable action candidates, and Web-Shepherd appears to be undertrained for handling these complex interaction patterns.

---

W5

When we first observed the results of multimodal input in Table 1, they seemed counterintuitive, and we initially suspected an error in our implementation. After extensive verification and repeated trials, however, we found no mistakes. We later realized that similar patterns of low performance with multimodal input have also been reported in prior work. For instance, AgentRewardBench $4$ , which introduces a benchmark for evaluating ORMs for web agents, similarly observes that incorporating both text and image observations can underperform compared to using text-only observation. This degradation is due to the increased complexity of multimodal input, which may distract the model and hinder learning.

On the other hand, we observe subtle improvements with visual information. In Table 6 (Appendix E.1), we provide results on WebRewardBench with the generated checklist — in the main table, we provide results with reference checklist. In the table below, we observe gentle improvements over all metrics without ACC (traj) in Mind2Web. It seems that self-generated checklists require much less reasoning cost compared to reference checklists that are unfamiliar to the model.

---

References

$1$ Scaling Evaluation-time Compute with Reasoning Models as Process Evaluators (Kim et al., 2025)

$2$ An Illusion of Progress? Assessing the Current State of Web Agents (Xue et al., 2025)

$3$ Workarena: How capable are web agents at solving common knowledge work tasks? (Drouin et al., 2024)

$4$ AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories (Lù et al., 2025)

Dear Reviewer nLse,

We appreciate your comments and positive feedback on our work. We will address the concerns and the questions raised by the reviewer in the comments below:

---

W1

We agree that process-based rewards have been previously explored and are known to provide denser supervision, often leading to improved downstream performance. Our work does not claim to introduce an entirely novel concept, but rather focuses on adapting and operationalizing PRM in the context of complex, multi-turn web environments—an application domain where reward modeling poses unique challenges. While it may not be surprising that a small model fine-tuned on the PRM dataset can outperform larger off-the-shelf models, we believe our contributions lie in the careful design of the reward modeling, extensive analysis, and the empirical insights obtained from adapting PRM to this setting.

---

W2

Thank you for highlighting the importance of prior work such as Rest-MCTS and VisualPRM. As the reviewer noted, these studies have effectively demonstrated the benefits of process-level supervision, and our work builds upon these insights. The key distinction between our approach and theirs lies in how the process-level reward data is constructed. While Rest-MCTS and VisualPRM rely on MCTS or Monte Carlo sampling to annotate reward scores, such methods are heavily dependent on the performance of the underlying policy model. This dependency poses a significant limitation in the context of web agents, where the action and observation spaces are extremely large, resulting in a very low likelihood of obtaining positive rewards through rollout-based methods.

Our work provides a path toward modeling PRMs for web agents where the process reward is hard to estimate from the outcome reward due to relatively low success rate and significant cost of agent-environment interaction time during rollout. We leverage synthetic and human annotation in a hybrid manner and improve generalization capability. We will make sure that this discussion is included in the next version of the draft.

---

Q1

Following the reviewer’s suggestion, we conducted analysis on the effect of the (1) number of instructions, and (2) number of rejected actions in the dataset on the performance of the PRM. Specifically, we construct datasets using the subset of WebPRMCollection 0.25, 0.5, and 0.75 percent of instruction and its corresponding chosen-rejected pairs and 1,2, and 3 number of max rejected actions. We trained variants of Web-Shepherd with these datasets using the same model (i.e., Qwen-2.5-3B-Instruct) and hyperparameters. The results are shown in the tables below.

Overall, if we use about half of the original dataset (in terms of both the number of instructions and the number of rejected actions), there is a drastic decrease in ACC (traj) on both of the benchmarks. Especially, in the out-of-domain benchmark, WebArena, instruction ablation results in ACC (traj) decreases from 60.0 to 15.0, which suggests it failed to generalize to unseen domains. In rejected ablation, only decreasing by one rejected action is critical, resulting 60.0-> 20.0 ACC (traj) score in WebArena. These results highlight that both the number of instructions and the number of rejected actions are critical for training an effective PRM; reducing either significantly impairs generalization, particularly in out-of-domain settings such as WebArena.

** $Ablation on the number of instructions$ **

** $Ablation on the number of rejected actions$ **