Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation

We greatly appreciate your valuable feedback for improving our work.

Q1. A more detailed literature review of the related work

We agree that “Agent Planning with World Knowledge Model (WKM)” is highly related work and we elaborate more on the similarities and the differences between ours and WKM below.

Differences

WKM predicts the task knowledge required to generate actions not the effect of actions: WKM aims to provide knowledge for understanding the current state (i.e., state knowledge) and high-level guidelines for planning (i.e., task knowledge) prior to action prediction, such as "The egg is most likely in the fridge ... The workflows are: 1) locate and take the egg; 2) clean the egg using sinkbasin ..". Thus, the task knowledge $\kappa$ is predicted by the world knowledge model $\phi$, solely based on the user instruction $I$, i.e., $\pi_{\phi}(\kappa|I)$. Then, the policy agent $\theta$ predicts a trajectory $\tau$ based on the task knowledge, i.e., $\pi_{\theta}( \tau | \kappa, I)$.

Our world model, however, focuses on predicting the effect of specific actions on the environment rather than the prior knowledge before the task. This approach aligns more closely with the definition of a world model, expressed as $p(s_{t+1}|s_{t}, a_{t})$, which predicts the next state given the current state and action.

While WKM and our world model are designed for different purposes, we believe these models can be used simultaneously to provide agents with better understanding of tasks and current states, while helping to mitigate negative impacts on the environment.

WKM needs expert annotation: WKM requires expert trajectories for training the knowledge model, which is difficult to obtain in real-life scenarios. In contrast, our world model is designed to learn environment dynamics solely from agent trajectories that can be automatically generated without expert annotations.

Similarities

Both ours and WKM learn from the environment and guide selecting proper actions: The key similarities between WMA web agents and WKM are leveraging trajectories to learn information from the environment and guiding the agents with it. However, as we mentioned above, our work and WKM focus on different aspects of information.

Other works related to WKM

A line of work has explored leveraging the knowledge that can obtained from the experience within the environment to learn from success and failure, similar to WKM:

• AutoGuide [1]: To leverage the previous experience on the environment to improve LLM-based agents, they first extract knowledge (guideline) that helps predict a proper action and retrieve it when the agent faces similar situations. The extracted knowledge is similar to the task knowledge of the WKM paper.

• KnowAgent [2]: KnowAgent extracts action knowledge, which consists of a sequence of actions and the description on it, and saves the knowledge into a knowledge base to incorporate the knowledge for planning at inference time.

We will include a comprehensive discussion of this research direction in the related work section of our paper.

[1] AutoGuide: Automated Generation and Selection of State-Aware Guidelines for Large Language Model Agents

https://arxiv.org/pdf/2403.08978v1

[2] KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents

https://arxiv.org/pdf/2403.03101

Q2. Impact of depth in exploration

We appreciate the suggestion of an interesting experiment. We additionally conduct the experiments and provide the results in the general response.

We sincerely appreciate your detailed feedback, which is invaluable in improving our work.

Weaknesses

W1. Performance of WMA web agents on WebArena
We design the framework to be easily applied to various agent frameworks, and in the experiments on WebArena we applied our framework to the most basic web agent, Vanilla CoT to understand its fundamental effectiveness.

We also show that our approach also holds effectiveness with stronger agent frameworks (e.g., AWM) than Vanilla CoT as we show in Table 3. Thus, while we did not achieve state-of-the-art (SOTA) performance on WebArena, we think that we can expect some performance improvement when it is adapted to the SOTA method, such as AWM.

W2. Need clarification on the experimental setup
We provide the answers to your questions below. Thank you again for the detailed review and suggestions!

W3. In-depth analysis on the world model.

We conduct in-depth analyses to evaluate how well our world model predicts the next observation, focusing primarily on coverage. Coverage measures how much information from the ground-truth observation is successfully captured in the model's prediction.
We define coverage as the ratio of the information of the ground-truth observation covered by the predicted next observation. Specifically, the coverage score is calculated as:

• (# sentences in ground-truth observation covered by the predicted observation) / (# total sentences in ground-truth observation).

For evaluation, we employ an LLM-as-judge approach using GPT-4o as the judge LLM. We begin by separating both predicted and ground-truth observations into sentences. Then, we use an LLM to determine whether information from the target sentence can be found in the source text, which consists of a list of sentences. We run the evaluation on 100 samples used in our preliminary analysis and compare our world model with a few different LLMs, including GPT-4o-mini and GPT-4o.

We find that 42.99% of the information in the ground-truth observation is covered by our world model, and there is a significant gap between general LLMs that did not learn environment dynamics.

W4. Other ways of using world models in web navigation.
We thought that there are two major ways to improve agents' policies during inference time. The first way is to sample multiple action candidates and select the best action (it can be regarded as BoN sampling), which is used as our method, and the second way is to refine the predicted action using the feedback.

We have also explored the second way in Section 6.1. We run experiments on the Map domain from WebArena, and we find that the improvement is not that significant compared to the first way (WMA web agent: 22.3% and Self-refine w/ simulated feedback: 13.4%).

Clarifying questions

What is “policy optimization” as applied to Tree Search Agent (Koh et al, 2024). I thought Figure 3 bottom shows “policy optimization” which seems like it would only apply to World Model Agent.

Q1. We use “policy optimization” to incur a method that selects the best action among the sampled actions according to the value score on the future states visited during the exploration. We have changed the wording to “action selection” for better clarity.

---

Correspondingly, what is "no policy optimization"? Is that just CoT? If so, why are the results different? Why is Tree Search Agent better?

Q2. Yes, it is a setting that just uses CoT. We think that the results of CoT baseline with max action step 5 (average Success Rate 15.0%) from Tree Search Agent paper are too high, considering that CoT result with max action 30 is reported as 13.1% in the original WebArena paper, which is an easier setting as it allows more attempts.

---

Is value function trained with the world model data or is it a pure value function? If pure value function, how does that work with the abstracted observations?

Q3. To finetune the value function, we transformed the original full observation into an abstract observation to apply to our world model. We will prepare a separate section in the Appendix to elaborate on the training details of the value function.

---

Line 284: I don’t get this line and how the citation matches. “explore diverse next states st+1 ∈ S (Wang et al., 2022)”

Q4. We agree that this sentence can mislead the readers. We intended that a diverse exploration of the next states might help the agent to find the best path compared to sticking to a single path. We thought it was similar to generating diverse reasoning paths in Wang et al., 2022. We will revise this sentence to improve clarity. A revised sentence might be: "We begin by sampling $k$ action candidates $\{a_t^1, a_t^2, ..., a_t^k\}$ from $\theta$ via top-$p$ decoding, to conduct diverse exploration on future observations $\{o_{t+1}^1, o_{t+1}^2, ..., o_{t+1}^k\}$ similar to Koh et al. (2024)."

---

Table 5: The value function is fine-tuned. Is the Q-value function being fine-tuned as well and in the same way? If not, does not seem like a fair ablation. Also, is the Q-value function given ability to reason about what the action will do before outputting the value?

Q5. Yes, it is fine-tuned as well in the same way. Also, we trained a Q-value function to explicitly generate rationales toward the values of which labels are annotated in the same way as our value function.

Other Questions

Why is the user instruction included in the data? Shouldn't the world model be able to predict the changes based on the action without needing to know the intent?

Q6. Yes, by the definition of the world model, the user intent is not a necessary component of the input. However, in our framework, the goal of the world model is to provide the predicted future observation to the value function, so we thought that the description focused on the user intent might help the value function better understand the observation.

---

Table 2: Why not include the baselines from table 1? At least the tree search agent which is repeatedly compared to.

Q7. It was hard to reproduce Tree Search Agent, as it requires around $1600 to run 812 instances in WebArena even if we use the cheapest commercial VLM, GPT-4o-mini.

---

Figure 1 shows that commercial LLMs cannot predict the next state well by looking at actual observations. Perhaps they would be able to pick the correct abstracted transition-focused observation? Running that same experiment with the transition-focused observation version (ground truth, not predicted) could be interesting.

Q8. We additionally run preliminary analysis 1 (Figure 1) with the transition-focused observation as you suggested. We find that the overall performances have increased, but still there is a gap between human performance, 0.83 .

---

Which leads me to the following question, how do larger commercial LLMs fare at predicting the next abstract transition? It would be nice to see Table 5 row 2 with a few different base LLMs.

Q9. We run the experiment with a few different base LLMs and the results are in the table below. We find that the performance increases by using a stronger base LLM as the world model, but the ability to predict the next observation is still limited compared to our world model.

---

Why is HTML-T5 considered SOTA for Mind2Web? It seems like there are many much stronger algorithms presented in table 1. Mind2Web is also an offline evaluation dataset so does not account for different ways of completing a task or different action spaces.

Q10. While we acknowledge that T5 itself is a relatively outdated model, HTML-T5 has a distinct advantage: it was specifically pretrained to process HTML format as input, enhancing web agents' environmental perception. This is similar to the recently released UGround, which uses massive visual input training to improve perception.

The choice of baselines in Mind2Web is constrained by its offline nature. Many advanced baselines (such as AutoEval and Tree Search Agent) rely on verification/evaluation of environmental interaction outcomes, which isn't possible in Mind2Web's offline setting. This limitation led us to include AWM as an alternative baseline.

---

Q11,12,13 (other notes). Thank you for the suggestions. We will update our draft accordingly.

Dear reviewer mAfj,

Thank you for your efforts reviewing this paper. Can you please check the authors' responses and see if your concerns have been addressed? Please acknowledge you have read their responses. Thank you!

We appreciate your insightful review and acknowledgment of our contribution.

W1. In-depth analysis on the world model

We conducted a direct evaluation to evaluate how well our world model predicts the next observation, focusing primarily on coverage. Coverage measures how much information from the ground-truth observation is successfully captured in the model's prediction.
We define coverage as the ratio of the information of the ground-truth observation covered by the predicted next observation. Specifically, the coverage score is calculated as:

• (# sentences in ground-truth observation covered by the predicted observation) / (# total sentences in ground-truth observation).

For evaluation, we employ an LLM-as-judge approach and we use GPT-4o as the judge LLM. We begin by separating both predicted and ground-truth observations into sentences. Then, we use an LLM to determine whether information from the target sentence can be found in the source text, which consists of a list of sentences. We run the evaluation on 100 samples used in our preliminary analysis and compare our world model with a few different LLMs, including GPT-4o-mini and GPT-4o.

We find that 42.99% of the information in the ground-truth observation is covered by our world model, and there is a significant gap between general LLMs that did not learn environment dynamics.

W3 (Q2). Extending the world model to take multimodal input

Thank you for your suggestion. We highly agree that the incorporation of both text and visual features is crucial for successful web navigation. Thus, we expanded our world model to take multimodal input (i.e., text-based observation and image-based observation) and we provide the results in the general response.

W2-1 (Q3). Clarification on the experimental setup

We assume the different performance of the w/o policy optimization setting and the reported Vanilla CoT stems from the number of max action limits, which was 5 for w/o policy optimization (SR 12.8%) and 30 for Vanilla CoT (SR 13.1%). We follow the setting of Tree Search Agent which also uses the max action limits as 5. As mentioned in the Tree Search Agent paper, an action limit of 5 is a much more difficult setting for the web agents compared to a limit of 30.

W2-2. Future potential of WMA web agent

Through the experiments in Figure 6 and in the general response, we find that scaling up the budget for the exploration and extending the world model to be capable of processing multimodal input can significantly improve the performance of our method. Also, we believe that scaling can close the performance gap between the tree search agent and our method and we are working on the experiment. We will share the results as soon as we get them.

Q1. We agree with your concern. While we collect the exploration data with GPT-4o-mini, we find that the world model trained with the exploration data from GPT-4o-mini also works with GPT-4o agent as we show in Table 1 (a notable improvement from the Vanilla CoT method). Also, while the world model is trained with the Vanilla CoT agent’s trajectory, we show that our world model can be applied to other agent frameworks, such as SteP (in general response) and AWM.

Thank you for the positive review and valuable comments.

W1(Q2). Application of world model to multi-step planning

We agree that extending our framework to a multi-step setting is crucial for web navigation tasks since web navigation generally involves long-horizon planning. To explore this direction, we conduct an experiment and provide the results in the general response.

W2 (Q1). Application of world model to multimodal input

We have extended our world model to take not only text observation but also multimodal observation. These results are also added in the general comment.

W3 (Q3). Human evaluation on the quality of transition-focused observation abstraction

To assess the quality of transition-focused abstraction, we conducted a human evaluation using Amazon Mechanical Turk raters. We first ranked instances in the training set based on the magnitude of changes between consecutive observations ($\\Delta(o\_t, o\_{t+1})$) and divided them into four quantiles:

• (25%, 0%]: Largest changes between observations
• (50%, 25%]
• (75%, 50%]

$100$ : Smallest changes between observations

We randomly sampled 50 instances from each group and asked 3 different raters per each instance to evaluate how well the abstraction preserved important information using a 5-point Likert scale, where:

• Score 5: Fully covers all important information.
• Score 4: Covers most important information, with minor omissions.
• Score 3: Covers some important information, but leaves out some detail.
• Score 2: Covers very little information, with major omissions.
• Score 1: Lacks most of the important information.

| # lines of $\Delta(o_t, o_{t+1})$ | Top $100$ | Top (75%, 50%] | Top (50%,25%] | Top (25%,0%] | | :---- | :---- | :---- | :---- | :---- | | Score | 3.91 | 3.85 | 3.84 | 4.06 |

The human evaluation results demonstrate that our transition-focused abstraction strategy effectively preserves important information, even in highly dynamic situations. While we observe a slight decrease in scores as transitions become more dynamic in Top (75%, 50%] and Top (50%, 25%], the group with the largest changes (Top (25%, 0%]) still maintains a strong score close to 4, which means most of the important information is preserved. These results validate that our approach successfully captures essential information despite significant changes between observations.

W4 (Q4). How WMA web agents scale with larger action candidates

We conducted an additional analysis to examine how increasing the number of action candidates ($k$) affects performance and computational efficiency. Since we adopted the Tree Search Agent codebase for implementing action sampling logic, action candidates are sampled in the following manner:

1. Generate $max(20, k*2)$ output sequences.
2. Build a counter dictionary consisting of (action candidate, frequency) to remove identical actions and rank actions with their probability.
3. Select top-$k$ actions from the dictionary.

Thus, the API cost for sampling action remains unchanged until $k\<=10$.

The table below shows how the number of action candidates ($k$) affects both the performance and computational cost of the WMA web agent on the Map domain from WebArena. As mentioned above, API costs remain similar for $k≤10$ but increase when $k\>10$. Meanwhile, the inference time, which is primarily bounded by environment interaction time, shows only a gradual increase as $k$ increases.