Proposer-Agent-Evaluator (PAE): Autonomous Skill Discovery For Foundation Model Internet Agents

Q5: How did we choose the baselines of open-source VLMs?

Again, we chose LLaVa Next series as they had the best performance on VisualWebBench (https://visualwebbench.github.io/) when main experiments were conducted. InternVL-2.5-Xcomposer and Qwen2VL are latest open source VLM releases around the time of the submission of this paper so they are also added to the paper.

Q6: In the alignment with human judgements, could you elaborate on how many human annotators were included in the user study and if possible how many hours they allocated to the task?

Five annotators participate in the user study and the entire annotation process takes around 40 annotator hours with the help of a designated user interface.

Q7: In the main results, under the generalization paragraph, line 405, why is LLaVa-7B PAE highlighted, when actually you present results for the larger LLaVa-34B PAE as well in Table 3? Also in Table 3, why did you drop some of the other models from Tables 1 and 2 (Claude 3.5, InternVL and vanilla LLaVa-7/34B)?

We bold the results of LLaVa-7B PAE to highlight the improvement of PAE compared to SFT on both the scale of 7B and 34B. The omission of the baselines in Table 3 that are not main points of comparisons is due to budgetary constraints. We are working on to get more compute and API credits to include those results in the final version of the paper.

Q8: Typos.

Thanks so much for catching the typos! We have fixed them in blue in the updated pdf.

Response to Ethics concerns.

In our experiments, all user demos are collected by the authors instead of actual users so there should be no permission/privacy concerns.

Thank you for your review and positive feedback on the paper! Please let us know if your questions are addressed. We are happy to discuss further. We answer your questions below:

Q1: It would be good to elaborate on why close-source models outperform all open-source models.

We have provided Failure Mode analysis in Figure 5 of the paper. In particular, the most outstanding failure mode that Claude Sonnet 3.5 reduces compared to other open-source models is low-level web navigation skills. We suspect that this might result from the training corpus of proprietary models containing more and better web-related data and observations in the form of set-of-marks.

Q2: It would be good to add more details on inference time analysis.

After the training process of PAE, the model directly outputs its thoughts and actions at each step without incurring any additional inference-time cost. As shown in the table below, in WebArena Easy, we found that the thoughts after PAE training got more accurate but not longer so PAE uses around the same inference-time analysis compared to the SFT policy as measured by the number of tokens in the action per step.

Q3: How is the reward defined for the action policy?

PAE makes use of an image-based outcome evaluator that evaluates the success of each trajectory from the last three screenshots and the answer from the agent. The outcome-based evaluator provides a reward of 1 at the end of the trajectory if the task is considered successful and 0 otherwise. Rewards elsewhere are all set to be 0. The use of an outcome-based only evaluator is actually one of our design choices that we have found to be able to take best advantage of the asymmetric capabilities of SOTA VLMs as evaluators and agents in web agent tasks (additional experiment results on this are provided in blue in Section 6 Discussions). In our preliminary experiments, we found that using VLMs to provide dense annotations such as per-step rewards are less robust due to hallucinations. We are re-running those experiments and will update these results once they are done.

Q4: Why did we choose Claude Sonnet 3 and LLaVa Next in our components?

We chose Claude Sonnet 3 because we happen to have credits for using this model but in principle any VLM that can reason across multiple images can be used for task proposers and evaluators. We have also provided additional results of using only open-source models such as Qwen2VL-7B and Qwen2VL-72B for proposing tasks and conducting evaluations, thereby eliminating the need of close-source models. We have updated the results in Section 6 discussions of the paper and also provide the table of most relevant results below for your convenience:

We chose LLaVa Next models because they were the best open-source VLMs on VisualWebBench (https://visualwebbench.github.io/) at the time when main experiments were conducted. However, more capable open-source VLMs such as Qwen2VL are also becoming available recently.

Q5: Comparisons with other trained web agents are missing.

According to a recent VLM agent benchmark (https://github.com/THUDM/VisualAgentBench/tree/main?tab=readme-ov-file#leaderboard), there are no other trained generalist open-source VLM web agents that can be applied directly without task-specific fine-tuning. We are happy to add additional baselines of such trained web agents, if they are open-source. Thus, another significant contribution of this work is to release the medium-size sota open-source checkpoints and large-scale dataset for generalist VLM web agents. As mentioned by Reviewer wpvf, the release of our sota open-source checkpoint and large-scale dataset can facilitate RL and fine-tuning research in the VLM web agent domain, while previously the community has to rely on very large models such as Qwen72B or proprietary models. We will also release our training infrastructure that supports efficient large-scale interactions with thousands of distributed web browsers.

Q6: The performance of other cutting-edge VLMs such as GPT4o is missing.

We would like to mention that the performance of GPT4o is orthogonal to the contribution of PAE as it is built upon a significantly more capable base model. We did not include GPT4o results because of practical constraints on OpenAI API credits and rate limits at the time of paper submission, but we are working on acquiring such credits to include the performance of GPT4o in the final paper.

Q7: Several typos

Thanks for pointing out those typos. We have corrected them in blue in the updated pdf.

[1] Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open-endedness via models of human notions of interestingness, 2024.

[2] Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code, 2024.

[3] Cedric Colas, Laetitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan, and Marc-Alexandre Cote. Augmenting autotelic agents with large language models, 2023.

[4] C´edric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Cl´ement Moulin-Frier, Peter Ford Dominey, and Pierre-Yves Oudeyer. Language as a cognitive tool to imagine goals in curiosity-driven exploration, 2020

[5] Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J. Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance, 2023b.

[6] Yuqing Du, Olivia Watkins, Zihan Wang, Cedric Colas, Trevor Darrell, Pieter Abbeel, Abhishek ´ Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models, 2023.

Open-source contributions: Another significant contribution of this work is to release the medium-size sota open-source checkpoints and large-scale dataset for generalist VLM web agents. As mentioned by Reviewer wpvf, the release of our sota open-source checkpoint and large-scale dataset can facilitate RL and fine-tuning research in the VLM web agent domain, while previously the community has to rely on very large models such as Qwen72B or proprietary models. We will also release our training infrastructure that supports efficient large-scale interactions with thousands of distributed web browsers.

Q2: Unclear why tasks proposed by context-aware VLMs can approximate real-world task distribution

While an VLM-based task proposer cannot perfectly approximate real-world task distributions, they are much cheaper and more scalable than asking humans to annotate tasks. However, as discussed above, careful design choices must be made so that the skills the agents acquire through synthetic tasks can generalize to the distribution of real-world tasks. We provide an additional ablation result showing the effectiveness of our design decisions of an additional reasoning step before operation actions.

In addition to the system-level design decisions and open-source contributions, PAE is also different from the prior literature in its incorporation of an additional reasoning step before taking the actions. Only the action parts determine the actual web operations but the reasoning traces are also included in the filtered BC loss. As shown in the above table and updated as Figure 6 in the pdf, while the agent can still learn diverse skills to a similar level on the proposed tasks without reasoning step, its generalization to unseen human-written tasks has significantly declined. This shows an example of the important difference of learning skills alone and generalization to unseen human-written tasks.

Q3: Is LLaVa-SFT trained on the same trajectories as LLaVa-PAE?

There might be an important misunderstanding of our approach, and we would like to clarify LLaVa-SFT and LLaVa-PAE use fundamentally different data. LLaVa-SFT is trained on trajectory demonstrations from Claude 3 Sonnet following common approaches so that it can have a non-trivial performance as web agents (better than 0.9% when LLaVa is used zeroshot as shown in Table 1 in the paper) to enter an autonomous RL loop of PAE. LLaVa-PAE is trained on top of with entirely self-generated trajectories. With additional experiments shown in the answer to the above question, the PAE training loop can be entirely independent of close-source models, leading to an entirely autonomous skill discovery system that can generalize to unseen real user tasks. This working autonomous skill discovery system is the novel contribution of this work.

Q4: Behavior Cloning successful trajectories labeled by a VLM should not be described as RL. And why is behavior cloning adopted instead of other training methods such as SFT?

In this work, filtered behavior cloning serves as a policy optimization technique for the rewards provided by the autonomous evaluator in an online RL loop where we iterate between rolling out the current policy to collect data and perform policy optimization. Importantly, the use of behavior cloning loss on successful trajectories as the policy optimization method does not alter the fact that our primary workflow adheres to an online reinforcement learning paradigm. However, we understand whether filtered behavior cloning should be considered as RL can be ambiguous. As such, we have updated the pdf in Section 4.4 to state Filtered Behavior Cloning as a policy optimization method instead of an RL method. It is important to note that PAE adopts an online RL paradigm instead of only SFT on the pre-collected successful trajectories.

To reply to your question on why behavior cloning is adopted instead of other training methods such as SFT: there might be a terminology difference between people from the RL community and the foundation model community. We use online Filtered Behavior Cloning to mean negative log likelihood on the successful trajectories collected online, which is the same as SFT on the successful trajectories collected online.

Thank you for your review and feedback on the paper. To address the raised questions, we have provided additional experiment results using only open-source models as task proposers and evaluators, showing that the performance of PAE does not depend on proprietary VLMs. We have also updated the pdf with major rewritings based your suggestions and frame the novel contribution of this work as a working system that can successfully generalize autonomous skill discovery to real-world human requests as opposed to the novelty of individual components. Please let us know if your concerns are addressed. We are happy to discuss further. We answer your questions below:

Q1: The method itself lacks novelty and has to rely on proprietary VLMs for instructions and successful trajectories.

First of all, we would like to clarify that PAE iteratively improves the agent models using self-generated trajectories instead of successful trajectories from proprietary VLMs. The design principle of PAE is to take the best advantage of the asymmetric capabilities of SOTA VLMs in terms of agent and proposer/evaluator. Intuitively, VLMs are very good at confirming whether a specific product has been added to the shopping cart (e.g. by looking at the final screenshot to see if the shopping cart contains the product), while less good at actually navigating the web to find the product and add it to the cart. To show that the improvement of PAE does not depend on close-source models. We have provided additional results of using only open-source models such as Qwen2VL-7B and Qwen2VL-72B for proposing tasks and conducting evaluations, thereby eliminating the need of close-source models. We have updated the results in Section 6 discussions of the paper and also provide the table of most relevant results below for your convenience:

To clarify more on the novelty concern, the novel contribution of this work is a system that can learn to solve real web agent tasks by using autonomous skill proposals and reinforcement learning. While a wide range of skill discovery with LLM task proposers have been proposed in the literature, such works typically focus on learning diverse skills on relatively simplified environments such as games[1, 2, 3, 4] and robotics tasks with a limited number of scenes [5,6]. This work takes an important step forward to study when those skills can generalize to human requests in realistic benchmarks in the context of web agents and what the best design choices are for such generalization., including an agent policy with additional reasoning steps, context-aware task proposer, and image-based outcome evaluator. While each individual component of this work is not novel, the novel contribution of this paper is to put up the first working system for such an unsupervised skill discovery procedure to generalize to unseen real-world benchmarks with human requests with the right design decisions and present extensive experiment analysis to the effectiveness of this framework (as recognized by reviewers uKzp, 4H8g, vLaT and wpvf). We have included discussions on all your cited papers and additional works along this line of autonomous Rl and task proposers in the related works.

Dear reviewer vLaT,

Thanks so much for providing your feedback, and we would like to let you know that we have gathered more evidence showing that the improvement of PAE stems from taking advantage of the asymmetric capabilities of SOTA VLMs instead of copying stronger proprietary models. We have carried out additional experiments and updated the main results Table 1, Table 2, and Table 3 with additional results in red using open source models as proposers and evaluators. As the results in Table 2 (of WebArena) are already included in the previous rebuttal, we mainly summarize the new results from Table 1 and Table 3 (of WebVoyager and unseen websites) in the table below:

Similar to the conclusion of the open-source experiments on WebArena presented in the previous rebuttal, our new results show more empirical evidence that the improvement of PAE is not bounded by the agent capability of SOTA VLMs where even Qwen2VL (with agent performance of only 1.4%) can be used as task proposers and evaluators to improve the performance of LLaVa7B-SFT from 14.9% to 21.7% on seen websites and from 9.1% to 13.7% on unseen websites. We believe that these results strongly validate our claim that the improvements of PAE stem from the asymmetric capabilities of SOTA VLMs as agents and as proposers/evaluators, instead of copying a stronger model. While our experiments using Qwen2VL-72B are still running, we expect it to yield even better results and will update it to the paper once it is ready.

Please let us know if the additional results have addressed your concerns and we are happy to engage in further discussions.

Thank you for your review and the recognition that “experiments are extensive” and “paper are easy to follow”. To address the raised questions, we provided more clarifications in terms of the novel contribution of the paper, and significant rewritten the relevant sections in paper in blue as suggested, and provided additional ablation experiments in the experiments section to support our contribution.

Q1: What is the difference of PAE with other methods for autonomous RL and task proposers.

The novel contribution of this work is a complete working system that can learn to solve real web agent tasks by using autonomous skill proposals and reinforcement learning. While a wide range of skill discovery with LLM task proposers have been proposed in the literature, such works typically focus on learning diverse skills on relatively simplified environments such as games[1, 2, 3, 4] and robotics tasks with a limited number of scenes [5,6]. This work takes an important step forward to study when those skills can generalize to human requests in realistic benchmarks in the context of web agents and what the best design choices are for such generalization., including an agent policy with additional reasoning steps, context-aware task proposer, and image-based outcome evaluator. While each individual component of this work is not novel, the novel contribution of this paper is to put up the first working system for such an unsupervised skill discovery procedure to generalize to unseen real-world benchmarks with human requests with the right design decisions and present extensive experiment analysis to the effectiveness of this framework (as recognized by reviewers uKzp, 4H8g, vLaT and wpvf).

Open-source contributions Another significant contribution of this work is to release the medium-size sota open-source checkpoints and large-scale dataset for generalist VLM web agents. As mentioned by Reviewer wpvf, the release of our sota open-source checkpoint and large-scale dataset can facilitate RL and fine-tuning research in the VLM web agent domain, while previously the community has to rely on very large models such as Qwen72B or proprietary models. We will also release our training infrastructure that supports efficient large-scale interactions with thousands of distributed web browsers.

Q2: Framing the algorithmic novelty can be done with more clarity.

Thanks for your suggestions on adding more comparisons with prior works along the line of OMNI, and concrete suggestions on phrasing the novelty as building a complete working autonomous skill discovery system that generalizes to real-world web agent applications. To better situate the contribution of our paper as the first working system for autonomous skill discovery that generalizes to real web agent requests instead of the novelty of individual components, we have adjusted the related claims in the paper (and colored in blue). Concretely, we have rewritten the introduction to focus on our contribution as a working autonomous skill discovery system for realistic web agent tasks, added an additional discussion of the difference of PAE from prior works on autonomous RL and task proposers in the related works section, rephrased the technical section to take more about the specific design decisions that we have made for VLM web agents, and added additional ablation results in the experiment section.

[1] Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open-endedness via models of human notions of interestingness, 2024.

[2] Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code, 2024.

[3] Cedric Colas, Laetitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan, and Marc-Alexandre Cote. Augmenting autotelic agents with large language models, 2023.

[4] C´edric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Cl´ement Moulin-Frier, Peter Ford Dominey, and Pierre-Yves Oudeyer. Language as a cognitive tool to imagine goals in curiosity-driven exploration, 2020

[5] Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J. Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance, 2023b.

[6] Yuqing Du, Olivia Watkins, Zihan Wang, Cedric Colas, Trevor Darrell, Pieter Abbeel, Abhishek ´ Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models, 2023.

[1] Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open-endedness via models of human notions of interestingness, 2024.

[2] Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code, 2024.

[3] Cedric Colas, Laetitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan, and Marc-Alexandre Cote. Augmenting autotelic agents with large language models, 2023.

[4] C´edric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Cl´ement Moulin-Frier, Peter Ford Dominey, and Pierre-Yves Oudeyer. Language as a cognitive tool to imagine goals in curiosity-driven exploration, 2020

[5] Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J. Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance, 2023b.

[6] Yuqing Du, Olivia Watkins, Zihan Wang, Cedric Colas, Trevor Darrell, Pieter Abbeel, Abhishek ´ Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models, 2023.

Q2: References and relations to prior works on autonomous RL and task proposals.

The novel contribution of this work is a system that can learn to solve real web agent tasks by using autonomous skill proposals and reinforcement learning. While a wide range of skill discovery with LLM task proposers have been proposed in the literature, such works typically focus on learning diverse skills on relatively simplified environments such as games[1, 2, 3, 4] and robotics tasks with a limited number of scenes [5,6]. This work takes an important step forward to study when those skills can generalize to human requests in realistic benchmarks in the context of web agents and what the best design choices are for such generalization., including an agent policy with additional reasoning steps, context-aware task proposer, and image-based outcome evaluator. While each individual component of this work is not novel, the novel contribution of this paper is to put up the first working system for such an unsupervised skill discovery procedure to generalize to unseen real-world benchmarks with human requests with the right design decisions and present extensive experiment analysis to the effectiveness of this framework (as recognized by reviewers uKzp, 4H8g, vLaT and wpvf). We have included discussions on all your cited papers and additional works along this line of autonomous Rl and task proposers in the related works.

Methodological difference from prior works in autonomous RL and task proposers.

In addition to the system-level design decisions and open-source contributions, PAE is also different from the prior literature in its incorporation of an additional reasoning step before taking the actions. Only the action parts determine the actual web operations but the reasoning traces are also included in the filtered BC loss. As shown in the above table and updated as Figure 6 in the pdf, while the agent can still learn diverse skills to a similar level on the proposed tasks without reasoning step, its generalization to unseen human-written tasks has significantly declined. This shows an example of the important difference of learning skills alone in prior works and generalization to unseen human-written tasks.

Q3: The writing of the paper can be improved based on the rationale of generalization and better terminologies.

To better situate the contribution of our paper as the first working system for autonomous skill discovery that generalizes to real web agent requests, we have adjusted the related claims in the paper (and colored in blue). Concretely, we have rewritten the introduction to focus on our contribution as a working autonomous skill discovery system for realistic web agent tasks, added an additional discussion of the difference of PAE from prior works on autonomous RL and task proposers in the related works section, rephrased the technical section to take more about the specific design decisions that we have made for VLM web agents, and added additional ablation results in the experiment section.

We have also updated Section 3.1 to emphasize that the ground-truth reward functions are only accessible for evaluation. We have also updated the terminology of “model-based” to be “VLM-based”.

Q4: Why sample tasks uniformly rather than prioritizing tasks by difficulty or learning progress.

In our experiments presented in Figure 6, we have considered the possibility of using user demos as contexts to inspire the task proposer to come up with more feasible tasks (similar to the “model of interestingness” as done in OMNI). Indeed, we have found that this prioritization of more feasible tasks can accelerate the learning process and lead to policies that generalize better to human-annotated tasks.

Thank you for your review and feedback on the paper. To address the raised questions, we have provided additional experiment results using only open-source models as task proposers and evaluators, showing that the SOTA performance of PAE is not bottlenecked by close-source models. We have also updated the pdf with major rewritings based on your suggested references to prior works, additional experiments, and the rationale of the novel contribution as a working system that can successfully generalize autonomous skill discovery to real-world human requests as opposed to the novelty of individual components.

Please let us know if your concerns are addressed. We are happy to discuss further. We answer your questions below:

Q1: The novelty and motivation of the paper are limited because it is bottlenecked by the dependence on close-source models.

The design principle of PAE is to take the best advantage of the asymmetric capabilities of SOTA VLMs in terms of agent and proposer/evaluator for web agents, instead of imitating a more capable VLM. Intuitively, VLMs are very good at confirming whether a specific product has been added to the shopping cart (e.g. by looking at the final screenshot to see if the shopping cart contains the product), while less good at actually navigating the web to find the product and add it to the cart. To show that the improvement of PAE is not bottlenecked by the dependence on close-source models. We have provided additional results of using only open-source models such as Qwen2VL-7B and Qwen2VL-72B for proposing tasks and conducting evaluations, thereby eliminating the need of close-source models. We have updated the results in Section 6 discussions of the paper and also provide the table of most relevant results below for your convenience:

Open-source contributions: Another significant contribution of this work is to release the medium-size sota open-source checkpoints and large-scale dataset for generalist VLM web agents. As mentioned by Reviewer wpvf, the release of our sota open-source checkpoint and large-scale dataset can facilitate RL and fine-tuning research in the VLM web agent domain, while previously the community has to rely on very large models such as Qwen72B or proprietary models. We will also release our training infrastructure that supports efficient large-scale interactions with thousands of distributed web browsers.

I thank the authors for the new experiments and clarifications on how this work is different from previous work. In response, I have increased my score.

new results using open-sourced models

Why not include the results on the open-source models in tables 1, 2, and 3? I think this is a valuable comparison to see to understand more deeply if the increase in performance by PAE is bounded by what the sota VLM is.

Minor things:

“generalize to the ground-truth task distribution and reward function only used as evaluations”

should be “which are only used for evaluations”

Dear reviewer SFbJ,

Thanks so much for providing your feedback, and additional suggestions for improving the paper to show more evidence of PAE not bounded by the use of proprietary models. To follow your suggestions, we have carried out additional experiments and updated the main results Table 1, Table 2, and Table 3 with additional results using open source models as proposers and evaluators. We have also updated the main results Section 5 with an additional discussion on whether PAE is bounded by SOTA VLMs in red. As the results in Table 2 (of WebArena) are already included in the previous rebuttal, we mainly summarize the new results from Table 1 and Table 3 (of WebVoyager and unseen websites) in the table below:

Similar to the conclusion of the open-source experiments on WebArena presented in the previous rebuttal, our new results show more empirical evidence that the improvement of PAE is not bounded by the agent capability of SOTA VLMs where even Qwen2VL (with agent performance of only 1.4%) can be used as task proposers and evaluators to improve the performance of LLaVa7B-SFT from 14.9% to 21.7% on seen websites and from 9.1% to 13.7% on unseen websites. We believe that these results strongly validate our claim that the improvements of PAE stem from the asymmetric capabilities of SOTA VLMs as agents and as proposers/evaluators, instead of copying a stronger model. While our experiments using Qwen2VL-72B are still running, we expect it to yield even better results and will update it to the paper once it is ready.

Finally, to provide even more justifications on how PAE makes the best design choices to take advantage of the asymmetric capabilities of sota VLMs, we compare the outcome-based evaluation included in PAE with other choices of evaluators in the related literature such as step-based evaluators and function-based evaluators in the table below:

In our implementation of the step-based evaluator, we ask Claude 3 Sonnet to evaluate whether each step is correct or not (i.e. whether it gets the agent closer to the goal) and behavior clone all the steps considered correct by the step-based evaluator. In our implementation of function-based evaluator, we provide 3 examples of verification functions as used by WebArena and ask Claude 3 Sonnet to also come up with verification functions to functionally verify the final task success rate (e.g. checking if the final website url is the same as the ground-truth url). As shown in Figure 3, both step-based evaluator and function-based evaluator perform worse than the outcome-based evaluator, where the use of step-based evaluator even leads to a worse performance compared to the SFT checkpoint to start with. We found that the step-based evaluator hallucinated more often and tended to be too ''generous'' in terms of considering the success of each step, potentially because the task of evaluating the success of each step is significantly harder compared to only evaluating the success of the final outcome. Furthermore, we found that oftentimes the function-based evaluator hallucinates on the success criterion for the verification function (e.g. making up a non-existing url that the agent needs to go to), therefore resulting in most tasks being impossible to learn. In contrast, the design choice of an outcome based evaluator can take best advantage of the asymmetric capability of the SOTA VLMs as agents and as evaluators, and best provide reliable reward signals for the policy to improve, resulting in better performances.

We have also fixed the minor grammer issue that you pointed out. We appreciate your help in improving the paper and would love to know if the additional results have addressed your concerns. We are more than happy to engage in further discussions.

Rewriting of the paper: To better situate the contribution of our paper as the first working system for autonomous skill discovery that generalizes to real web agent requests instead of the novelty of individual components, we have adjusted the related claims in the paper (colored in blue). Concretely, we have rewritten the introduction to focus on our contribution as a working autonomous skill discovery system for realistic web agent tasks, added an additional discussion of the difference of PAE from prior works on autonomous RL and task proposers in the related works section, rephrased the technical section to take more about the specific design decisions that we have made for VLM web agents, and added additional ablation results on our design decisions in the experiment section.

Q2: Will the author build a large-scale dataset that can be released to the public?

Yes, another significant contribution of this work is to release the medium-size sota open-source checkpoints and large-scale dataset for generalist VLM web agents. As mentioned by Reviewer wpvf, the release of our sota open-source checkpoint and large-scale dataset can facilitate RL and fine-tuning research in the VLM web agent domain, while previously the community has to rely on very large models such as Qwen72B or proprietary models. We will also release our training infrastructure that supports efficient large-scale interactions with thousands of distributed web browsers.

Q3: Will the batch RL be affected since only parts of the data have rewards?

It is important to note that our PAE is an online framework where the agent periodically interacts with the environment and performs policy optimizations, instead of batch RL/offline RL setting where the agent performs policy optimizations in an offline setting. As shown in Figure 6 in the new pdf, the requirement of being able to interact with the environment to collect new on-policy data is essential to the steady improvement of the agent without the pessimism issues of batch/offline RL[7]. The use of an outcome only evaluator is actually one of our design choices that we have found to be able to take advantage of the asymmetric capabilities of SOTA VLMs as evaluators and agents in web agent tasks (addition experiment results on this are provided in blue in Section 6 Discussions). In our preliminary experiments, we found that using VLMs to provide dense annotations such as per-step rewards (where we use VLM to annotate whether each action takes the agent closer to the goal and behavior clone all the actions that bring the agent closer to the goal) are less robust due to hallucinations. We are re-running those experiments and will update these results once they are done.

[1] Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open-endedness via models of human notions of interestingness, 2024.

[2] Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code, 2024.

[3] Cedric Colas, Laetitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan, and Marc-Alexandre Cote. Augmenting autotelic agents with large language models, 2023.

[4] C´edric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Cl´ement Moulin-Frier, Peter Ford Dominey, and Pierre-Yves Oudeyer. Language as a cognitive tool to imagine goals in curiosity-driven exploration, 2020

[5] Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J. Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance, 2023b.

[6] Yuqing Du, Olivia Watkins, Zihan Wang, Cedric Colas, Trevor Darrell, Pieter Abbeel, Abhishek ´ Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models, 2023.

[7] Kumar, Aviral, et al. ‘Conservative Q-Learning for Offline Reinforcement Learning’. arXiv [Cs.LG], 2020, http://arxiv.org/abs/2006.04779. arXiv.

Thank you for your review and the recognition that “experiments are extensive” and “paper are easy to follow”. To address the raised questions, we provided more clarifications in terms of the novel contribution of the paper, and significant rewritten the relevant sections in paper in blue as suggested, and provided additional ablation experiments results of PAE with open-source models alone to show that PAE is not simply imitating the data generated from a stronger VLM. We are also currently running the experiments comparing our outcome-based evaluator with a dense step-based evaluator and will provide experiments as soon as they have finished. Please kindly let us know if your concerns are addressed or if further clarifications and experiments are required. We are happy to discuss further. We answer your questions below:

Q1: The main issue is still the novelty. Utilizing existing foundation models to process data or automatically generate data is not novel. Using RL to finetune the model is not novel either.

The novel contribution of this work is a complete working system that can learn to solve real web agent tasks by using autonomous skill proposals and reinforcement learning. While a wide range of skill discovery with LLM task proposers have been proposed in the literature, such works typically focus on learning diverse skills on relatively simplified environments such as games[1, 2, 3, 4] and robotics tasks with a limited number of scenes [5,6]. This work takes an important step forward to study when those skills can generalize to human requests in realistic benchmarks in the context of web agents and what the best design choices are for such generalization., including an agent policy with additional reasoning steps, context-aware task proposer, and image-based outcome evaluator. While each individual component of this work is not novel, the novel contribution of this paper is to put up the first working system for such an unsupervised skill discovery procedure to generalize to unseen real-world benchmarks with human requests with the right design decisions and present extensive experiment analysis to the effectiveness of this framework (as recognized by reviewers uKzp, 4H8g, vLaT and wpvf).

To give more context, the design principle of PAE is to take the best advantage of the asymmetric capabilities of SOTA VLMs in terms of agent and proposer/evaluator for web agents, instead of imitating a more capable VLM. Intuitively, VLMs are very good at confirming whether a specific product has been added to the shopping cart (e.g. by looking at the final screenshot to see if the shopping cart contains the product), while less good at actually navigating the web to find the product and add it to the cart. To show that the improvement of PAE is not bottlenecked by the dependence on close-source models, we have provided additional results of using only open-source models such as Qwen2VL-7B and Qwen2VL-72B for proposing tasks and conducting evaluations, thereby eliminating the need of close-source models. We have updated the results in Section 6 discussions of the paper and also provide the table of most relevant results below for your convenience:

Dear reviewer uKzp,

Thanks for your efforts in reviewing the paper. We have got additional experiment results included in the paper (Figure 3 and discussions in Section 6) with respect to the effect of different design choices of evaluators. We would appreciate it if you could kindly suggest if these new experiment results and clarifications above have addressed your concerns. We are happy to engage in further discussions.

As shown in the table above, we compare the outcome-based evaluation included in PAE with other choices of evaluators in the related literature such as step-based evaluators and function-based evaluators. In our implementation of the step-based evaluator, we ask Claude 3 Sonnet to evaluate whether each step is correct or not (i.e. whether it gets the agent closer to the goal) and behavior clone all the steps considered correct by the step-based evaluator. In our implementation of function-based evaluator, we provide 3 examples of verification functions as used by WebArena and ask Claude 3 Sonnet to also come up with verification functions to functionally verify the final task success rate (e.g. checking if the final website url is the same as the ground-truth url). As shown in Figure 3, both step-based evaluator and function-based evaluator perform worse than the outcome-based evaluator, where the use of step-based evaluator even leads to a worse performance compared to the SFT checkpoint to start with. We found that the step-based evaluator hallucinated more often and tended to be too ''generous'' in terms of considering the success of each step, potentially because the task of evaluating the success of each step is significantly harder compared to only evaluating the success of the final outcome. Furthermore, we found that oftentimes the function-based evaluator hallucinates on the success criterion for the verification function (e.g. making up a non-existing url that the agent needs to go to), therefore resulting in most tasks being impossible to learn. In contrast, the design choice of an outcome based evaluator can take best advantage of the asymmetric capability of the SOTA VLMs as agents and as evaluators, and best provide reliable reward signals for the policy to improve, resulting in better performances.