VeSX: A Framework Featured by Verification, Self-Correction and In-context Learning for Web Automation Tasks

Q1.1: How would the system perform in a real-world scenario with unstructured tasks and environments? Have you considered conducting a user study or statistical analysis to test its practical application and effectiveness?

AQ1.1: See AW1.1.

Q2.1: While you provide an ablation study, could you expand on the evaluation of the self-correction feature? How does it handle various failure cases, and what are the associated costs (in terms of time, resources, etc.) for error correction?

AQ2.1: See AW2.1 about the evaluation of the self-correction feature.

In terms of computational costs, we supplemented some experiments for analysis. The extra computational cost caused by error correction mainly lies in two aspects: workflow design, and the more action steps.

1. Workflow Design: From the experiments, the verification, reflection and replanning modules account for approximately 10.7% of the input tokens and 16.1% of the output tokens. This indicates that the three main modules in VeSX workflow, verification, reflection, and replanning, do not introduce significant more computational cost.

2. More Action Steps: We compared our method with the Sodhi et al. (2024) approach, which uses human-labeled subtasks to guide the LLM in breaking down and solving tasks. Their average action step is 9.1, while our average action step is 13.1. From this perspective, VeSX requires approximately 43.9% more actions compared to human-guided workflows to achieve competitive results.

Due to the significant time and cost involved in reproducing other approaches, we do not fully reproduce all of them. Compared to the sota method with human guidance, VeSX takes about ~50% more computational cost to achieve competitive results totally. It is nearly impossible for other methods to achieve the same accuracy by 1.5x scaling up without other design. The details of this experiments can be found in Appendix B.1.

Q3.1: Beyond its application focus, do you see any potential for extending the technical contributions of VeSX? For example, could the subgoal-guided verification or hierarchical self-correction be applied in other domains?

AQ3.1: Yes, the verification scheme, the self-correction approach, and the construction process of the exemplar bank used in VeSX may have the potential to be effective in other tasks. In simple terms, we believe that VeSX will be useful in complex tasks that require multi-step reasoning and execution. We have added a new section, Section 5, to discuss about the current limitations of VeSX and potential directions for future expansion. We will also consider leaving this part for our further studies.

Q4.1: PS: content under Section 2.1 Overview is missing.

AQ4.1: Thank you for your correction! The missing section 2.1 has now been added.

We sincerely thank the reviewer for reading our paper and providing valuable insights.

W1.1 The system is more focused on practical application rather than introducing a novel method or algorithm. Additionally, there is no follow-up evaluation of the entire system under real-world conditions. It would be beneficial to see both quantitative and qualitative analyses of VeSX in real-world usage scenarios to better understand its performance in practical settings. E.g., a statistical evaluation or user study focusing on whether this system truly works for real-world tasks would significantly strengthen the paper. A field study or feedback from real users would also provide practical insights into how the system performs in dynamic, unstructured environments.

AW 1.1: We invited ten volunteers and asked them to experience our pipeline. User feedback indicates that simple user needs can be fulfilled in VeSX, but the total execution time is long. We show some cases in Appendix A.1.2 and Appendix C, which are close to users' daily require when surfing on the web.

W2.1: The concept of "self-correction" is promising, but the evaluation of this feature is not comprehensive enough. Although the paper includes an ablation study, a more detailed analysis of the self-correction mechanism is necessary to demonstrate its effectiveness. For example, breaking down how self-correction functions in different failure cases or assessing the time and resource costs associated with error correction would provide deeper insights into the feature’s utility.

AW2.1: We provide some case studies in Appendix C to analyze the effect of the self-correction mechanism. We also conduct experiments to calculate the computational cost of self-correction in Appendix B.1. The verification and self-correction module account for 10.7% of input tokens and 16.1% of output tokens.

W3.1: The paper does not address what happens if the system or specific components fail. While self-correction is included, there is no discussion of how the system handles scenarios where self-correction or verification mechanisms fail. For real-world applications, understanding the system’s resilience and fallback options is crucial. Including an analysis of fail-safe protocols would enhance the system’s reliability and robustness.

AW3.1: Understanding the robustness of the system is important for practical applications. Currently, we have set a maximum number of actions for the entire process to prevent VeSX from getting stuck in an infinite loop. This setting is taken by many related works. Furthermore, we have some designs for resilience except for the self-correction module:

1. We set a limit to the count of actions within one subtask. When the limit is reached, the subtask is considered as infeasible and the replanning module will be triggered directly.

2. We make the reflection module to check whether there is an error in verification to avoid the false negative caused by verification.

3. The verification module often be constructed with multiple functions to avoid over rely on one function's result.

We make some analysis of each modules to see how often they make mistakes, which can be found in Appendix B.2.

The robustness of the system is a critical yet challenging issue, and we find it difficult to resolve comprehensively in a short period. We hope our experiments can clarify your concerns and provide guidance for further improving the system's stability in future research and engineering efforts.

We sincerely thank the reviewer for reading our paper and providing valuable insights.

W1.1: The literature review on LLM-based agents appears incomplete, missing several relevant recent works

AW1.1: Thank you for your valuable advice. We add a new paragraph to talk about the recent research on LLM-based agents, which can be found in Section 4.1.

W2.1: About "subgoal-based verification," process supervision is a well-studied research direction[1]. This paper's key difference lies in the hierarchical verification mechanism. However, to prove the effectiveness of hierarchical verification，more comparison experiments and discussions should be made

AW2.1: It is an interesting and valuable question! In fact, process rewards were our initial exploration direction during the research, while hierarchical verification and error correction were later discovered paths. We found that step-by-step verification is difficult to implement in such scenarios and has low error correction efficiency. Previous verification methods were mostly used in mathematical tasks, and there has been little work related to LLM agents that require interaction with the environment. During our exploration, we identified two main reasons:

1. First, there is inherently a lack of a verification target in such scenarios. In mathematical tasks, the verification target is often whether a particular expression is incorrect, but in the agent’s environment, there is no clear right or wrong verification target. To address this issue, we proposed the idea of subgoal-guided verification and a scheme for generating expectations while predicting actions.

2. Second, the granularity of step-by-step verification is not suitable for capturing errors in agent tasks. In many cases, the errors are caused by actions taken several steps earlier. Step-by-step verification only ensures current stability and does not involve error correction functionality, which is why we designed hierarchical self-correction.

We did not provide such comparative experiments because there is currently no good solution for step-by-step verification in web automation. In fact, about 56% of our subtasks involve only one action (excluding finish_subtask).

W3.1: Although the authors don't use ground-truth labels, their exemplar construction process still utilizes tasks from the target domain. While this doesn't constitute supervision in the traditional sense, it does provide the model with domain-specific information that zero-shot baselines may not have access to, potentially creating an unfair comparison if the baselines are purely zero-shot.

AW3.1: All other baselines used in-context-learning methods. Our idea comes from precess of the data collection with a model or behavior policy. So we think the comparison is not unfair.

Q1.1: Could the authors clarify whether the 60 sampled tasks used for creating exemplars overlap with the test set? If so, how do they address potential data leakage concerns?

AQ1.1: Yes, the exemplars are sampled from the benchmark and included in later evaluation. But this process does not use the ground-truth labels or any other outer supervision. We let the LLM do reflection by itself and this idea comes from precess of the data collection with a model or behavior policy. So it will not lead to an unfair comparison, though it is more acceptable to use a new set of queries to construct the exemplar bank. We will leave this for our further studies.

Q2.1: How does the system handle cases where the verification phase produces false positives or false negatives? Is there any analysis of the verification accuracy?

AQ2.1: Good question. VeSX let reflection module to judge whether error happened during verification first. The example shown in Appendix A.1.2 is under such circumstance.

And we supplement the experiments to give more analysis of verification phase. We give an estimation of the sensitivity of verification:

1. The false positive rate can be estimated as 0.57. This result demonstrates that the verification may not be reliable when classified as a positive sample.

2. The false negative rate needs to be estimated using the success rate when not verifying and the success rate when verification fails, and can be estimated as 0.20. This result demonstrates that verification rarely classifies positive samples as negative, which does not incur much unnecessary self-correction overhead.

3.The true negative rate can be estimated as 0.79. This result demonstrates that the verification has high rate to capture the error in the process.

Here, the predicted positive means "verification pass", which is defined as the instance that all verifications in this process pass, while the predicted negative means "verification fail", which is defined as the instance where at least one verification does not pass.

The details of these experiments can be found in Appendix B.3.

Q3.1: How scalable is the exemplar bank approach as the number of tasks and domains increases? Is there a strategy for managing the growing size of the exemplar bank?

AQ3.1: We supplement some experiments to test the impact of scaling of exemplar bank. We mainly conduct two experiments:

1. Scaling of Exemplar Bank: We set four settings: (1) sampling 70% of the whole exemplar bank, (2) using the whole execution exemplar bank but not the planning exemplar, (3) using the whole planning exemplar bank but not the execution exemplar, and (4) using the whole exemplar bank. The results shows that using only 70% of the whole bank causes a obvious decrease of SR comparing to even removing the whole planning exemplars or execution exemplars. It indicates the more diversity of the execution or the planning exemplars will bring more increase and the potential of the scaling of the exemplar bank.

2. Scaling of In-Context_Learning Exemplars: The SR improves when the count of ICL exemplars increases, underlying the potential of the scaling of exemplars during the inference process.

We do not conduct experiments to quantify whether exemplars from other domain contribute to the current domain because the executions of subtasks in different scenarios do not share the same pattern. For example, you can search the items and click to view its information when in shopping sites, but you may enter a certain subreddit to search for some post in forums. The logic to accessing contents in shopping sites and forums vary, indicating the difficulty of scaling across domains.

One approach worth considering to manage the growing exemplars is to remove functionally similar exemplars from the exemplar bank, allowing the remaining exemplars to be more diverse. Currently, the size of the exemplar bank does not require a sophisticated management strategy. We will consider leaving this point for our future research after a system capable of collecting large amounts of data is implemented.

Thanks for the authors' response. Regarding Q 1.1, even though VeSX did not use the ground truth from the test set, it still utilized some tricks to formulate better in-context exemplars (Line 307), which is beyond the framework itself. Furthermore, as I already mentioned, the model is actually hacking some domain-specific knowledge in this situation. Given the potential data leakage risk, I decide to maintain my score.

We sincerely thank the reviewer for reading our paper and providing valuable insights.

W1.1: Limited experimental results and analysis including missing computational cost analysis...

AW1.1: We supplemented some experiments about computational cost analysis. The extra computational cost mainly comes from three aspects: workflow design, examples for in-context-learning and the more action steps.

1. Workflow Design: From the experiments, the verification, reflection and replanning modules account for approximately 10.7% of the input tokens and 16.1% of the output tokens. This indicates that the three main modules in VeSX workflow, verification, reflection, and replanning, do not introduce significant more computational cost.

2. Examples for ICL: From our experiments, the examples for ICL account for 35.3% of the input tokens, which is a big deal. However, considering almost every other methods use ICL technique, the computational cost of these examples are acceptable especially in that our scheme of exemplar bank can provide exemplars effectively and efficiently. Furthermore, we do summarization for the examples' input to save the input tokens while some of other methods use the redundant web context for ICL.

3. More Action Steps: We compared our method with the Sodhi et al. (2024) approach, which uses human-labeled subtasks to guide the LLM in breaking down and solving tasks. Their average action step is 9.1, while our average action step is 13.1. From this perspective, VeSX requires approximately 43.9% more actions compared to human-guided workflows to achieve competitive results.

Due to the significant time and cost involved in reproducing other approaches, we do not fully reproduce all of them. Compared to the sota method with human guidance, VeSX takes about ~50% more computational cost to achieve competitive results totally. It is nearly impossible for other methods to achieve the same accuracy by 1.5x scaling up without other design. The details of this experiments can be found in Appendix B.1.

W1.2: Limited experimental results and analysis including missing ..., error analysis especially when linked to the various contributed components in their framework

AW1.2: As suggested by the reviewer, we analyze the performance of each new module of VeSX, which is in Appendix B.3. And we conduct some case studies on the errors occurring in each module, including reflection and replanning, which can be found in Appendix C.

W2.1: Typing and grammar mistakes

AW2.1: We have carefully reviewed our article again and made revisions to the textual and figure errors. Thank you for the suggestions.

We sincerely thank the reviewer for reading our paper and providing valuable insights.

W1.1: The novelty is limited. Compared to previous work on web automation, the paper integrates self-reflection and retrieval-augmentation components, both of which have been widely explored. The paper also lacks discussions on relevant works on reflection and retrieval augmentation.

AW1.1: The self-correction technology has been rarely used in the field of web automation, and there is currently no solution centered around self-correction for web automation. VeSX implements a self-correction approach for web automation tasks with sophisticated design at various stages, such as subgoal-guided verification and a reflection & replanning error correction hierarchy. We have also designed specific elements within these stages, such as our proposed execution with expectation in the reflection phase. The retrieval-augmentation technology has been widely studied across various fields and is also commonly used in web automation. We have made some incremental improvements in this area. We addressed the issue of context length when applying previous retrieval-augmented methods to real-world scale scenarios, and we designed a metadata construction scheme to facilitate retrieval.

W2.1: Section 2.1 Overview is empty

AW2.1: Thank you for your correction! The missing section 2.1 has now been added.

W2.2: Clearly indicate success rates as percentages by adding the percentage sign (e.g., 34% instead of just 34)

AW2.2: We compared several presentation options. Including a percentage seems somewhat redundant, while omitting it does not clearly convey that this is a success rate. After weighing our options, we decided to use decimals for clearer representation.

W2.3: It will be better to put short descriptions in the captions for terms in the table (‘Shop’, ‘CMS’, ‘Red’, ‘Git’, ‘Map’).

AW2.3: In section 3.1, we introduced which scenarios correspond to each abbreviation. As suggested by the reviewer, we have also added descriptions of the terms in the table header to help readers obtain information more clearly and quickly from the table.

W2.4: Adding example prompts would provide readers with a practical understanding of the pipeline.

AW2.4: We also have added some examples of each module in the appendix. Please refer to Appendix A.2 section.

W3:

‘orders’ rather than ‘oreders’ in the teaser figure.

Miss left bracket for ‘click sorted by]’. Is ‘click [sorted by]’ and ‘click [sortby]’ the same operation?

the texts frequently touch or cross the boundaries of the icons.

Some figures are blurry and difficult to interpret. (e.g. In Figure 2, it is not clear what the four boxes below the environment represent.)

The figure captions should be refined to clearly describe each major component.

AW3: Thank you for your thorough reading of our article and for pointing out so many formatting errors! We have carefully reviewed our article again and made revisions to the textual and figure errors.

Q1.1: How does VeSX distinguish itself from approaches like Tree of Thought (which leverages branching and self-verification of reasoning steps), Reflexion (which incorporates self-reflection mechanisms), and various retrieval-augmented frameworks?

AQ1.1: The ToT and related tree search frameworks have been widely used in various agent tasks, including web automation. The approach that leverages branching and self-verification of reasoning steps has been adopted by many agent frameworks and has become a mainstream agent paradigm. Among the baselines we compared in our experiments, some solutions, such as Koh et al. (2024) also used tree search methods, but their performance was not as strong as ours. This is due to VeSX's sophisticated design in how verification and error correction are handled. What we aim to showcase is how to design effective verification and error correction solutions, an area that lacks relevant research in web automation. From this perspective, our work is entirely innovative. Regarding the comparison with the reflection and retrieval-augmented frameworks, we addressed this question in AW1.1.

Q1.1: How many total tasks are there for each scenario?

AQ1.1: Here we list out the tasks for each scenario:

Q2.1: Out of the 60 sampled tasks for each scenario, how many exemplars were produced?

AQ2.1: Here we list out the details of the exemplar bank:

We also update this information in Appendix B.2.

Q3.1: After re-planning is done or self-correction, does the process start from the beginning again? Is there a limit to the number of times self-correction or reflection is allowed? Is this the same as in other papers?

AQ3.1: VeSX will not restart from the beginning but from the errorous subtask. We are not limit the time of self-corrections but set a restrict to the total steps of the executions for the whole process to prevent it stuck in an endless loop. This setting is follwed by other papers.

We sincerely thank the reviewer for reading our paper and providing valuable insights.

W1.1: Presentation: I am a bit confused about the overall workflow. It would be helpful to have it written in an algorithm.

AW1.1: As suggested by the reviewer, we have improved the presentation by adding an algorithm of the whole workflow in the appendix part of our revised version. Please refer to Appendix A.1.1 section.

W1.2: Presentation: It would also be helpful to see more examples.

AW1.2: We also have added some examples in the appendix. Please refer to Appendix A.1.2 section.

W2.1: Extra Time and Cost: How much extra time and tokens does it take for this method compared to others (if available for other methods)? If these other methods also had access to more compute, they might also have higher performance.

AW2.1: We supplemented some experiments about computational cost analysis. The extra computational cost mainly comes from three aspects: workflow design, examples for in-context-learning and the more action steps.

1. Workflow Design: From the experiments, the verification, reflection and replanning modules account for approximately 10.7% of the input tokens and 16.1% of the output tokens. This indicates that the three main modules in VeSX workflow, verification, reflection, and replanning, do not introduce significant more computational cost.

2. Examples for ICL: From our experiments, the examples for ICL account for 35.3% of the input tokens, which is a big deal. However, considering almost every other methods use ICL technique, the computational cost of these examples are acceptable especially in that our scheme of exemplar bank can provide exemplars effectively and efficiently. Furthermore, we do summarization for the examples' input to save the input tokens while some of other methods use the redundant web context for ICL.

3. More Action Steps: We compared our method with the Sodhi et al. (2024) approach, which uses human-labeled subtasks to guide the LLM in breaking down and solving tasks. Their average action step is 9.1, while our average action step is 13.1. From this perspective, VeSX requires approximately 43.9% more actions compared to human-guided workflows to achieve competitive results.

Due to the significant time and cost involved in reproducing other approaches, we do not fully reproduce all of them. Compared to the sota method with human guidance, VeSX takes about ~50% more computational cost to achieve competitive results totally. It is nearly impossible for other methods to achieve the same accuracy by 1.5x scaling up without other design. The details of this experiments can be found in Appendix B.1.

W3.1: Original of exemplars: Are the exemplars produced from questions in the benchmark? Are those questions included in the final results? This could also lead to an unfair comparison.

AW3.1: Yes, the exemplars are sampled from the benchmark and included in later evaluation. But this process does not use the ground-truth labels or any other outer supervision. We let the LLM do reflection by itself and this idea comes from precess of the data collection with a model or behavior policy. So it will not lead to an unfair comparison, though it is more acceptable to use a new set of queries to construct the exemplar bank. We will leave this for our further studies.

W4.1: One stated advantage of the approach is that human guidance is not needed. Is any human guidance used to design the prompts for the different steps? Is the exemplar bank used as in-context examples for all of the different steps?

AW4.1: In VeSX, the same prompt is used for different steps in execution (except for retrieving different exemplars from the exemplar bank for in-context-learning). The exemplar bank is same for all steps.