Agent Workflow Memory

Thank you for your recognition in the novelty of our AWM method design, the effectiveness shown by extensive experiments, and AWM’s strong generalization abilities.

W1: Writing needs to be improved We appreciate your comments. However, since the feedback is somewhat general, and given that other reviewers have noted the clarity of our writing, we would greatly appreciate any additional details you could provide. If you could clarify specific aspects of the story or identify areas where motivation and clarity could be further strengthened, we would be more than happy to make those improvements.

W2: Workflow Induction Clarity The workflow induction process prompts an LLM with examples ((NL, trajectory)) pairs, and expects it to output workflows (as defined in section 2.2). Please find more discussions in formalized ways (throughout section 2.3), in graphical ways (Figure 2, 3, and 4), in natural language descriptions (throughout section 2.3), and with concrete examples (Appendix A). Please let us know which dimension you found unclear so that we can improve on it! Also, we would like to clarify that we do not “choose particular workflows over the others” (as stated by the reviewer). Our method incorporates all induced workflows into agent memory without explicit selection.

W3: Discuss limitations We will discuss more limitations of our AWM approach, including its current limited capability in generalizing across tasks/domains, W4: Considering factors like efficiency and scalability Thank you for recognizing the importance of efficiency/scalability beyond task success rate. We agree with this and have already evaluated this dimension, by reporting the number of steps the agent took to solve the task, in the “# steps” column in Table 1 — which shows AWM takes fewer steps thus has better efficiency. Besides the action generation step, our AWM approach adds two other steps – trajectory evaluation and workflow induction. We calculate the average compute of all three steps by the number of input/output/total tokens per step, the average number of times that the step occurs per task, and the total number of tokens used on average for a task. As shown in the table below, the trajectory evaluation step and workflow induction step only take 4.0% and 6.8% of the compute of the original action generation step. Compared to the baseline method (using action generation step only), our AWM approach only adds 10.8% computation overhead, but brings a 51.5% accuracy increase in WebArena tasks, demonstrating the cost-effectiveness of our AWM approach.

Furthermore, we carefully study the learning process of our AWM agent and plot its learning curve after processing an increasing number of examples, in Figure 5 and Figure 1. We show that our AWM approach enables agents to quickly acquire essential workflows, thus superior skills in task-solving, by processing only tens of examples (<40) with zero supervision, demonstrating the scalability of our method.

Q1: Discussion of the terminology used We agree and have acknowledged these relevant inspiring works in a full paragraph (line 524–532) in section 6. Nonetheless, we would like to clarify a key difference in our “agent memory”, and why we favor it against the “skills” and “library” in previous works such as Voyager: our “agent memory” presents in text, which provides several benefits that programmatic “skills” do not have: (1) less requirement in accuracy: the text can be free-form and verbalized in flexible ways, without having to be executed successfully, alleviating the requirement on induction step; (2) our workflows can easily generalize across websites and domains, whereas programmatic skills need to be modified to execute on other websites. Furthermore, we have demonstrated turning workflows into actions, i.e., “skills”, in Section 5, and showing its limited applicability in web navigation tasks, as of now.

In short, we do appreciate and acknowledge these inspiring works, but proposed our textual workflow memory that can solve the web navigation task more flexibly and accurately.

Thank you for recognizing our reasonable method design, clear writing and figures, as well as the extensive empirical analysis throughout the sections.

W1: Limitation in generalizing to different websites First of all, we do see improvements in generalization with our AWM approach in (i) cross-website transfer in WebArena (Section 3.1.3), and generalization across tasks, websites, and domains on Mind2Web (Section 3.2.2), compared to all baseline methods. We design AWM exactly because of the limited generalization of LLMs out-of-the-box, by enabling agents to learn new workflows starting with zero experience on an unseen website.

W2: Unfair online setting We argue that our online setting is fair to compare with baseline methods. AWM has the same information as the baseline methods (i.e., all test queries), only has access to test queries, and no annotated solutions for any problems are needed. Our method simply changes the way the agent processes the test queries — by deriving and reusing workflows on the fly, instead of processing each test query independently. More importantly, we believe our online setting is a more realistic setting for evaluating agents, where user queries come in a stream, and no/little supervision is provided. In many other reasoning/agent tasks, people are now beginning to explore other ways to synergize multiple test examples rather than treating them independently, such as batched prompting [1]. We think these different ways (including ours) to process test queries are valid and constitute fair comparisons.

Despite the online setting, our offline AWM approach also achieves superior performance in task success and generalization, showing the effectiveness of our method.

W3: Table 4 MindAct cross-domain results reports gpt-3.5 Thank you so much for pointing this out! We will update the results to GPT-4 scores. In particular, the GPT-4 step success rate is 26.4, which is still lower than our AWM approach, with 32.6 (offline) and 35.5 (online), so our conclusions still hold.

[1] Cheng, Zhoujun, Jungo Kasai, and Tao Yu. "Batch prompting: Efficient inference with large language model apis." arXiv preprint arXiv:2301.08721 (2023).

Thank you so much for your response!

W2: I disagree. A lot of methods do not assume the existence and perception of tasks from the same test task distribution; i.e., they do not accumulate knowledge across different tasks from the same test task distribution. I'm not suggesting that exploiting such information is a bad practice, but the purpose of academic benchmarks is to simulate real-world scenarios on a smaller scale. In order to show real-world effectiveness, I believe its scalability should be demonstrated at a much larger scale, where the task distribution is broader and making connections between different test tasks is nontrivial.

W1: In addition to my response regarding W2 above, my point about generalization was that we cannot always assume relevant experiences from the same test task distribution.

Overall, I appreciate the authors' efforts in preparing the rebuttal, but I keep my original view on this submission.

Thank you for recognizing the effectiveness of our method, the comprehensive experiments we conducted, and the clarity in the method description.

W3: AWM sensitivity to example order It’s a good observation that example ordering may affect the AWM agent performance. We can test the effect of example ordering on the Mind2Web dataset (cross-task split) using AWM online approach. Specifically, we compare original order, random shuffling, ordering from easiest to hardest, and ordering from the hardest to the easiest (the more steps in the ground-truth trajectory, the harder the task). Results are shown in the table below.

First of all, AWM using any example ordering still substantially outperforms the MindAct baseline. Moreover, the ordering of examples does not significantly affect the performance of our AWM approach, where all four example ordering achieves similar success rates. Coupled with a careful analysis of the derived workflows, we found that our design of “sub-task level” workflow contributes to AWM’s robustness to example ordering — regardless of the complexity of the task, our method can induce useable workflow.

However, for the WebArena dataset we experimented on, the examples are intentionally ordered in a specific ordering to maintain the validity of the browser environment. Therefore, we could not change the orders arbitrarily to examine this effect.

W4: “# steps” is unclear We argue that “# steps” is more informative than “# tokens” in agentic settings, where each step represents an agent action such as “click(12)”; where tokens may mislead this number, e.g., a step “click(12)” and a step “send_msg_to_user(“The highest score is 98”)” takes roughly same amount of time and same number of LM calls; however, if presented by tokens, “click(12)” has four tokens [click, (, 12, )] and “send_msg_to_user(“The highest score is 98”)” has 13 tokens [send, _msg, _to, _user, (, “, The, highest, score, is, 98, “, )], which misleadingly shows a big difference. Moreover, the number of actions/steps is a commonly adopted metric in agent works, e.g., SteP [1].

W5: Explain “wrapping each workflow into a high-level function” We provide concrete examples for wrapping workflows into high-level functions in line 480–482, where we rewrite the textual workflows to executable functions, and the step in textual workflows turns to lines of programs in the high-level function. More concretely, given a textual workflow:

Workflow: login to the website
# I will first click the username box to input
click(box1-id)
# next, type in the username
type(box1-id, username)
# after finishing the username, I will now click the password textbox
click(box2-id)
# then, input the password
type(box2-id, password)
# lastly, click the submit button to initiate the login process
click(submit-id)

We turn it into a high-level function:

def login_to_website(username, password, box1-id, box2-id):
    click(box1-id)
    type(box1-id, username)
    click(box2-id)
    type(box2-id, password)
    click(submit-id)

Q1: Explaining high gains on Maps Website In short, compared to the other websites, the tasks on Maps website are relatively simple and similar to each other, thus benefiting more from our workflow reuse strategy.

We provide a detailed illustration of the agent learning process on the Maps website in Figure 1, where all marked workflows and accumulative success rates are accurately depicted. As we stated in section 3.1.3, validated with quantitative experiment results in Table 2, and illustrated the major pattern of AWM in Figure 6, our AWM-equipped agent learns increasingly complex workflows built on top of earlier, easier ones, thus being able to solve more and harder problems over time, leading to higher results than baselines. We do not modify our method in any way particularly for the Maps website.

W1: Refine Figure 4 Thank you for pointing this out. To possibly improve the figure, we could (1) add arrows to indicate example ordering and streaming, (2) use icon to highlight the “induce” part. Free feel to provide more suggestions and we are happy to incorporate them.

W2: Causing WebArena data leakage AWM does not leak WebArena data. As we highlighted in Section 1, 2, and Figure 4, our method only requires test queries, and never uses ground-truth trajectories nor evaluations. Since our AWM agent never has access to any potential “answers”, we believe our method will not cause WebArena data leakage.

[1] Sodhi, Paloma, et al. "Step: Stacked llm policies for web actions." First Conference on Language Modeling. 2024.

Thank you for recognizing the effectiveness of our method, careful experiments, and clarity in paper writing.

W1: Add description for implementation details of closed-source models We agree that closed-sourced models such as the GPT models provide limited explainability, nonetheless, interfacing with closed-source modes is common, as in many impactful works and importantly most related works. Moreover, we want to show that our AWM approach is compatible with and improves state-of-the-art models, which are all closed-source. To provide more implementation details for our AWM method, we described all hyper-parameter settings in lines 214–215 (for webarena) and lines 317–323 (for mind2web). We also provided our entire codebase in the supplementary material, which includes all implementation details of our method.

W2: Not clear how memory is extracted/used As illustrated in Figure2 and explained in section 2.3, we induce workflows from past experiences then add them to the agent memory (Figure 2, step 3) – we input (NL, trajectory) examples to the model and prompt it (see prompt in Appendix A.1) to output workflows (see examples in Appendix A.2); we add workflows to model contexts at next action generation process. If you are asking how the database of all workflows is used to solve a new problem — we add all workflows in the database into the model inputs. The database as 7.3/7.4 workflows on average for WebArena/Mind2Web, so can be easily fitted into model context.

Q1: How many workflows can be stored in memory We do not limit the number of workflows stored in the memory. Empirically, our agent uses 7.4 workflows per website on WebArena, and 7.3 workflows per website for Mind2Web, as presented in Table 10.