Agent Workflow Memory

Thank you for the feedback and for pointing out these relevant works. We list the core differences between AWM and these works below, and will add these more detailed discussions in our revised paper version!

Difference from Reflexion

1. Content: AWM learned procedural knowledge on how to solve specialized, reusable tasks, i.e., only correct trajectories; Reflexion receives errors and comes up with plausible (erroneous, correct) solution pairs.

2. Context window: AWM carries reusable procedural knowledge across multiple tasks experienced over time, therefore could leverage and benefit from the procedural similarity across tasks. Reflexion only maintains learned reflections within the current single-task solving session.

Difference from Expel

1. Requirement on training data: Expel requires learning from training examples; AWM not only supports learning from training/offline tasks, but can also be directly applied online/testing without the need for any training data beforehand.

2. Expel stores raw trajectories in memory (similar to Synapse below); where our AWM stores sub-trajectory level, context-abstracted procedures, which have shown to be more reusable and beneficial in our experiments (Table 3 and Table 4).

Difference from Synapse

1. Full-trajectory vs. Sub-trajectory: AWM proposes to learn reusable procedures that are often part of the trajectory for a given task; in other words, a complex, long-horizon task usually needs to concatenate multiple workflows to solve it correctly. Synapse only remembers the full trajectory, but is often less reusable for other tasks (as the example in lines 147-161).

2. Dependent vs. Independent on example-specific context: Synapse records specific contexts to individual examples, e.g., “Flight from New York to Pittsburgh”; while AWM abstracts away those variable contexts to, e.g., “Flight from {departure} to {destination}”, so that it could be more easily reused to apply to other examples such as booking a flight from Chicago to Toronto (also refer to the example in lines 147-161).

Claims and Evidence 1: AWM Improvement May Come from Structured Nature of Actions

First, we argue that WebArena and Mind2Web queries are realistic and human-annotated without intentionally injecting superficial structures, so we believe that experimenting on them is well-motivated. Our method depends on structures that are shared across trajectories (e.g., adding an item to a cart, searching for a term), so individual trajectories need not be repetitive. Meanwhile, datasets for other tasks, e.g., embodied/robotics, actually often have more superficially structured queries (e.g., “harvest 1 dirt” and “harvest 8 dirt” on Minedojo), so we may not fully agree that switching to other tasks would be a perfect solution for alleviating learning superficial task structures.

Claim and Evidence 2 & Experimental Design 1: Dependency on Prompting Strong LLMs

Although we experiment with strong GPT models, we do achieve very substantial improvements (over 50%) compared to using the vanilla GPT-4o model. For this reason,we would like to argue that AWM is effective, instead of having “limited” “methodological advancement”. One prerequisite for AWM (and many other autonomous agent frameworks) is that the LM backbone needs to have reasonable abilities in task-solving. Our main focus in this work is to establish this promising self-adaptive agent framework. It would be interesting future work to investigate the prerequisites for LM capabilities or other aspects for AWM to show effect.

Ambiguity in the Definition of Workflow Induction

As we described in lines 141-161, we input several examples into GPT model and ask it to generate workflows, which are different from input examples in two aspects: (i) they have sub-trajectory level granularity and thus are more reusable than full-trajectory examples, and (ii) they abstract away example-specific contexts and replace them with descriptive variables (e.g., “New York” to “{destination city}”) to facilitate generalization across tasks.

Unclear Control over Workflow Granularity

We agree that since we mostly rely on LMs to come up with workflows, we have limited control over the granularity generated by the LM. The alternative approach is rule-based induction, which hardly produces any workflows at all given the dynamic environment nature of web navigation tasks. Empirically, we have carefully analyzed the quality of all induced workflows in Appendix A.3, about the number of workflows induced, their coverage on test queries, their functional overlap, etc. Please find more analysis in Appendix A.3 and example workflows in Appendix A.2. Throughout the analysis, we find the workflows have reasonable granularity, as we prompted the LLM to generate sub-trajectories and abstract away example-specific contexts.

Benchmark missing: WebVoyager, MiniWob++, WorkArena

WebArena and Mind2Web are more widely used. Webvoyager may raise safety concerns as it operates on real-world websites; MiniWob++ is simplified and agent performance is not saturated (over 95% success); WorkArena is application-specific and largely follows the WebArena setup. We are interested in seeing future works experimenting with AWM with these more specialized benchmarks.

O1. Definition of Memory Size

In the offline setting, we learn workflows from training examples and keep the memory size fixed in the later inference process. In the online setting, the memory size is continuously growing as the agent solves more tasks and induces more workflows.

O2. Unclear observation and action space

As mentioned in lines 189-195, we adopt the browsergym framework and use the webpage accessibility tree as the agent observation. We adopt the WebArena action space supported by BrowserGym. We will add the list of actions to the revised paper version.

O3. What is the “base memory”

Base memory refers to the original agent memory before adding any induced workflows. Implementation-wise, this contains instructions for the model to follow the user query and solve the tasks. We provide the exact prompt, a.k.a., the base memory, in Appendix A.1.

Thank you for recognizing the effectiveness of our AWM approach!

AWM’s robustness to varied offline task selections

We agree that offline task selection is crucial for AWM to achieve better performance, where higher distribution overlap between offline/training tasks and online/testing tasks improves agent performance. As we directly adopt the Mind2Web dataset in our experiments on Mind2Web (to fairly compare with existing methods), its three test sets (as introduced in sec 3.2 beginning) already introduce mismatches in tasks, websites, and domains to those covered in the training examples. As shown in Table 4, our AWM still outperforms baselines on all three test sets, demonstrating its robustness when the train-test task distribution varies to different extents.

Extend memory beyond subroutines

We agree with the reviewer that memory can take various forms. Our work aims to take a first step in learning one of the most crucial memory types -- reusable, procedural workflows -- that boost agent success and efficiency. It would be a promising direction to extend this adaptive AWM method to other types of memories, such as the experienced states and failed trials suggested by the reviewer. However, as this requires substantial extra engineering effort and ideation, we think it’d be more suitable to study these memory variants in another standalone work.

Generalization across tasks

We provide one case for this cross-task generalization behavior in Figure 6, where the agent learns to build new workflows on top of previously learned, easier ones. Nonetheless, our agent can choose not to refer to existing workflows and come up with new ones solely based on the action trajectory when solving new examples (as illustrated in Figure 2 pipeline), and therefore to some extent can avoid bias from irrelevant existing workflows.

Reliance on memory

Our AWM pipeline induces workflows only from trajectories that are predicted as correct, and therefore is not expected to generate erroneous workflows which cannot solve tasks correctly. We also manually evaluated all the induced workflows (7.4 per website, so it is feasible to carefully check all of them; more detailed in Appendix A.3) and found them correct and useful. We cannot provide example workflows on each website in this response due to length limits, please find more examples of induced workflows in Appendix A.2. It would be an interesting follow-up study to examine and improve the quality of workflows and make AWM more effective, if possible.

Search for a Product
# To search for a product, I need to enter the search term in the search bar and press Enter.
fill('search_bar_id', '{search_term}')
press('search_bar_id', 'Enter')

Find Directions Between Two Points
# To find directions between two points, I will use the directions feature on OpenStreetMap. I will start by clicking on the "Find directions between two points" link.
click('149')
# Next, I will fill in the "From" and "To" fields with the respective locations and select the mode of transportation.
fill('158', 'FROM_LOCATION')
fill('163', 'TO_LOCATION')
select_option('166', 'MODE_OF_TRANSPORTATION')
click('171')
# I will then retrieve and send the calculated travel time and distance to the user.
send_msg_to_user('The estimated travel time from FROM_LOCATION to TO_LOCATION is ESTIMATED_TIME.')

Thank you for recognizing the soundness of our method, the general applicability in online and offline settings, and the comprehensiveness of our experimental analysis!

W1. Experiments tailored to web agents

We fully agree that AWM can be similarly applied to other agentic tasks such as robotics or tool use. We conducted very comprehensive experiments with two major web agent benchmarks emphasizing varied correctness and generalization aspects, which we believe are sufficient contributions to this work and prove AWM’s effectiveness; as many influential works on agents focus on one domain as well [1,2]. Nonetheless, applying AWM to other tasks such as robotics would require substantial engineering effort (setup hardware and software), which we think would be more suitable to explore in standalone future works.

W2. LM introduces bias into workflow construction and judgment processes

For workflow construction, we have compared LM-based induction and rule-based induction (no LM involved) in Section 4.1 & Table 5. The rule-based method directly adopts content from the example without introducing any bias from the model, but their comparable scores show that LM does not introduce any “negative bias” into the workflows. For the judgment process, we have compared using LM-based evaluation and ground-truth evaluation on WebArena in preliminary studies, and found that LM-based judgment leads to a higher average success rate. We analyzed results and found ground-truth evaluations to be overly strict and disagree more with human correctness judgments; whereas LM is more flexible and makes more accurate judgments. For example, ground-truth evaluation checks if the final state url exactly matches the annotated one (“xxx.map.com/cafe-near-cmu”), while the agent may land on “xxx.map.com/cafes-near-cmu” that searches the plural “cafes” instead of “cafe” but still get the correct results; ground-truth evaluators will inaccurately judge the result to be wrong, but LM-based evaluator and humans often still mark it as correct.

W3 & Q4. Compare to using raw experiences

We have compared AWM to using raw experiences on both datasets. For Mind2Web results in Table3, the Synapse method is conditioning on retrieved raw examples for task-solving, and AWM outperforms it by 13.1% in success rate. For WebArena results in Table4, the rule-based induction is a stronger version of using raw examples, in that we further remove the invalid steps in raw examples to improve their quality. As AWM achieves comparable results to this stronger raw-example-using baseline, we expect AWM to perform better than using the original raw examples.

Q1. Online evaluation granularity and quality

The LM-based evaluation is applied to the overall outcome, i.e., on the full trajectory. We induce “correct” workflows from trajectories that are predicted as correct, preventing incorrect trajectories from being converted into workflows. We made this design choice to ensure the quality of workflows and the effectiveness of the entire pipeline. It would be an interesting extension of our work to further consider adding incorrect trajectories for workflow construction, but it will necessitate stronger trajectory evaluators and design changes in the pipeline to make it more effective in improving end success rate.

Q2. Can AWM generalize to ALFWorld and SciWorld

Yes, our AWM method generally applies to agent frameworks as everything is built on top of general agent components. While it depends on the task and domain overlap between our experimented benchmarks (WebArena, Mind2Web) and other datasets (ALFWorld, SciWorld) to tell if workflows produced in our experiments can be directly used to solve other datasets, it would be interesting to explore AWM performance on ALFWorld and SciWorld in future work.

Q3. AWM Requirement on Underlying Model Capabilities

AWM does require the LM to have certain capabilities in task-solving, trajectory evaluation, and workflow induction. We have experimented with LLaMa 70B and Deepseek-R1 32B and find they benefit less from AWM — achieving 7.9→6.2 and 13.5→11.6 success rate before→after applying AWM – we will reference these results in the revised paper. Nonetheless, our main goal is to showcase this promising self-adaptive agent framework as agents get increasingly stronger.

[1] Liu, Evan Zheran, et al. "Reinforcement learning on web interfaces using workflow-guided exploration."

[2] Wang, Guanzhi, et al. "Voyager: An open-ended embodied agent with large language models."