CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization

> The ablation doesn't show the usefulness of the causal abstraction

We apologize for being unclear here: When we drop the constraint that the memory should have a causal structure ("is necessary" / "does not contribute"), the memory generator can generate anything - and, in fact, stops generating causal abstractions and instead generates generic and often unhelpful advice (as others doing unconstrained memory generation have also found). Some example unconstrained generations are:

1. Avoid repeating actions: The agent performed the "look around" action twice at the beginning of the task. This is unnecessary and wastes time. Once you have a clear understanding of your surroundings, move on to the next step.
2. Be mindful of the order of actions: The agent moved to the workshop and read the instructions after looking around twice. It's better to first read the instructions to understand the task requirements.
3. Be clear with your actions: The agent made an ambiguous request when trying to mix the cup containing red paint. Be specific with your actions to avoid confusion.
4. Avoid unnecessary movements: The agent teleported to the art studio after picking up the paper. Plan your actions carefully to avoid unnecessary movements.

The ablation (Figure 5d) thus shows the value of having a memory focused on understanding the causal effects of actions (rather than just any old reflection), following our rationale that this will help an agent mentally simulate their choices before acting. This is a key contribution of our work, and distinguishes it from other reflective approaches. For better readability, we included this example in the Appendix.

> What kind of information is still missing for the long tasks?

Longer tasks often require actions whose effects are not observed immediately but observed later in the time. Many longer tasks require frequent "waiting," allowing the environment to have the necessary changes (waiting for the bees to cross-pollinate). While our agent could not perform well at the initial trials later added a new memory: "Repeatedly waiting after opening the bee hive may be necessary to the growth of the cherry plant.". However, such a memory, while helpful, is often not precise about how long the agent needs to wait for the cross-pollination to happen, a stochastic process. Due to a lack of dense reward and sparse signal from the observations, our agent sometimes cannot complete longer tasks, which is indicated by Figure 4a.

Figure 4c shows the % of episodes where there is a positive change in scores as compared to the Base agent (first trial). For shorter tasks, the room for improvement is less, as the average absolute score is higher as compared to longer tasks. However, we show higher relative improvements for longer tasks, indicating CLIN is able to bridge the performance gap by rapid adaptation.

Thank you for taking the time to review our paper! We were happy to read that you appreciated our main points: that our framework non-parametrically updates its learning through a persistent memory, that it can recover from past errors by abstracting causal inductive rules about the world, and that CLIN is efficient in solving complex interactive tasks just using few trials achieving superior generalization performance.

> What are the design criteria and the rationale for the memory generator?

This is an important question! The rationale for the memory generator is to generate memories that help the agent mentally simulate the causal effects of actions, thus making more informed action choices in the future. This was missing in prior work, where memories were either task-specific factual instructions (e.g., "I should go to desk 1") or general hints ("Think about an object's usage when searching for it."). The idea of learning action causality has been previously explored in the formal planning literature (learning "action models") - our work is the first to take this idea into the modern context of LLMs, show its benefits, and make an important contribution.

There are several ways this rationale can be operationalized, and in CLIN, we explore the simplest approach, namely guiding the memory generator to produce memories using two commonsense relations: "necessary" to indicate a positive effect of an action and "does not contribute" to indicate an undesired effect of an action. As you suggest, given this rationale, future work could explore alternative operationalizations of how action knowledge could be expressed, e.g., richer sentence templates. The important contribution of our work here is to introduce the general idea of learning causal action knowledge in a language-based setting to enable better performance in the future.

> How accurate are the generated memories?

Thank you for asking this question! While the final performance with memory is indicative of their effectiveness, we performed additional human evaluation of generated memory insights for correctness. For both the Gen-Env and Gen-Task setups, we randomly 10 task-environment combinations to evaluate the correctness of memories used in them, notably the meta-memory used for trial 0 (GEN) and memory adapted for the best trial (G+A). Two annotators rated all the insights (cohen's kappa 0.72) for correctness with reference to the respective gold trajectories, and the results are as follows:

So, it is possible to have some meta-memories (here ~27%) that are not applicable for a new task or environments at Trial 0; however, with adaptation, in later trials, the correctness of the memory insights significantly improves (17-19% absolute), leading to a direct increase in task performance. We included this analysis in the Appendix.

> Does the performance improve because the LLM generates the correct memory, or just provides non-causal details of the task?

The memory generator is constrained (via prompting) to always produce causal statements, so there are no non-causal details in memory. From sample analysis, we observe that the generated memories are almost always correct with respect to the trials that have been seen, and empirically, performance improves using them (same task). However, when the environment or task changes, those learned memories may be over-general or incorrect (e.g., "activating the stove is necessary for heating water" is no longer correct in a new environment where the stove is broken). In such cases, we see an initial performance drop, but then further iterations cause those errors to be corrected, and performance improves even further. This can be seen in Figures 5a and 5b: In the same environment/task, CLIN improves (blue line), but then if the environment/task is changed, its performance drops as some learnings in memory are too specific (start of red line vs. end of blue line), but it then later recovers and improves even further (end of red line).

Just looking at the task description and the initial state of the environment is not enough to produce grounded causal rules

Meta-memory when only the task description and the initial state are provided (Note ScienceWorld is a partially observable world)

NEW TASK: Your task is to grow an orange. This will require growing several plants and then being crosspollinated to produce fruit. Seeds can be found in the bathroom. To complete the task, focus on the grown orange. This room is called the bedroom. In it, you see: the agent, a substance called air, a bed. On the bed is: a mattress. On the mattress is: a white pillow, a book shelf (containing A book (Moby Dick) titled Moby Dick by Herman Melville, A book (Pride and Prejudice) titled Pride and Prejudice by Jane Austen, A book (Sherlock Holmes) titled Sherlock Holmes by Arthur Conan Doyle), a closet: The closet door is closed, a drawing, a table: On the table is: nothing. You also see: A door to the hallway (that is open).

1. Air may be necessary to grow an orange.
2. The bed does not contribute to growing an orange.
3. The mattress does not contribute to growing an orange.
4. The white pillow does not contribute to growing an orange.
5. The book shelf may not contribute to growing an orange.
6. The books on the shelf do not contribute to growing an orange.
7. The closet may be necessary to store tools or materials for growing an orange.
8. The drawing does not contribute to growing an orange.
9. The table may be necessary to place the plant pots for growing an orange.
10. The door to the hallway may be necessary to move between rooms and gather necessary items for growing an orange.

Without any interaction with the world, it could only generate trivial insights like the above that describe whether each object it observes at the start of the trial "is necessary/ does not contribute" to the task. Unless the agent navigates to an appropriate room, it won't see objects like "seed jar", "flower pot", "jug", "bee hive" etc. that are essential to complete the task. On the other hand, CLIN's meta-memory is composed of much more informative insights based on interactions with the environment from its previous experience on growing orange in different environment configurations. The learnings below mention essential objects, relevant actions, and how they "are necessary / do not contribute" to the task and subgoals within the task.

1. Looking around may be necessary to identify the objects in the room.
2. Moving to the appropriate room should be necessary to find the seed jar.
3. Picking up the seed jar should be necessary to start the process of growing an orange.
4. Moving to the greenhouse should be necessary to plant the seeds.
5. Placing the seeds in the flower pot should be necessary to plant the seeds.
6. Picking up the jug may be necessary to water the seeds.
7. Filling the jug at the sink should be necessary to get water for the seeds before pouring it into the flower pot.
8. Pouring water from the jug into the flower pot should be necessary to water the seeds.
9. Waiting may be necessary for the seeds to grow into an orange plant.
10. Repeatedly waiting may contribute to the growth of the orange plant.
11. Opening the bee hive may be necessary to crosspollinate the plants.
12. Repeatedly waiting after opening the bee hive may contribute to the growth of the orange plant.

CLIN produces generalized exploration bias, self-adapt if necessary, the novelty of our framework. When there is a huge search space for possible actions, any attempt to focus on desirable (part of the) trajectories (or memories) to glean relevant insights can be seen as a necessary exploration bias to perform well in unseen tasks/environments.

Thanks for your response!

We would like to just push back a little:

CLIN does differentiate between memories within a single task+environment (its normal memory) and when the task/environment changes (the meta-memory)

For example, within a single task+environment, it may learn "going to the kitchen is necessary for finding a cup". However, given a new task+environment, it will generate a meta-memory from previous memories of earlier (different) task+environments. CLIN is self-adaptive: the degree of generalization depends on the variation it's seen earlier. For example, if a cup has always been seen in the kitchen, it will retain "going to the kitchen is necessary for finding a cup." However if the cup appears in different places in different earlier trials, CLIN will drop or generalize (e.g., "going to a room is necessary for finding a cup") such a statement when building the initial meta-memory. In this way, CLIN automatically adapts to learn what is constant (e.g., the stove is always in the kitchen) and what is not.

Similarly, in ALFWorld, if it has always seen remote controls to be found in drawer 1 or drawer 2, then the generalization would be “Going to the drawers may be necessary to find the remotecontrols.” While this looks simple for ALFWorld, in ScienceWorld, CLIN learns compositional generalization as a part of the meta memory:

Task: boil chocolate.

Environment 1: You start from art studio, Stove is working (revealed during interaction)

Memory 1:

Moving to the kitchen should be necessary to access cooking tools and ingredients. Turning on the stove should be necessary to boiling the chocolate.

Task: boil chocolate.

Environment 2: You start from kitchen, Stove is not working, lighter available (revealed during interaction)

Memory 2:

Using the lighter on the metal pot should be necessary to heat the chocolate. Turning on the stove does not contribute to boiling the chocolate if the stove is broken.

Meta memory from Memory 1 and Memory 2:

Moving to the kitchen should be necessary to access cooking tools and ingredients. Using a heat source (stove, lighter) on the container should be necessary to heat the substance. Turning on the stove does not contribute to the task if the stove is broken. CLIN can adapt further (for the specific task+environment): If such learnings in the meta-memory are violated in the new environment (e.g., although it’s always seen the cup in the kitchen, now it's suddenly no longer there!), then CLIN will continue to adapt by again generalizing or dropping such memories in the next iteration.

In other words, the degree of specificity in the memory is constantly being updated, and the memory is being continually self-tuned to the tasks and environment. If the environment is not changing much, then memories will tend to be (desirably) fairly specific. If the environment is changing a lot, then memories will be more general. This self-regulation of CLIN is an important point and a key contribution of the work that distinguishes it from earlier work. We hope this addresses your concern here. We will make this clearer in the paper!

Thank you for taking the time to review our paper! We were happy to read that you appreciated our main points: that our framework outperforms many existing works by a large margin and that our work is informative and well-presented.

> Can you clarify the novelty?

There are two key novelties:

1. While the idea of having a memory is not new, our novel contribution is to make this a dynamic, evolving memory over time, in contrast to the short-term "reflect, use, then discard" approach used in Reflexion and other memory-based agents. This allows CLIN to progressively build insights about the world over multiple experiences in different environments. It also allows CLIN to recover from bad or not-useful learnings in memory, as these will get dropped when they are found not to help as the memory is updated.

2. The role of the memory is itself novel, namely to help the agent mentally simulate the (learned) causal effects of actions, thus making more informed action choices in the future. This was missing in prior work, where memories were either task-specific factual instructions (e.g., "I should go to desk 1") or general hints ("Think about an object's usage when searching for it."). The idea of learning the causal effects of actions ("action models"), to help simulation/planning, has been previously explored in formal planning. Our work is the first to take this idea into the modern context of LLMs, and show its benefits.

> Can causal abstractions really be generated from different environments or tasks, and if so, do they still provide benefit?

Yes, such generalized causal abstractions can indeed be generated: Our meta-memory (abstracted over past trials in different tasks or environments) contains such abstractions. For example, from the causal rules (1) "Using the lighter on the metal pot should be necessary to heat the water" and (2) "Turning on the stove should be necessary to create heat", CLIN generates the abstract rule "Using a heat source (stove, lighter) on the container should be necessary to heat a substance" (see Figure 3). Empirically, we do see that meta-memory provides benefits in novel tasks, both immediately and after further learning (Figure 5a and 5b, the with-meta-memory red line is above the blue line).

> More support showing CLIN can generate generalized insights where common rules may be hard to find useful for individual tasks such as ALFWorld.

Indeed, tasks in ALFWorld require specific insights such as "going to fridge 1 is necessary to find the tomato." However, in terms of complexity and task type, the Pick&Place tasks in ScienceWorld subsume most of the ALFWorld tasks, and agents (e.g., ReAct) that perform well on ALFWorld perform considerably poorly in similar tasks in ScienceWorld. On the contrary, CLIN achieves perfect scores in such tasks during generalization.

As a prototype for generalized memory in ALFWorld, we accrue two successful trajectories of Pick&Place tasks, and generate a meta-memory using CLIN's causal abstraction for a target task: "Find two remotecontrols and put them in armchair":

1. Going to the drawers may be necessary to find the remotecontrols.
2. Opening the drawers should be necessary to access the remotecontrols.
3. Taking the remote controls from the drawer should be necessary to obtain the remotecontrols.
4. Going to the armchair 1 should be necessary to place the remotecontrols.
5. Putting remotecontorls in/on armchair 1 should be necessary to complete the task.

We manually compared the correctness of such generalized insights with the underlying PDDL file that spawns the task and found that the memory captures most of the necessary (causal) information needed to solve the target task, indicating CLIN-style meta-memory will be effective in ALFWorld-like environments too.

> Related work: AdaPlanner and "Reason for the Future, Act Now"

Thank you for pointing out these interesting works, we will add them! As points of contrast, AdaPlanner's primary strength is adaptation/self-refinement of a plan during execution, rather than continual learning between plans/tasks. It does have a skill memory of successful plans, but does not abstract over them nor "learn" from failures beyond a datapoint. "Reason for Future, Act for Now", a concurrent work, is also interesting: they propose a memory-based approach to algorithmically reason over possible hypotheses for the next goal and action over a longer trajectory. However, they also do not persist insights in one trajectory to generalize over unseen tasks or environment – a key difference between recent works and our work.

As a prototype, we implemented a version of AdaPlanner for Pick&Place tasks in ScienceWorld, similar to those it has been predominantly successful in ALFWorld. On such tasks, AdaPlanner achieves an average score of 38.3, slightly lower than ReFlexion (40.7), whereas CLIN (our agent) achieves a perfect score of 100 in all such (Pick&Place) tasks.

Just looking at the task description and the initial state of the environment is not enough to produce grounded causal rules

Meta-memory when only the task description and the initial state are provided (Note ScienceWorld is a partially observable world)

NEW TASK: Your task is to grow an orange. This will require growing several plants and then being crosspollinated to produce fruit. Seeds can be found in the bathroom. To complete the task, focus on the grown orange. This room is called the bedroom. In it, you see: the agent, a substance called air, a bed. On the bed is: a mattress. On the mattress is: a white pillow, a book shelf (containing A book (Moby Dick) titled Moby Dick by Herman Melville, A book (Pride and Prejudice) titled Pride and Prejudice by Jane Austen, A book (Sherlock Holmes) titled Sherlock Holmes by Arthur Conan Doyle), a closet: The closet door is closed, a drawing, a table: On the table is: nothing. You also see: A door to the hallway (that is open).

1. Air may be necessary to grow an orange.
2. The bed does not contribute to growing an orange.
3. The mattress does not contribute to growing an orange.
4. The white pillow does not contribute to growing an orange.
5. The book shelf may not contribute to growing an orange.
6. The books on the shelf do not contribute to growing an orange.
7. The closet may be necessary to store tools or materials for growing an orange.
8. The drawing does not contribute to growing an orange.
9. The table may be necessary to place the plant pots for growing an orange.
10. The door to the hallway may be necessary to move between rooms and gather necessary items for growing an orange.

Without any interaction with the world, it could only generate trivial insights like the above that describe whether each object it observes at the start of the trial "is necessary/ does not contribute" to the task. Unless the agent navigates to an appropriate room, it won't see objects like "seed jar", "flower pot", "jug", "bee hive" etc. that are essential to complete the task. On the other hand, CLIN's meta-memory is composed of much more informative insights based on interactions with the environment from its previous experience on growing orange in different environment configurations. The learnings below mention essential objects, relevant actions, and how they "are necessary / do not contribute" to the task and subgoals within the task.

1. Looking around may be necessary to identify the objects in the room.
2. Moving to the appropriate room should be necessary to find the seed jar.
3. Picking up the seed jar should be necessary to start the process of growing an orange.
4. Moving to the greenhouse should be necessary to plant the seeds.
5. Placing the seeds in the flower pot should be necessary to plant the seeds.
6. Picking up the jug may be necessary to water the seeds.
7. Filling the jug at the sink should be necessary to get water for the seeds before pouring it into the flower pot.
8. Pouring water from the jug into the flower pot should be necessary to water the seeds.
9. Waiting may be necessary for the seeds to grow into an orange plant.
10. Repeatedly waiting may contribute to the growth of the orange plant.
11. Opening the bee hive may be necessary to crosspollinate the plants.
12. Repeatedly waiting after opening the bee hive may contribute to the growth of the orange plant.

CLIN produces generalized exploration bias, self-adapt if necessary, the novelty of our framework. When there is a huge search space for possible actions, any attempt to focus on desirable (part of the) trajectories (or memories) to glean relevant insights can be seen as a necessary exploration bias to perform well in unseen tasks/environments.

> How does the system handle potential changes in state that are not immediately reacted to an action?

These do indeed occur in ScienceWorld, e.g., water changes to vapor multiple several steps after the action "activate stove" with a pot of water on it. In this case, the actions "wait" or "take the water temperature" may be executed. CLIN learns through experimentation that some tasks require such actions, and such learnings may then be added to memory (e.g., when growing plants, "waiting may be necessary to allow the plants to grow", Figure 10).

> How does the executor modify an action when it doesn't align with valid actions?

If the executor generates an invalid (unexecutable) action, a simple check detects this and the executor is iteratively called again (with a note not to generate the invalid action) until a valid action is produced (or a max number of attempts is exceeded). While refinement, the executor is still conditioned on the intended goal provided by the controller. This utility essentially implements a self-refine technique Madan et al., 2023 to address this problem in the executor module.

Thank you for taking the time to review our paper! We were happy to read that you appreciated our main points: that our work demonstrates a novel learning methodology with continual learning and causal memory, that our experiments are extensive and rigorous, and that our work is well-presented.

> CLIN carries forward context to new experiences. Will misleading context affect performance?

In the short term, yes: errors in the memory will affect performance. However, note that CLIN also recovers from such errors, as the memory is updated after each trial, thus evolving over time - this dynamic, evolving memory is a key contribution of the work. Specifically, if a causal rule is wrong or unhelpful in a given trial, then CLIN will drop or update it during the memory update process, after it is observed to be ineffective. We see exactly this phenomenon in Figures 5a and 5b: In the same environment/task, CLIN improves (blue line), but then if the environment/task is changed, its performance drops as some learnings in memory are too specific (start of red line vs. end of blue line), but it then later recovers and improves even further (end of red line).

> Is CLIN's ability to self-explore limited by known information?

CLIN's memory both guides CLIN towards a solution (via the "is necessary for" links), but also explicitly encourages exploration if earlier attempts fail (via the "does not contribute to" links). For example, if in trial T, action X did not contribute to goal Y, this will be noted in memory ("X does not contribute to Y"), causing CLIN to explore a different action X' in the next trial. In this way, CLIN maintains a balance between exploration and exploitation, depending on the success and failures in earlier trials.

> What are the significant breakthroughs in CLIN?

There are two significant advances in CLIN:

1. While the idea of having a memory is not new, our novel contribution is to make this a dynamic, evolving memory over time, in contrast to the short-term "reflect, use, then discard" approach used in Reflexion and other memory-based agents. This allows CLIN to progressively build insights about the world over multiple experiences in different environments. It also allows CLIN to recover from bad or not-useful learnings in memory, as these will get dropped when they are found not to help as the memory is updated.
2. The role of the memory is itself novel, namely to help the agent mentally simulate the (learned) causal effects of actions, thus making more informed action choices in the future. This was missing in prior work, where memories were either task-specific factual instructions (e.g., "I should go to desk 1") or general hints ("Think about an object's usage when searching for it."). The idea of learning the causal effects of actions ("action models"), to help simulation/planning, has been previously explored in formal planning. Our work is the first to take this idea into the modern context of LLMs, and show its benefits.

We have updated the Introduction in the revised paper to clarify these better.

> How does the system handle tasks where the relations are not causal or obscure?

In general, non-causal knowledge has a causal import, which we observe is then captured in the memory. For example, "the block is in the workshop" would be captured by the causal expression "moving to the workshop may be necessary to find the block." In general, causal abstractions indirectly capture factual knowledge relevant to the tasks being considered.

There are definitely other kinds of relations that are required for ScienceWorld -- like retrieving property knowledge. An agent needs to know the melting points of objects, the needs of living things, the affordances of objects, etc., which are critical to solving the tasks. We empirically show that our agents are achieving perfect scores (add numbers) in many of these tasks requiring apparent non-causal knowledge. However, qualitative analysis reveals that CLIN is largely able to convert such knowledge with the structured causal import and effectively use it for completion of the tasks.

> How robust is the system when facing incorrect or misleading information stored in its memory? Is there any mechanism in place to correct or override these inaccuracies?

Yes - in fact, a key contribution of CLIN is its ability to correct errors in its memory via its dynamic evolution over time, unlike prior work. The memory is always updated after each trial. If the latest trace indicates some memories are unhelpful or simply wrong, they will not be retained in the updated memory, given the LM is following its prompt-based instructions correctly. Please see our new results in the "General Response to Reviewers" (also Appendix C, Table 2 in the revised paper) on CLIN's recovery from wrong knowledge in its memory via continual learning.

> What happens if the world follows some new logic, and does the causal abstraction generation still be correct?

This would be problematic! CLIN avoids thousands of trials (e.g., in RL) by leveraging the LLM's commonsense, so in a "nonsensical" world, this would not be effective. In fact, this is the key "breakthrough" insight of language-based agents: that with modern LLMs, commonsense can replace blind trial-and-error in real-world-like environments.

> If the environment dynamics change and old causal relationships are wrong, does the agent know when to update and unlearn the original knowledge?

Yes - the memory is updated after each trial, and as causal insights learned from past experiences no longer apply, or are found now to be incorrect, the LM will adapt or drop them based on the information provided in the latest trial. For example, CLIN learned that "activating the stove is necessary to boil a substance" but then encountered a trial with a broken stove. It subsequently updated that rule in the memory update to "activating the stove does not contribute to boiling a substance when the stove is broken." Again, this update process is not perfect, but our experiments show the memory dynamically evolves to a high-performing result, an exciting and novel finding.

Thank you for taking the time to review our paper! We were happy to read that you appreciated our main points: that CLIN can continually gain new knowledge based on online interactions with the world, that CLIN can abstract the world model (actual physics) of a new environment, and that our experiments are sound.

> How novel is CLIN? (the idea of having a memory of past experiences is not new)

There are two key novelties:

1. Indeed, the idea of having a memory is not new. Our novel contribution is to make this a dynamic, evolving memory over time, in contrast to the short-term "reflect, use, then discard" approach used in Reflexion and other memory-based agents. This allows CLIN to progressively build insights about the world over multiple experiences in different environments. It also allows CLIN to recover from bad or not-useful learnings in memory, as these will get dropped when they are found not to help as the memory is updated.

2. The role of the memory is itself novel, namely to help the agent mentally simulate the (learned) causal effects of actions, thus making more informed action choices in the future. This was missing in prior work, where memories were either task-specific factual instructions (e.g., "I should go to desk 1") or general hints ("Think about an object's usage when searching for it."). The idea of learning the causal effects of actions ("action models"), to help simulation/planning, has been previously explored in formal planning. Our work is the first to take this idea into the modern context of LLMs, and show its benefits.

We will make this clearer in newer versions of the paper and have updated the Introduction to clarify these better.

> Does the memory depend entirely on the agent's exploration trace? If the agent doesn't explore enough, can the agent gain the required knowledge?

Like any agent, the agent learns from exploration, so some required knowledge (e.g., a fridge is in the kitchen) necessarily needs exploration. If CLIN doesn't explore enough (e.g., it fails to find a fridge in the living room), it can gain the required knowledge in the next iteration by adding a memory: "going to the living room does not contribute to finding a fridge". This "negative knowledge" biases CLIN to explore in a different direction in the future, helping it gain the required knowledge for the task, and balance exploration and exploitation.

> Is CLIN just continually learning for a single task?

No, we applied and evaluated it in two settings: (a) single task ("adapt" setting), (b) multiple different environments (Gen-Env), and tasks (Gen-Task) ("generalization" setting). In the multiple-task setting, given a new (previously unseen) task, the meta-memory is first built from traces on different training tasks, and then applied to the new, unseen task (see Gen-Task in Section 4, and Figure 5c). For example, CLIN performs a handful of boiling tasks before solving an unseen freezing task. The fact it can perform well on unseen tasks is important and an important contribution - we will highlight this more in the paper.

> When generating causal abstractions, how does the LLM resolve the credit assignment problem?

Good question: By informed trial and error, using commonsense. When initially generating causal abstractions, the LLM is biased towards proposing commonsense abstractions vs. nonsensical abstractions (e.g., it is more likely to propose "opening the kitchen door may be necessary to move to the kitchen" than "looking around may be necessary to move to the kitchen"). Second, when the memory is updated, regenerating a new set of causal abstractions based on the old ones and new experiences, we rely on the LLM to follow its prompt-based instructions correctly, in particular, dropping memories that are unhelpful or simply wrong. Madan et al., 2023 show LLMs are good at refining their outputs, and Ma et al., 2023 show that LLMs are powerful reward generators. Mirroring these two insights, we find similar phenomena: for example, our LLM-based memory generator can connect an action (activate stove) and an observation (stove is broken) using its prior commonsense to propose a rule that activating stove when it is broken does not contribute to boiling the water. Of course, such proposals may sometimes be wrong, in which case they should later be dropped in subsequent trials when they hurt or are unhelpful. Please see our new results in the "General Response to Reviewers" (also Appendix C, Table 2 in the revised paper) on evidence that CLIN is able to recover from wrong knowledge in its memory via continual learning.