Learning Grounded Action Abstractions from Language

Thank you for your thoughtful review and positive comments on our work! We answer your questions and general clarifications below:

Are the planning assumptions realistic? The assumptions we make are standard for many grounded instruction following settings in prior work (including many of those that use the ALFRED benchmark, recent work that uses LLMs to interact with the Minecraft domain through the implemented Mineflayer API, like our Voyager baseline.) That being said, our approach should actually be applicable to environments under uncertainty. The symbolic operators that we learn do not need to assume access to perfectly downward refinable predicates (and in fact, in ALFRED, as we discuss in 4.1, the symbolic predicates are not actually a perfect model of low-level geometric detail in the environment) -- we give this formulation because it helps clarify the optimal learning case. The low-level policy learning technique we use is also a general one that is applicable to arbitrary stochasticity in the underlying environment as well. Thanks, and we will update our paper to clarify this!

Can the proposed method be also applied to some offline scenarios without these interactive environments? And is there any efficient approach to this? This is also an interesting idea that we can discuss as a direction for future work. Our paper is focused on the online RL learning setting (and uses a standard formulation for online policy learning), but our formulation could be adapted to use offline data in several ways and we can discuss this in future work. For instance, we might easily imagine initializing our low-level policy learning algorithm with direct supervision on offline expert demonstrations and state transitions. However, our approach is primarily geared at the online learning setting because we think it's much more interesting -- if you had a set of offline annotated expert demonstrations, there might be less need to use language as a prior to learn good action abstractions.

The detail of the baseline in Table 1 is unclear. For example, for "Code Policy Prediction," the authors prompt the LLM to predict 1) imperative code policies in Python and 2) the function call sequences with arguments. What is the modification made for each? Thank you for your feedback -- we will update our paper to try and further clarify this baseline, though as it is intended to reimplement key aspects of the Voyager model, we also refer to that for details. Following that paper (and adapting it to be as close as possible to the learning setting in our work), the code policy prediction model also attempts to learn and refine a library of skills in two stages -- these stages just involve Python code (instead of planner-compatible PDDL operators) and use the LLM as a planner instead of Fast-Downward + Policy search. In stage (1), much like our task decomposition step, we condition on the lingusitic goal and ask the LLM to produce the names and the symbolic code definitions of functions with arguments that define composable policies (eg. for picking_up_wood as part of an overall plan to make an axe). Then, in (2), as in Voyager, for each specific task, conditioned on the linguistic task description and a full symbolic description of the initial environment state, we allow the LLM to import these previously defined functions (or define new ones on the fly), and produce a plan as a sequence of grounded function calls with their arguments. We execute this sequence of grounded function calls to see if it satisfies the goal to score task accuracy on the benchmark.

Let us know if we can provide further clarification, thank you!

Thank you for the detailed review! To clarify and answer your questions:

In the results section, "How does or approach compare to using the LLM to predict just goals, or predict task sequences?", can you call out the specific parts of the proposed algorithm that address the limitations observed in the baseline approaches? The section you name (S4.1) does explain why each of these two baselines underperforms relative to our model. However, in revision we can use the additional space to re-iterate how our approach specifically addresses these ablations. To briefly summarize these points here:

• Low-level planning only: As defined, this baseline has no hierarchical planning at all: it directly attempts to search over the low-level action space towards the LLM-predicted goal. This shows that the problems are, indeed, too long-horizon to make direct low-level search feasible. The fact that our approach predicts operators at all -- decomposing an otherwise infeasible long-horizon goal into higher level components that can be searched for sequentially with associated learnable policies -- is what makes hierarchical planning with 'good' operators better than this approach in general; our performance on these benchmarks shows that we do in fact learn 'good' operators.
• Subgoal-prediction: As we point out in the results, the fact that the LLM cannot itself accurately predict the sequence of subgoals showcases the value of the explicit high-level planning algorithm that we are able to leverage by learning planner-compatible operators. In our results, we also provide additional qualititative analysis showing that the LLM struggles to accurately track environment state. Our approach uses a symbolic planner at the high-level that definitionally tracks this state and finds satisficing plans.

How would results change with different LLMs? Please see our general response for new results using GPT-4. As we show there, our model does not substantially change with GPT-4. We also re-run the Voyager baseline with GPT-4, as this is the closest baseline to our own work, and the original paper specifically uses GPT-4. We find that it does not close the performance gap between that model and ours.

How much noise can the system handle? Please see our general response for results from three different replications on all experiments. Overall, we find that our proposed method consistently outperforms all baselines across replications.

How do results compare with hand-coded action abstractions? Please see our general response for a post-hoc analysis of the learned operators. The ALFRED benchmark provides hand-coded operators for the dataset. Our model recovers all of the ground-truth operators (modulo one slightly underspecified case on the Slice operator that we discuss).

How often are the plans our system learns suboptimal? Our high-level planning algorithm is FastDownward, so it searches for optimal plans at the high-level given the operator set and goal. Our low-level planning algorithm is a breadth-first planner that also will not take unnecessary actions. This is a benefit of using structured planners -- we should never take 'valid but unnecessary' actions! We do qualitatively find that our other baselines, which use LLMs to predict sequences of actions (like the code-policy baseline), do not have this guarantee and can take unnecessary actions. We will highlight this!

How sensitive is our system to task ordering? Please see our general response on the randomized replications -- we use random task orderings, and our system learns robustly regardless of task ordering. This is because the algorithm itself is agnostic to task ordering. At each iteration, we always first propose operators in parallel, and evaluate operators on a per-task basis over the entire training dataset. Like other library-learning algorithms (eg. DreamCoder), we effectively recover our own 'curriculum', because our system can solve 'easier' problems first as it encounters them -- if it has already learned the right operators -- and then return to more challenging problems at the next iteration.

Dear Reviewer ikxW,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Thanks for the thoughtful review and close read -- these are all great clarification points. To answer your questions and provide clarification on your comments:

Can you provide statistics on task and plan complexity (high and low-level plans) for your benchmarks? Great question, and we will update our Appendix to include these numbers:

• On ALFRED, high-level plans compose 5-10 high-level operators, and low-level action trajectories have on average 50 low-level actions. There over 100 objects that the agent can interact with in each interactive environment. We refer to the ALFRED paper (https://arxiv.org/pdf/1912.01734.pdf) for additional details on this benchmark.
• On Mini-Minecraft: mining problems have 2-4 high-level actions and on average 6.9 low-level actions; crafting have 4-6 high-level actions and on average 10.24 low-level actions; and compositional tasks have 2-26 high-level actions and on average 41.5 low-level actions.

How do we identify and discount semantically similar (redundant) operators from being added in the operator library? Also a good question, and one that helps distinguish our work from other skill learning papers (eg. Voyager). We avoid problems with redundancy in two ways:

• First, the off-the-shelf high-level planning algorithm we use (FastDownward) is designed not to incur any additional computational cost for redundancy in the operator set. This is one major advantage of using a planning-compatible operator representation, rather than general purpose code -- we can leverage existing planners like this one.
• Second, however, in general we don't want to include semantically redundant operators for library interpretability. We do this by incrementally growing the library as needed during learning -- we always attempt to plan first with the highest scoring operators, then only try out new operators if planning fails (indicating that we may need a new skill). Wewill update the manuscript to clarify this important detail. Empirically, as you can see in our Appendix (A.2, Learned Operators), this means we don't learn redundant operators (and as we discuss in the general response, on ALFRED we in fact manage to just recover the ground truth hand-designed operator library without redundancy).

How do we avoid learning operators that are merely 'feasible' but not 'useful' if the overall goal success is not included in the operator score? Also a good and subtle point that we will highlight in our revision. This falls out of how the high-level planner algorithm works: it is an optimizing planner that searches for minimum-description length plans that only include operators which are useful for achieving the goal specification. As a consequence, operators are only used in plans in the first place if they are 'useful' towards the predicted goals; they will only be verified at the low-level and retained in the library if they were both useful and achievable in the low-level environment.

Minor Comment: Sec 3.4: s/b > \tau_r: Thank you for pointing out the typo. We will fix the typo in the updated version of the paper.

Please let us know if we can provide any additional clarification! Thanks again.

Thank you for your thoughtful review. We address your comments and questions here:

How is our model different from Voyager?

• First of all, we actually compare to Voyager, which is reimplemented as one of our baselines (the 'code policy prediction model', which uses an LLM to propose a library of Python-based policies that it retrieves and composes to solve tasks). We find that our model dramatically outperforms this model on both the Mini Minecraft and ALFRED benchmarks. Our discussion includes a failure analysis of this model and highlights several reasons why we outperform the Voyager implementation.
• Unlike Voyager, our algorithm uses the LLM as a prior over the task decompositions and action definitions, but nests the LLM within an outer learning objective designed to refine and verify the library of skills to learn a compact library tailored to the environment and task distribution. In contrast, we find empirically (as in the original Voyager paper) that the Voyager objective function tends to learn 100s-1000s of highly overfit skills that are locally relevant to specific tasks (eg. a function specific to mining two pieces of wood, rather than a more general mine_wood function). In practice, we find that this overfitting prevents generalization and efficient planning on more complex tasks, especially ones (like ours) that require adapting to novel environments. We suspect that the success of the original Voyager paper on Minecraft is due in part to the fact that it uses a relatively high-level, human-designed API that also appears in the LLM training data.
• Unlike Voyager, we also aim to specifically learn planning-compatible, grounded action representations that support existing hierarchical planners from the robotics literature for long-horizon planning. We show in our experiments that bridging between LLMs and hierarchical planners allows us to solve tasks and generalize to more complex tasks much more accurately than systems that rely on the LLM itself as a planner (consistent with other literature we cite in our related work whcih documents the difficulty of using LLMs direclty to predict plans).

Our new experiments also show that the gap between the Voyager model and our approach is not closed just by using GPT-4 for code policy prediction instead of GPT3.5.

Are these benchmarks too simple? While a key goal in future work is to show how this approach can scale to other domains (as we discuss in the general discussion), we would actually push back against the claim that these are very simple benchmarks for testing long-horizon planning from linguistic goals. Our baseline results demonstrate that both benchmarks (ALFRED and Mini Minecraft) are very challenging for a suite of different LLM-based planning methods, including the Voyager-based code-policy baseline. We see our strong performance on this benchmark as a good sign that -- in the optimal case in which we have access to fully specified linguistic goals -- the model can learn and use highly efficient action abstractions to solve the domain of tasks. As we point out in S4 (Domains), Mini Minecraft tasks are actually quite challenging planning tasks -- the longest crafting tasks in this benchmark compose over 26 high-level skills and have an action space of over 2000 potential actions at each time step, because we define the benchmark to permit the creation of many potential new objects during action execution.

How generalizable are methods (like ours) that use symbolic environment predicates in planning? Please refer to our general response for an answer to this!

Additional citations on planning and LLMs. We will update our related work section on prior LLM and planning work to cite the papers you mention; thank you for bringing these to our attention! To briefly discuss the contributions of our model in relation to these:

• [1] This paper considers a somewhat orthogonal technique (self-explanation by an LLM to fix initially proposed plans) for improving an initial LLM-produced subgoal sequence (using a basic method that is similar to the 'subgoal prediction baseline' we use in our work.) We agree that this is an important adjacent contribution to using LLMs for planning, with similar Minecraft and ALFRED-based evaluations, and will cite this work; the self-explanation technique to fix plans is likely one that could be integrated positively into our approach and any of the other baselines we report. This model does not seek to directly learn a composable library of hierarchical actions and does not seek to bridge between LLMs and structured hierarchical planning algorithms, which we see as the core contributions of this work, but does share a broader goal of decomposing long-horizon problems into subgoals.

• [2] This paper also uses an LLM to decompose long-horizon planning tasks (focused on the full Minecraft game) into goal specifications and subgoals, and (similar to Voyager) also uses the LLM to retrieve the full range of plans used in prior successful task executions (including to reference previously learned actions). We will also cite this in prior work. However, as we mention in our discussion of the Voyager model, we see the controlled long-horizon experiments (using our Mini Minecraft domain) and the need to learn under ambiguity (in our ALFRED domain) as highlighting the key contribution we make over these approaches, which use an LLM as an all-purpose policy proposal, decomposition, and retrieval model without the compact library-learning objective we frame in our work: as we find our experiments, we believe that planning-specific hierarchical representations and structured planning algorithms play an important role in accurately and efficiently using learned skills in general; and we believe that we contribute a general library learning objective for learning a compact, efficient skill library adapted to the planning algorithm at hand. This prevents learning 1,000s of arbitrary "skills" that may be accurate in the short term but overfit to the distribution of goals, and we believe is a key part of learning compositional, intuitive skills.

We hope this has been helpful -- please let us know if you have any additional questions!

Hi Reviewer ukwA,

Thank you again for taking the time to review this work. As the crux of your review seemed to hinge on novelty -- especially in comparison to Voyager, which we see our work as substantially improving over both in theoretical framing and empirical results -- please let us know if there's anything else we can help answer here to finalize or increase your score, with the review period closing shortly. Thank you!

Best, The Authors

Dear Reviewer ukwA,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Thank you for your thoughtful review and positive feedback about this work! We answer your questions below: How does the system ensure that the decomposition is optimal or near-optimal for complex tasks? Our operator-learning objective function explicitly optimizes for efficient and accurate planning (which we take as our definition of 'optimality') for each individual task and on the domain as whole. On any individual task, we use a hierarchical planning algorithm that searches explicitly for high and low-level plans that are accurate (solve the specified goal) and efficient (minimize the plan length, both in the number of operators used and the number of low-level actions taken overall.) We see one of the benefits of learning symbolic operators as precisely this ability to take advantage of efficient planning algorithms from robotics and decision making literatures. Our baselines demonstrate that the hierarchical planner is important, because ablating any part of the hierarchical planner (or substituting it with an LLM as the planner) dramatically reduces performance. On the domain as a whole, our scoring function over operators ensures that we learn operators which consistently support efficient and accurate hierarchical planning across the entire set of tasks, including simple and more complex tasks (like those in both of our benchmarks.) At both the individual task and domain level, our formulation allows us to use the LLM as a proposal mechanism for task decomposition and operator definition, but nests this proposal within a larger hierarchical planning loop to optimize for learning useful abstractions and producing optimal plans.

Could you elaborate on how the system currently ambiguities in goal specification? As we discuss in the paper, the human-annotated ALFRED benchmark contains many instances of ambiguity in goal specification: for instance, people may ask to slice a tomato and put it on a table when there are at least four possible concrete interpretations of which table is intended in the underlying benchmark (eg. the underlying benchmark distinguishes between the dining table, side table, coffee table, and kitchen counter). Currently, for our implementation and all baselines, we address this by explictly prompting the LLM to list N=4 possible interpretations of a given goal description, then attempting to plan towards all N of these interpretations (all results report the best accuracy out of these N=4 possible attempts). We also find empirically that directly prompting the LLM we use in our paper (GPT3.5) to produce N=4 distinct interpretations if ambiguities exist is more effective than sampling N=4 interpretations from the model posterior. This formulation suggests several ways to address ambiguity, and we will update our paper to address this. First, of course, we can simply take a greater number (N>4) of samples, to address cases where there are more than 4 interpretations (which is the case for many goals in ALFRED that have ambiguity in both the intended object and final goal location). More importantly, sampling explicit formal interpretations of each goal allows us to directly quantify uncertainty over the goal interpretation, as we can quantify how many unique interpretations there are (and their conditional probability given the linguistic goal under the LLM) for a given goal. In the future, as you mention, we could use this uncertainty score in interactive settings to decide when to ask a user for clarification, or to provide multiple possible concrete interpretations directly to a user for them to select the intended one. We will add this in our future work, thanks!

The paper demonstrates that action libraries from simpler tasks in Mini Minecraft generalize to more complex tasks. Are there plans to test this generalization capability in more diverse environments or tasks outside of the current benchmarks? The ALFRED benchmark also tests generalization amongst simple and complex tasks (the original benchmark includes tasks that compose multiple skills, such as the 'heated, sliced' or 'chilled and chopped' tasks that we mention in the paper). More generally, evaluating the relative generalization and compositionality of the learned operators is certainly a key part of our future work, and we will make sure that this is adequately highlighted as a potential strength of this approach, as the learned operators are designed to be composed/planned over via standard robotics planning algorithms (though, as we mention in the general discussion above and in our paper, the tasks we show here are already quite complex in terms of the actual long-range planning involved, as Mini Minecraft involves sequences of 20+ composed operators).

Hi Reviewer uRvH,

Thank you again for your detailed initial review -- we were encouraged by your positive response to the work! The feedback you gave here (eg. to clarify how goal ambiguity is handled, and to encourage comparison to other LLMs) has resulted in several new experiments that will definitely improve the overall work. With the end of the review period closing shortly, we just wanted to check in to see if there was anything else we could help answer here to finalize or increase your score.

Best, The Authors

Dear Reviewer uRvH,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

How accurate is each part of our model pipeline? (goal translation vs. operator learning) This is a good question, especially because our paper is primarily about learning libraries of grounded actions, not about parsing languge into formal goal specifications (which has been addressed in many other LLM+planning works). In our paper, we provide a qualitative failure analysis and mention that goal misspecification contributes to goal accuracy. We ran an additional post-hoc analysis to determine how each part of the pipeline (using LLMs to translate goals into formal specs; versus learning operators themselves) contributes to overall planning accuracy. We will report the following quantitative results for our model on ALFRED (as our model achieves 100% accuracy on the Mini Minecraft baselines). We also report these for the GPT-4 run to clarify the results we find above.

Goal translation on ALFRED:

• Our model (GPT 3.5): 92% of the time, the ground truth goal appears as one of the top-4 goals translated by the LLM.
• Our model (GPT-4): 82% of the time, the ground truth goal appears as one of the top-4 goals translated by the LLM.

Operator accuracy on ALFRED: To quantify operator accuracy, we manually compare the semantics of the learned operators with the ground-truth set of operators defined by the original ALFRED engineers.

• Our model (GPT 3.5): the ground-truth operator set contains 8 distinct operators corresponding to different skills. We learn semantically-identical (eg. same predicate preconditions and postconditions) for all of these ground-truth operators except for the Slice skill, which, as we discuss in the paper, is learned as two separate slicing operators, each using one of the possible 'knife' types to slice. See Appendix (A.2, Learned Operator Libraries) for all of these operators.
• Our model (GPT-4): this model also learns all operators in the ground truth set except for the Slice definition (and we do notice that, unlike GPT3.5, it actually only learns one operator using one of the two possible knife types to slice). As we also discuss above, however, we find that GPT-4 proposes a larger and more diverse initial set of operator definitions than GPT 3.5, and learning therefore takes longer to try, verify, and ultimately refine down the library to this operator set.

How well would the results compare to a system that uses hand-coded operators designed by a human engineer? On Mini-Minecraft, our model performs at ceiling (100% accuracy) after two iterations of learning. On ALFRED, the results above show that we also recover all of the ground-truth operators designed by the original ALFRED dataset engineers after learning (except for the fact that we learn two separate Slice operators instead of one).

How generalizable are methods (like ours) that use symbolic environment predicates in planning? The general scalability of symbolic planning operators is an important point that we will extend in our discussion. Our model does assume access to an initial set of symbolic predicates that describe the dynamics of an environment. This is a standard assumption in many robotics settings, and indeed, symbolic predicates (for high-level planning) that ground out into lower-level classifiers are a major part of many state-of-the-art Task and Motion Planning systems, motivating our formulation in this paper. Indeed, many of the other papers we cite that integrate LLMs and planners -- including the models we adapt for all of our baselines, and the Voyager model -- make this assumption in order to integrate symbolic planning and goal verification methods or to learn and execute code.

In terms of future scaling, however, we are especially optimistic about the prospects for integrating our algorithm with work that learns symbolic predicates from raw visual inputs and interactive experience (eg. Migimatsu and Bohg, 2022), suggesting a route for scaling our approach to operate directly over initial inputs even in settings with no initial predicate classifiers. We will update our paper to cite this line of work as a route towards scaling this approach to learn directly from perceptal experience.

Migimatsu and Bohg, Grounding Predicates through Actions, IEEE International Conference on Robotics and Automation (ICRA), 2022

How replicable are these results / how robust are the results to random task ordering, environment noise, and LLM sampling? We ran and will update the manuscript to show results from n=3 random initialized replications for our model and baselines. Replications test robustness to noise from the random task ordering during learning; environment noise (in ALFRED); and LLM sampling noise. In all cases, we find that our reported results are robust across all experimental conditions and see minimal differences across replications; we report standard error for all replications.

Mini Minecraft (n=3 replications)

ALFRED (n=3 replications)

What would happen if you used GPT-4 in your model? Would it improve the code-policy ("Voyager") baseline? We try running our model (and the Voyager-based model, as the original Voyager paper specifically mentions GPT-4) using GPT-4 to see if this improves results from the Voyager baseline. We find the following results. Mini Minecraft (GPT-4)

ALFRED (GPT-4)

We find that:

• For code policy prediction (the Voyager-based baseline), swapping in GPT-4 moderately improves performance on the ALFRED benchmark and makes essentially no change to performance on the Mini Minecraft benchmarks. Overall, the code policy model (with GPT3.5 or GPT4) performs far worse than our model.
• For our model, we find that using GPT-4 may slightly improve the robustness of our results on the mini-Minecraft results (we run 3 replications and see perfect performance after learning; in our replications reported above, the Crafting benchmark has a larger STE). On ALFRED, we find that using GPT-4 leads to similar but slightly worse performance than our reported results with GPT 3.5. Upon inspection, we find that this is because (a) GPT-4 is actually slightly less accurate at goal translation on this benchmark, as reported next, and (b) GPT-4 proposes a larger and more diverse set of initial operators, which takes more time to verify and leads to lower lifelong performance over 2 iterations of learning. We also see this 'greater diversity' in goal translation and operator proposal on minecraft. For learning, this diversity in operator proposals means our approach would additional iterations of learning (by the end of n=2 iterations, we do in fact recover all desirable skills on ALFRED). To address inaccuracy in goal translation, however, we suggest (as we mention in our paper) that in future work, rather than using general-purposed closed-source LLMs, we might instead seek to use LLMs specifically trained on domain-agnostic goal->PDDL translation for this part of the pipeline.