MaestroMotif: Skill Design from Artificial Intelligence Feedback

Thank you for your feedback!

There is not much discussion about how robust the system is to the quality of human feedback, and my guess is that the system could be quite brittle.

We believe there might be in a misunderstanding. The annotation effort is done directly by an LLM, in our case Llama 3.1 70b, and not by a human. (i.e., we use AI feedback, not human feedback). The task of converting from a language description of the skills to actual policies is therefore entirely carried out automatically, without any human annotations. We believe this is a strong argument in favour of our method, which makes learning skills significantly more scalable than previous approaches.

To better communicate this, we will modify the following sentence at L237 to clearly specify that the annotation is done from an LLM: "we mostly reproduce the protocol of (Klissarov et al., 2024), and use an LLM independently annotate pairs of observations collected by a Motif baseline." The annotation task itself in fact requires no human effort.

The reliance on pre-collected expert / exploratory trajectories to generate the skill-specific reward function is a limitation of the approach that is not discussed.

Please see our response below, where we answer in detail this point and its associated questions.

Results are limited to a single environment, and the authors do not discuss how this approach would work in other environments, where it might be useful or not

We presented the general methodology in section 3.1 and 3.2, and separated this from the specific choices made within NetHack. We believe (as highlighted by reviewer QXfj) that this methodology is fundamentally generalizable. We decided to focus on a specific domain as previous work did, to go deeper into the analyses of the behavior of our method.

At its core, MaestroMotif builds on Motif and on the capabilities of LLMs to write code. There already exists extensions of Motif in different domains, such as robotics (RL-VLM-F) and web agents (DigiRL), which future work could build on to apply MaestroMotif to a variety of domains. We are also witnessing weekly releases of improved coding abilities in LLMs, which further increases the possibility to apply MaestroMotif elsewhere.

The authors did not specify whether the code will be released publicly.

We will release all the material that can help reproducibility and future research. This includes all the code used for MaestroMotif in order to reproduce our experiments. We will also release the trained skill neural network, the learned reward functions and the LLM annotations/dataset.

MaestroMotif is first training the skills, which costs GPU time too. So we should ideally compare this training resources and sample efficiency to the one of training a policy for the final downstream task. For fair comparisons, the "RL from scratch" approaches should be given the same training time and environment interactions as is used for MaestroMotif pretraining; is that the case?

This is indeed the case, each of the task-specific methods (Motif, Embedding Similarity and RL w/ task reward + score) are given 4B timesteps during which they learn the task, which is exactly the same amount used by MaestroMotif to learn the skills. Notice that these baselines are given this budget for each of the tasks, whereas MaestroMotif, as the reviewer points, only uses it once.

I do get the argument that when targeting a new task, there is no additional cost for MaestroMotif and there is one for the "RL from scratch" approach, but this also assumes that the test task can be decomposed with the existing skills, which is conveniently the case here, but might not be in general.

As inherent to any hierarchical approach (e.g., SayCan) and stated in our paper, it is indeed true that a given set of skills will allow the agent to solve a specific family of tasks. Accordingly, we are not making the argument that a finite set of skills will solve any test tasks. However, we believe that MaestroMotif allows a way more efficient way of learning a new set of skills if the agent designer wants to tackle a new family of tasks. We think the problem of automatically growing the set of skills is very interesting, albeit challenging, and we are excited about the possibility of addressing it in future work.

Thank you for your feedback!

The proposed technique is quite general, being applicable across many domains outside of NetHack.

We are grateful that the reviewer sees the generality of the proposed method; we certainly hope to see future work extending such approaches to robotics and web agents.

The technique only trains reward functions on an existing offline dataset, which may be out of distribution with respect to the distribution of states encountered by the trained RL agent. This choice is borrowed from the original Motif paper (which said that the reward model was not fine-tuned during RL training due to simplicity), but based on the poor performance of the RL baselines (used to collect the original dataset) it seems like there may be a greater distribution shift in this work. Results determining whether the trained reward model generalizes to the online distribution would be appreciated (i.e. checking how much the annotator agrees with the reward model on trajectories generated by the final policy).

Note that, despite we are using an offline dataset, we take advantage of a highly diverse dataset of human data (please see Datasets and LLM choice in Section 3.3), which reduces the chances that the reward functions will get inputs that are completely out of distribution. Indeed, we believe that, if that was the case, even the skills trained by MaestroMotif, and not only the Motif baselines, would be hard to learn. The fact that MaestroMotif can reliably use the reward functions learned from the annotated dataset to train its skills is a strong sign of the adequacy of the data.

The technique relies heavily on the LLM's ability to generate code. However, even with unit tests and code refinement this seems to be difficult for many LLMs according to Figure 9. It is unclear from the paper whether the 405B model has achieved close to "perfect" performance or if there are still key issues with the generated code. Further analysis of the failure modes of the LLM coders would be useful for readers.

The performance of the 405B model is indeed not perfect, a quantitative indication being that exists a variation of performance, albeit small, across the seeds (we repeat the experiment from scratch a few times). When looking deeper into the generated code, we notice that it produces sometimes slightly different behaviour. Overall, these discrepancies are small enough such that, in the end, the skills are still learned successfully. We will include such code generations and discuss them in the paper.

MaestroMotif performs very well relative to all baselines, but still performs poorly in settings like Minetown, where only a 7.2% success rate is achieved. What are the bottlenecks to better performance? I.e. is the RL policy unable to complete the skills in this setting or is the generated "skill chooser" code bad for some tasks?

We believe the main reason why the sucess rate is bottlenecked mostly by the ability of the skill policies face a variety of situations in the game. NetHack is this incredibly complex domain that is procedurally generated, partially observable, and stochastic. For example, reaching Minetown first requires finding the staircase leading to the Gnomish Mines, which can be generated anywhere between levels 2 and 4. This would mean that the agent has to alternate between these levels and cover as much ground in each of them in order to find this specific staircase. During all this time, difficult monsters are likely to be generated (since the agent is no more in the safety of level 1), traps have to be avoided and hidden hallways have to be uncovered. To make further significant progress, we believe that a multimodal LLM, one that could see and understand such difficulties (which are not always apparent in the messages and the player statistics), would be key to obtain even more precise reward models to train more capable policies.

What architecture is used for the reward functions? Prior RLHF literature initialized RMs from smaller LLMs, but it would be interesting to see if different sized RMs would lead to different results from RL training.

The architecture for the reward model is the same one used in Motif, which is based on the encoder from the Torchbeast baseline (a relatively small neural network). We initially explored the idea using a larger reward model for more precise control (specifically we used both a BERT encoder or a Llama encoder) but we did not see a significant difference in performance at the time.

Thank you for your feedback!

Learning the options networks seems to require intense manual effort, as a human must label different states with preferences/task-alignment to learn different options.

We believe there might be in a misunderstanding. The annotation effort is done directly by an LLM, in our case Llama 3.1 70b, and not by a human. (i.e., we use AI feedback, not human feedback). The task of converting from a language description of the skills to actual policies is therefore entirely carried out automatically, without any human annotations. We believe this is a strong argument in favour of our method, which makes learning skills significantly more scalable than previous approaches.

To better communicate this, we will modify the following sentence at L237 to clearly specify that the annotation is done from an LLM: "we mostly reproduce the protocol of (Klissarov et al., 2024), and use an LLM independently annotate pairs of observations collected by a Motif baseline." The annotation task itself in fact requires no human effort.

Thank you for your feedback!

I wonder whether the performance gain primarily comes from the human domain knowledge or the method itself.

We would like to highlight the results of Table 1, in particular, the ones under "Task-specific training" and "Reward Information". All of these baselines receive human domain knowledge in different ways about the goals to be achieved. For example, the "Motif" baseline leverages the task information within a prompt given to an LLM to annotate pairs of transitions with preferences. These preferences are distilled in a reward function which an RL agent maximizes. The "Embedding Similarity" baseline similarly leverages a natural language description of the task to calculate a cosine similarity between the task description and the current state, which is then used as a reward to guide an RL agent. Finally, the "RL w/ task reward + score" uses the game score as a dense reward and a task specific reward that detects whether the task has been achieved or not. A particular highlight of these experiments is that each of these baselines are trained for 4B timesteps for each of the tasks separately, whereas MaestroMotif, after training the skills, adapts zero-shot to the task. These same baselines are also used within Table 2 for the composite tasks. Additionally, we investigated using an LLM directly as a policy ("LLM Policy" baseline in the paper) to achieve each of the tasks, giving information about the task at hand.

To complement the results in the paper, we further explore the LLM Policy and Motif baselines by giving all prior knowledge used within the MaestroMotif algorithm as context to a strong LLM (GPT-4). This prior knowledge include skill descriptions, high-level descriptions of the task and the generated code by the policy-over-skills that is used within MaestroMotif. We refer to this additional baseline as LLM Policy (Equivalent Prompting), which is inspired from related works such as [1]. An additional baseline we provide is to inject this same prior knowledge into the Motif baseline, by prompting the Llama 3.1 model expressing preferences with the skill descriptions, high-level descriptions of the task and the generated code by the policy-over-skills. We refer to this baseline as Motif (Equivalent Prompting). We present the results hereby. We notice that, despite the significantly more information given to both baselines, their performance does not improve. Although additional information is provided, the burden on how and when to leverage this information, from context, makes it very challenging.

Overall, we believe that this demonstrates the potential of the options formalism used by MaestroMotif for efficiently leveraging structure in complex environments.

[1] Nottingham et al., 2024, Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills

I'm not entirely convinced that the proposed five skills are general enough to tackle any downstream task in the NetHack environment

As inherent for any hierarchical approach (e.g., SayCan) and stated in our paper, it is indeed true that a given set of skills will allow the agent to solve a specific family of tasks.

However, MaestroMotif allows a significantly more efficient way of learning a new set of skills if the agent designer wants to tackle a new family of tasks. We think the problem of automatically growing the set of skills is very interesting, albeit challenging, and we are excited about the possibility of addressing it in future work.

Dear Authors,

Thank you for your effort in conducting additional experiments. However, I am afraid that I am still not fully convinced by the claim that the performance gain comes from the proposed method itself, rather than from human priors. In my view, the essence of this work lies in the decomposition. The ultimate task requires complex planning, so the proposed method decomposes the task into smaller components and trains each skill individually with the assistance of an LLM. Subsequently, a high-level LLM determines how to combine these skills to solve the overarching task.

This raises an important question: Who determines the appropriate decomposition? Isn't it the human? According to my understanding, humans select the set of skills—such as Discoverer, Descender, Ascender, Merchant, and Worshipper. Furthermore, the method requires additional human effort to refine prompts to properly learn each skill. If this decomposition were performed by the LLM rather than by humans, I would agree that the method is scalable and that the performance gain stems from the method itself.

If humans play such a crucial role in the process, I have doubts about the scalability of the method. For instance, let's say we have a different, imaginary video game environment called "XYZ_WONDERLAND." We want to apply MaestroMotif in this new environment. What would be the first step? Humans would need to play the game extensively to master and understand its core mechanics. Only then could they come up with an appropriate decomposition of the ultimate task into manageable tasks. If a framework requires lots of human efforts for individual target task, I regard that as less generalizable.

In the additional experiments, the authors claim to have provided equivalent human priors for all baseline methods. While this may be true in terms of the information provided to the systems, I believe they differ in terms of the explicitness of incorporating human prior knowledge. Did the baseline methods also undergo explicit task decomposition, as MaestroMotif did? I suspect they did not. Solving the task as a whole without proper decomposition may be challenging. Thus, decomposition is the key factor, and this is done by humans. A fair comparison would involve decomposing the task into the same five skill sets proposed in the paper, then using different methods (e.g., LLM-policy or Motif) to train each skill and combining them.

This leads me to question: What is the core contribution of this work? If the decomposition is performed by humans, is the contribution limited to learning each skill? But this is already accomplished using existing work such as Motif. Then, is the contribution in using a policy-over-skills approach? However, employing an LLM as a high-level strategy planner has been suggested in many prior works. In this sense, I struggle to identify the core contribution of this paper.

If I have missed anything important, please let me know. I'm really open to adjusting my score if my understanding is incorrect.