Motif: Intrinsic Motivation from Artificial Intelligence Feedback

Thank you for your feedback!

I think the point of this paper is that "joint optimization of preference-based and extrinsic reward helps resolve the sparse reward problems". As the source of feedback, either humans or LLMs are OK. I think describing this as LLM's contribution might be an overstatement. [...] As a preference-based RL method, I guess there are no differences from the original paper [1].

We never claim that an LLM's feedback is inherently better than the feedback coming from humans, even though we believe assessing whether that could be the case is an interesting avenue for future work. Instead, we simply leverage an LLM's feedback because of its scalability: in just a few hours on a small-sized cluster, one can annotate thousands of pairs of games events, which would require significant amounts of human NetHack experts's labour otherwise. This scalability, leveraged also in recent work on chat agents (e.g., Constitutional AI from Bai et al, 2022), allows for a method like Motif to fully leverage human common sense to bootstrap an agent's knowledge.

In the LLM literature, [2] leverages GPT-4 to solve game environments, and [3] incorporates LLM-based rewards for RL pretraining.

In [2] the Voyager algorithm uses complex prompt engineering involving significant amounts of human knowledge and engineering (such as deliberately prompting the LLM to use the “explore” command regularly). Additionally, Voyager relies on a Javascript API to bridge the gap between the LLM's language outputs and the high dimensional observations and actions of Minecraft. Finally, Voyager relies on perfect state information about certain features of the game (e.g. agent position and neighbouring objects). Voyager also builds on GPT-4's strong coding abilities, which would likely not be the case of current open models. Altogether, these factors strongly limit Voyager's general applicability and reproducibility. On the other hand, Motif relies on very limited human knowledge, being able to get significant performance even without any information about NetHack. Moreover Motif is way simpler to implement, with very few, clearly separated, moving pieces, providing a robust solution for leveraging prior knowledge from large models. This makes our approach a significantly more general method that has the potential to be applied to multiple domains or be possibly combined with powerful Large Vision Language Models.

We have additionally compared Motif to the ELLM approach of [3], adapted as described in the general response, showing that Motif significantly outperforms it in all NLE tasks. Please see the common response for a detailed description of the experiment. The paper also highlights in Section 5 (Related Work) the important differences between Motif and the ELLM algorithm. In particular, ELLM's reward function cannot, by design, exhibit the exploration and credit assignment properties of the one produced by Motif (see Section 4.2 of the paper, "Alignment with human intuition"). We believe those differences are key to the strong performance of Motif.

Terminology: I'm not sure if a preference-based reward should be treated as an "intrinsic" reward. I think it is extrinsic knowledge (from humans or LLM).

As standard, we refer to extrinsic reward as the reward that comes from the task to be performed in the environment, whereas intrinsic rewards are provided by the algorithm. From that point of view, the reward provided by the LLM is intrinsic (as opposed to extrinsic) to the agent. Please notice that this terminology has previously been used in the literature (see, for instance, the ELLM [3] paper).

Which RL algorithm is used for Motif? I may miss the description in the main text.

We use the asynchronous PPO implementation of Sample Factory (Petrenko et al., 2020). This information is available in the paper on the bottom of page 4. We chose this implementation as it extremely fast: we can train an agent on 2B steps of interactions in only 24h using only one V100 GPU.

Are there any reason why employ LLaMA 2 rather than GPT-3.5 / 4?

Yes, we believe there are important and significant reasons to prefer using Llama 2 rather than GPT3.5 or GPT4. GPT3.5 and GPT4 are subject to changes over time, require significant financial efforts to be used at scale, and rely on unknown methodologies and practices. Despite the fact that they might provide better performance, they are problematic for rigorous scientific reproducibility, and thus they are significantly less preferrable than Llama 2 for conducting scientific research. We explicitly made this decision for the benefit of the scientific community, and we will also release our code and dataset to ease experimenting with a method like Motif for other members of the community.

(Minor Issue)

We thank the reviewer for spotting the typo. We corrected it in the updated version of the paper.

Dear reviewer,

We believe our response addresses your concerns. In particular, we have now a total of 7 different baselines, including two additional baselines using language model-based rewards (sentiment analysis and ELLM), which are all significantly outperformed by Motif.

If there are any remaining concerns which prevent you from increasing your score, please let us know in the discussion as soon as possible and we will do our best to address them.

Thank you,

The Authors

Thank you for the detailed response.

My questions on (1) the difference between yours and voyager/ELLM, (2) the RL algorithm, and (3) the choice of LLMs become clear now. Here are the remaining comments:

Contribution:

The response to my concern seems to be reasonable, but not explicitly stated in the main text. So, the reader should infer the hidden intention why LLM is chosen rather than humans as the preference labeler (though alignment with human intention is emphasized many times). It is necessary to include your explanation (the scalability of LLM allows Motif to fully leverage human common sense to bootstrap an agent's knowledge, etc.) into the paper in this revision phase.

Terminology:

I cannot agree with your explanation because, in deep RL literature, the preference-based reward should be distinguished from "intrinsic motivation". For instance, The strong performance of Motif with "intrinsic reward only" in Figure 1 causes confusion as to why the agent can learn decent behavior only with "intrinsic motivation", since "intrinsic reward" often reminds the community of exploration. However, if the reader understands that Motif learns preference-based reward as a proxy for extrinsic reward, the results in Figure 1 will make sense. I express strong concerns about the description that equates "intrinsic reward" with a preference-based reward that should be a proxy for extrinsic reward and be learned from external knowledge.

Dear reviewer,

We hope our last answer addressed your remaining comments. Let us know if you have any remaining concerns, or if you could now recommend our paper for acceptance.

Best,

The Authors

Thank you for your feedback!

Can Motif be applied to other environments beyond the NetHack Learning Environment (NLE)?

We could not investigate this question in our current paper, as it is already compact with detailed analysis on the behavior, the risks of using LLMs and the possibilities for defining diverse rewards. By adding other environments, we could not provide such in-depth analysis.

We strongly believe that Motif is a general method, and it can applied to other environments after reasonably-sized efforts, when its main assumptions are satisfied. In particular, Motif's LLM needs to have enough knowledge about the environment, which is related to the presence on the Internet of text related to it, and the availability of an event captioning system. These assumptions are realistic in many environments, both when dealing with a physical system (e.g., a robot accompanied by a vision captioner) and a simulated/virtual world (e.g., a commonly-played videogames or Web browsing). Additionally, one could apply the general architecture of Motif to any environments based on visual observations, by just substituting a VLM in place of the LLM. We believe this is an exciting direction for future work.

To give some context on our choice of environment, NetHack is a challenging and illustrative domain to deploy an algorithm like Motif. Captions in NetHack are non-descriptive: they do not provide a complete picture on the underlying state of the game. Moreover, these captions are sparse, appearing in only 20% of states. This means that overall there is a high degree of partial observability. Despite this challenge Motif is able to thrive and show results that we have not witnessed in the literature previously.

We believe that if we were to apply Motif in other environments with more complete descriptions we could see even stronger performance. This would bring important questions to be studied: what exactly is the impact of partial observability on preferences obtained from an LLM? Do more detailed captions unlock increasingly more refined behaviors from the RL agent? Such important questions could be investigated by future work.

What a RL algorithm is used for RL fine-tuning?

We use the asynchronous PPO implementation of Sample Factory (Petrenko et al., 2020). This information is available in the paper on the bottom of page 4. We chose this implementation as it extremely fast: we can train an agent on 2B steps of interactions in only 24h using only one V100 GPU.

Thank you for you feedback!

The paper could benefit from a more extensive comparison to other methods, especially those that also attempt to integrate LLMs into decision-making agents.

First of all, we refer the reviewer Figure 7 in the Appendix F, in which we show that Motif outperforms four competitive baselines, including E3B (Henaff et al, 2022) and NovelD (Zhang et al., 2021), two state-of-the-art approaches specifically created for procedurally-generated domains such as NetHack.

In the updated paper, we have now additionally added a comparison to ELLM (Du et al., 2023), a recent approach for deriving reward functions from LLMs, showing that Motif's performance significantly surpasses such LLM-based baselines across all tasks. This is due to the peculiar features of Motif's intrinsic reward (e.g., its anticipatory nature), which, by design, are not implied by a reward function based on the cosine similarity between a goal and a caption. Our implementation is described in detail in the general answer above.

There is a lack of discussion on the computational cost and efficiency aspects of implementing Motif.

Due to our implementation being modular and asynchronous (that will be publicly released), dataset annotation is not especially expensive. Please see the general response for complete computational considerations. In addition, in Figure 15 we show that the performance of our method is particularly robust to the size of the dataset of annotations: Motif is able to outperform the baselines even with a dataset that is five times smaller (i.e., ~100k annotations).

While the paper makes a strong case for Motif, it doesn't delve deeply into the limitations or potential drawbacks of relying on LLMs for intrinsic reward generation.

We believe, as the reviewer does, that addressing limitations in LLM-based work is critical: this is why a substantial fraction of our paper is devoted to analyzing limitations and pitfalls of intrinsic motivation from an LLM's feedback. In particular, we want to highlight that we dedicated a full page of the paper to demonstrating evidence for, explaining, and characterizing misalignment by composition, a negative phenomenon relevant to our framework, whose emergence is a current limitation of Motif. In addition, we studied the sensitivity of Motif to different prompts, showing in Figure 6c that semantically-equivalent prompts can lead, in complex tasks, to drastically different behaviors. We believe this is a limitation of current approach based on an LLM's feedback, and hope that future work will be able to address it. Finally, we also included in Appendix H.3 a study on the impact of the data diversity of the dataset (through which we elicit preferences) and the resulting final performance.

Please notice that we are also very upfront about the fundamental assumption behind Motif, which is also a fundamental assumption behind the zero-shot application of LLMs to new tasks: that the LLM contains prior knowledge about the environment of interest. Our Introduction is centerered around this assumption.

Could the authors offer insights into why agents trained on extrinsic only perform worse than those trained on intrinsic only rewards?

Our paper already provides some insights in the "Alignment with human intuition" paragraph of Section 4.2, but we will now provide an additional perspective that can be beneficial to understand this result. By inspecting the messages preferred by the intrinsic reward function, one can quickly realize that the agent will receive from the LLM's feedback three kinds of rewards: direct rewards, anticipatory rewards and exploration-directed rewards. Direct rewards (e.g., for "You kill the cave spider!") leverage the LLM's knowledge of NetHack, implying a reward very similar to the score (i.e., the extrinsic reward). Motif's reward, however, goes beyond this. Anticipatory rewards (e.g., for "You hit the the cave spider!") implicitly transport credit from the future to the past, encouraging events not rewarded by the extrinsic signal and easing credit assignment. Finally, exploration-directed rewards (e.g., for "You find a hidden door.") directly encourage explorative actions that will lead the agent to discover information in the environment. Together, these three types of rewards allow the agent to maximize a proxy for the game score that is way easier to optimize compared to the actual game score, explaining the increased performance.

What's the best strategy to optimally balance intrinsic and extrinsic rewards during training?

We show in Figure 10c in Appendix that Motif is quite robust to how the two rewards are balanced. Broadly speaking, given that the nature of Motif's intrinsic reward brings it closer to a value function, future work can explore potentially more effective ways to leverage such type of intrinsic reward, for instance via potential-based reward shaping (Ng et al., 1999).

Thank you for your feedback!

Using a 70-billion LLM to generate a preference dataset from given captions is quite expensive

Due to our implementation being modular and asynchronous (that will be publicly released), dataset annotation is not especially expensive. Please see the general response for complete computational considerations. In addition, in Figure 15 we show that the performance of our method is particularly robust to the size of the dataset of annotations: Motif is able to outperform the baselines even with a dataset that is five times smaller (i.e., ~100k annotations).

while I understand this is out of the scope of the paper, perhaps using a large VLM to annotate frames without captions might have been more economical?

The question of whether annotations extracted from a VLM would be more efficient than the ones extracted from running an LLM on captions is interesting. Unfortunately, our experiments with current open VLM models suggest that none of them are able yet to interpret visual frames well enough to provide an effective evaluation or even accurate captions (most likely because current open models are predominantly trained on natural images). However, given the current pace of VLM research, this may change very soon. Thus, investigating the difference in the efficiency of various types of feedback will be an interesting avenue for future research.

In general, the question of large VLMs operating on images vs LLMs operating on captions brings interesting tradeoffs. Large VLM are more general, since they do not assume access to captions, but are faced with a more challenging task since they work with complex images rather than compressed text descriptions. From a purely computational standpoint, if captions are available and are of high quality then LLMs are likely more economical since their inputs are smaller.

it might be worthwhile having a baseline that gives preferences using a simpler model (say sentiment analysis) and learn the RL policy using this intrinsic reward model.

We added the results of an experiment using a sentiment analysis model as a preference model in the updated paper (Figure 8 of Appendix A.6). We use a T5 model fine-tuned for sentiment analysis, and extract, for each message, a score computed as the sigmoid of the confidence of the model in its positive or negative sentiment prediction. Then, for each pair in the dataset, we assign a preference based on the message with higher sentiment score. Finally, we execute reward training and RL training as with Motif.

Results on the score task show performance close to zero, both with and without extrinsic reward. This poor performance can be easily explained: a generic sentiment analysis model cannot capture the positivity or negativity of NetHack-specific captions. For instance, killing or hitting are generally regarded as negative statements, but they become positive in the context of killing or hitting monsters in NetHack. Llama 2 can understand this out-of-the-box without any fine-tuning, as demonstrated by our experiments. Also note that such a vanilla sentiment analysis model cannot be easily steered, thus losing any opportunity for the controllability offered by Motif.

To attest to Motif's strong performance, we also compared with an additional LLM-based baseline (as requested by Reviewer VWkY). The details of this additional experiment are presented in the common response above.

the paper mentions that the agent exhibits a natural tendency to explore the environment by preferring messages that would also be intuitively preferred by humans. Is this a consequence of having a strong LLM, or is it due to the wording of the prompt?

We believe this is due to the fact that the LLM was pretrained on massive amounts of human data, and then fine-tuned on human preferences. Indeed, even when using the zero-knowledge prompt presented in Prompt 2 of the Appendix B, Motif's reward function allows agents to play the game effectively even without any reward signal from the environment (see Figure 6b).