Aligning Agents like Large Language Models

We thank you for your review and feedback on our work, and we’re pleased that you found the core idea interesting.

Although the idea is interesting, the technique is too incremental, integrating IL for pretraining, IL for SFT, and REINFORCE for alignment.

Regarding novelty, we agree that the individual components of our work, and even the combination of imitation learning and reinforcement learning from preferences, is not novel, as discussed extensively in our related work in Section 2. However, there have since been many improvements to the basic pipeline of imitation learning followed by RLHF (introduced by [1]) in the context of large language models which have not been investigated back in the context of agents. Our main contribution is to apply this modern pipeline to a visually rich and complex domain with real human data, with the application of recent advances to improve the efficiency of the costly online alignment. For example:

1. Separation of imitation learning into pre-training on general data and fine-tuning on limited task-specific data

Pre-training on general text followed by instruction fine-tuning is essential for training helpful LLMs, but in the context of agents the usual procedure is to only imitate task-specific data. We demonstrate in Sections 4.1 and 4.2 that similar pre-training followed by task-specific fine-tuning is beneficial for task performance in our context for agents. We’ve also now added an additional ablation in Appendix D that demonstrates that the pre-training stage is beneficial over imitating the task-specific behaviour alone.

2. Use of Agent Model / Encoder for Initialising Reward Model

We demonstrate in Section 4.4 that utilising the agent model trained with an imitation learning loss as the initialisation for the reward model significantly improves the performance of the reward model and improves the feasibility of training a reward model with limited costly human feedback. While this is now typical for LLM reward models, the demonstration that this also applies in our case of reward modelling from pixels for decision-making agents across data sizes is a valuable contribution.

3. Introduction of Preference Fine-Tuning on Highest Reward Trajectories

We demonstrate in Section 4.5 that additional fine-tuning on a subset of the trajectories used for training the reward model with greatest reward significantly improves the efficiency of the subsequent alignment of the model with online reinforcement learning. This approach was concurrently introduced for language models [2], but is novel in the context of agents as far as we are aware. This is a big quality of life improvement for an end-user or game-designer that would like to align a (large) generalist agent, since fine-tuning on existing data is much less expensive than online reinforcement learning in a complex domain.

Regarding other potential directions for investigation that you mention:

Scaling Law. Whether model size scales, data scales, then the performance improves still holds for this setting?

Regarding the scaling of the model, we’ve also added additional results on how model size affects performance in our added ablation of the pre-training stage in Appendix D. As well as demonstrating the benefit of the unsupervised pre-training stage with supervised fine-tuning rather than training from scratch on the fine-tuning dataset, we found that larger model sizes (up to the 100M parameters we considered) are beneficial for our task even for training only on the more limited fine-tuning data.

In-context Learning. Whether preferences with some demonstrations in the model context can be captured? Longer model context. Currently, the context length is 32 timesteps. Compared to typical LLMs with at least 2k steps, isn’t it too small to memorize useful information in the context?

Our context corresponds to 32 previous timesteps as you mention. Since this is sufficient to determine the agent’s current location and motion, it is possible for the agent to learn our desired navigation behaviour with this context. While longer context could be interesting in more general settings given the partial observability of the environment, we note that this would also be significantly more expensive in terms of compute and memory for little gain, given that there is a lot more information contained within each image of context than there is in each word of context.

Investigating whether demonstrations could be incorporated in the agent’s context for in-context learning is an interesting idea, but out of scope of this work on efficiently aligning agents with preferences.

We thank you for your review and honest feedback on our work. We appreciate that you found the paper easy to follow, but would like to clarify our motivation and contribution to the community.

Weaknesses:

No human behaviour data is going to be released. The game environment is not going to be released. This makes the reproducibility of the results impossible.

Our main contribution is our analysis of applying the modern LLM training pipeline to such a challenging environment, which as you mention is interesting and different to those environments usually used by the community. On the other hand, as a result of being a real video game, there are many challenges and limitations of using Bleeding Edge as an environment, such as the hardware requirements (the game has to be run in real time on dedicated hardware for rendering) that make it unsuitable for release in its current state. As a result of these challenges, we do not claim to be providing a benchmark or dataset paper. However, we carefully provided our entire training procedure in Section 4 (and Appendix A) and all training details in Appendix B in order to make our results reproducible by practitioners looking to apply our procedure in other environments, even if it is not possible for them to be precisely replicated in Bleeding Edge.

To be even more concrete, in addition to our general procedure the main contributions from our analysis are as follows:

1. Separation of imitation learning into pre-training on general data and fine-tuning on limited task-specific data

Pre-training on general text followed by instruction fine-tuning is essential for training helpful LLMs, but in the context of agents the usual procedure is to only imitate task-specific data. We demonstrate in Sections 4.1 and 4.2 that similar pre-training followed by task-specific fine-tuning is beneficial for task performance in our context for agents. We’ve also now added an additional ablation in Appendix D that demonstrates that the pre-training stage is beneficial over imitating the task-specific behaviour alone.

2. Use of Agent Model / Encoder for Initialising Reward Model

We demonstrate in Section 4.4 that utilising the agent model trained with an imitation learning loss as the initialisation for the reward model significantly improves the performance of the reward model and improves the feasibility of training a reward model with limited costly human feedback. While this is now typical for LLM reward models, the demonstration that this also applies in our case of reward modelling from pixels for decision-making agents across data sizes is a valuable contribution.

3. Introduction of Preference Fine-Tuning on Highest Reward Trajectories

We demonstrate in Section 4.5 that additional fine-tuning on a subset of the trajectories used for training the reward model with greatest reward significantly improves the efficiency of the subsequent alignment of the model with online reinforcement learning. This approach was concurrently introduced for language models [1], but is novel in the context of agents as far as we are aware. This is a big quality of life improvement for an end-user or game-designer that would like to align a (large) generalist agent, since fine-tuning on existing data is much less expensive than online reinforcement learning in a complex domain.

Limited details are provided of the environment, such as the internal mechanism of the game, how it simulates the physics. These critical details are essential given the game is not widely known in the research community.

The environment we consider in this work is a real AAA video game, and so has relevant information publicly available [2,3] (including regarding the physics engine, PhysX [4]). We described what we believed to be the relevant aspects of the game for our research in Section 3, but have now also included links to these additional references in our paper, so thank you for highlighting that these additional details may also be of interest to the reader.

References:

[1] Reinforced Self-Training (ReST) for Language Modeling, https://arxiv.org/abs/2308.08998

[2] https://www.bleedingedge.com/en

[3] https://www.pcgamingwiki.com/wiki/Bleeding_Edge

[4] https://www.nvidia.com/en-gb/geforce/technologies/physx/

The task is not particularly challenging and focus on a niche mechanics in the game. The task is only focusing on a small part of the full game, with a maximum rollout episode of 100 timesteps (equivalent of 10 seconds of game play). The task is comparably simpler than a lot of the locomotive control problems in the simulated experiments, with only 16-dim action space.

No benchmark is established for the task. Please compare the result with some basic model-free or model-based RL agents. Even classical control algorithms should be able to provide decent benchmarks.

Similarly, we feel that we should clarify the motivation of our work here: the navigation task we consider is representative of a behaviour that an end user might want an agent to complete. This task necessitates our proposed procedure since there is no reward function available. Therefore, how can we efficiently train an agent to complete such a behaviour? Even with access to the reward model to provide a single reward per trajectory (which already requires a reasonable policy/data to train), it would not be possible to use any model-free / model-based / classical control algorithms as baselines since they would never reach the goal to receive signal given we are training in an open 3D world from pixels only. Therefore we must use limited task-relevant / demonstration data for imitation learning. However, imitation learning may not provide exactly the preferred behaviour and is not very robust online due to distribution shift, particularly from pixels, necessitating final alignment with online preference-based reinforcement learning to reliably achieve the preferred behaviour, as demonstrated by our results in Section 4.

We hope that our responses above have helped to clarify the contribution of our work, and that you can now better appreciate the value of our insights and contribution towards bridging the gap between the LLM and gaming agent communities.

We thank you for your very constructive and helpful review. We’re pleased that you appreciate the importance of the topic, the valuable contribution we have made to exploring this idea, and our thorough investigation.

To address your concerns regarding the scope of our analysis:

-Width: the authors have claimed they "investigate how the procedure for aligning LLMs can be applied to aligning agents from pixels in a complex 3D environment". Although I do agree that the environment can be visually perplexed, the task, however, might not be challenging enough. Is the goal hard to be recognized/identified due to the complextity of the 3D environment? Are there many distractions or interferences? Is the goal semantically rich therefore understanding it could become a challenge? Does the human preference over such goal go beyond simple multi-choice selection? Unfortunately, I find many of these aspects of interest when exploring pretraining - instruction-tuning - RLHF in game AI unchecked in this paper. Point is, this scheme has been verified in the language domain teaming with many of these aforementioned challenges and it actually work pretty well. Therefore, investigating this on a different domain (game AI), but without on par challenges, could be less illuminating to the community.

We believe that our task is sufficiently challenging, and more importantly, illustrative for our analysis. Since we are using a real AAA video game with real human data from visual-only states, obtaining any behaviour reliably is challenging. The navigation task we considered is complex enough to necessitate our procedure, since there is no reward function to use for RL (and the preference reward model is too sparse for tabula rasa RL given the 3D environment), meaning RL alone is not possible. Therefore we must use limited task-relevant / demonstration data for imitation learning. However, imitation learning may not provide exactly the preferred behaviour and is not very robust to online distribution shift, particularly when learning from pixels, necessitating final alignment with online preference-based reinforcement learning to reliably achieve the preferred behaviour, as demonstrated by our results in Section 4. The task also still involves all the complexities of real human data, with the full visual input and gamepad action output spaces.

As you highlight, this approach has demonstrated success in the language domain, but it is not trivial to apply it to agents in the game domain. In particular, there is significant offline to online distribution shift in our task due to character selection and visual modifications as we discuss in our new Appendix E, as well as the context (equivalent to prompting) that the agent is provided with at initialization as we discuss in Section 4.1. Additionally, learning from pixels and in an environment in which online reinforcement learning can only be performed in real time (due to game engine constraints) provides a much more challenging domain than language benchmarks. Our work therefore demonstrates what can be learned and applied from progress in the language domain, while highlighting additional challenges in the game domain, as in Section 4.5.

We thank you for your comprehensive review and helpful suggestions. We’re pleased that you appreciate the meaningful motivation, the multiple contributions of our paper, and our thorough experiments and discussion.

To address your concerns:

The paper shows that the same overall process for finetuning LLMs can also be applied to agents within this game. However, the overall method has limited novelty – as noted in the related works section, the general process of bootstrapping RL with imitation learning and further finetuning agent behaviors with reward models learned from human preferences have both been studied in prior work training RL agents outside of LLMs.

Regarding novelty, we agree that the individual components of our work, and even the combination of imitation learning and reinforcement learning from preferences, is not novel, as discussed extensively in our related work in Section 2. However, there have since been many improvements to the basic pipeline of imitation learning followed by RLHF (introduced by [1]) in the context of large language models which have not been investigated back in the context of agents. Our main contribution is to apply this modern pipeline to a visually rich and complex domain with real human data, with the application of recent advances to improve the efficiency of the costly online alignment. For example:

1. Separation of imitation learning into pre-training on general data and fine-tuning on limited task-specific data

Pre-training on general text followed by instruction fine-tuning is essential for training helpful LLMs, but in the context of agents the usual procedure is to only imitate task-specific data. We demonstrate in Sections 4.1 and 4.2 that similar pre-training followed by task-specific fine-tuning is beneficial for task performance in our context for agents. We’ve also now added an additional ablation in Appendix D that demonstrates that the pre-training stage is beneficial over imitating the task-specific behaviour alone.

2. Use of Agent Model / Encoder for Initialising Reward Model

We demonstrate in Section 4.4 that utilising the agent model trained with an imitation learning loss as the initialisation for the reward model significantly improves the performance of the reward model and improves the feasibility of training a reward model from pixels with limited costly human feedback. While this is now typical for LLM reward models, the demonstration that this also applies in our case of reward modelling from pixels for decision-making agents across data sizes is a valuable contribution.

3. Introduction of Preference Fine-Tuning on Highest Reward Trajectories

We demonstrate in Section 4.5 that additional fine-tuning on a subset of the trajectories used for training the reward model with greatest reward significantly improves the efficiency of the subsequent alignment of the model with online reinforcement learning. This approach was concurrently introduced for language models [2], but is novel in the context of agents as far as we are aware. This is a big quality of life improvement for an end-user or game-designer that would like to align a (large) generalist agent, since fine-tuning on existing data is much less expensive than online reinforcement learning in a complex domain.

We hope these three main examples highlight the novelty of our contribution in the context of aligning agents.

References:

[1] Reward learning from human preferences and demonstrations in Atari, https://arxiv.org/abs/1811.06521

[2] Reinforced Self-Training (ReST) for Language Modeling, https://arxiv.org/abs/2308.08998