LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models

We thank the reviewer for their feedback. We would like to address your questions and concerns by 1) further explaining our data generation process 2) explaining how to replicate the data generation process, and 3) directing your attention to Appendix D on hyperparameters. We address your comments point by point below:

“You use the data from the GPT-3.5 and create several games for the 8 tasks?

Yes, for our three dialogue tasks, we do the following: 1) Generate data with GPT-3.5 based on specific prompts (for an example for Twenty Questions, see Appendix, Section C.6) 2) Distill this to two GPT2 models one for the agent, one for the environment 3) Use these distilled models to generate a new dataset that we will use for training our algorithms. See Section 4.4 of our paper for further information on data generation and reference Figure 2.

“How to replicate the data generation process?”

The data generation process can be replicated by using the GPT2 checkpoints provided to regenerate the data. We also provide our prompts and data generation scripts in our code base, for which we have provided an updated version.

“How did you evaluate the quality of your data?”

We evaluate the quality of our data by 1) inspecting the generations for signs of problems 2) performing a human evaluation where humans interact with the simulator model that generated the data. If humans are able to successfully interact with the models, it is a clear signal that our data is also natural and contains the desired properties. Additionally, we have found better performance from our algorithms compared with the BC models, signaling that the data the simulator was trained on is providing a useful signal for the improvement of RL algorithms.

“Any validation of the data generation process? How to make sure the game/task you generate is POMDP?”

To validate the data generation process, we report statistics as can be seen in Table 1. We also provide further details on the dataset generated in Appendix C. For the text-games we ensure it is a POMDP, by creating environments such that the next state is only dependent on the previous state and action. For dialogue tasks, the state can be thought of as a concatenation of all previous utterances. With this definition, this situation will also be a POMDP.

“LLM with RL/PPO in this complicated task is very sensitive to the hyper-parameters. I did not find the analysis or report on these….Any details about the hyperparameters for baselines?”

We have reported our hyperparameters for training the baselines in Appendix D.

“The baseline results are not convincing enough. PPO Instabilities in GPT-2-small are very obvious. I do not think the results in the benchmark should only use GPT-2-small. Any technique you use here to overcome it?”

We overcome instabilities in PPO by 1) increasing the number of rollouts 2) tuning the KL coefficient and 3). Regarding using GPT2-small, we were limited by compute and simply cannot train 7B+ models on all these settings. However, for the models we use as the environment, we have used GPT2-XL for the Car Dealer task.

We thank the reviewer for their helpful feedback in improving the paper, and hope that you find these reasons convincing in raising your score!

We thank the reviewer for their feedback. We've addressed the main issues raised in your review by: (1) rewriting Section 4.1 of the paper to improve the definitions of the RL capabilities and how they apply to each task; (2) conducting a user study to test the naturalness of conversation from our simulators; (3) providing further examples of conversations from our trained models against the LLM simulator. We address your comments point by point below:

“While the state and action spaces are language-based, it is not clear the extent to which some of the environments assess performance on language-based tasks, such as the maze, chess, and chess endgames. While the environments assess reasoning abilities, they do not assess the full complexity of reasoning abilities a LLM needs nor abilities that are key to LLM success. The reasoning abilities are more general abilities we would want from a RL-train agent. For example, it is important for LLMs to be factual and harmless, which means correcting behavior learned during pretraining.”

It's definitely a valid point that not all the tasks test performance on realistic language-based tasks, though some do. However, each of the tasks in the benchmarks serves a different purpose. Some tasks (Guess My City, Car Dealer) aim to evaluate tasks with realistic natural language. Some tasks aim to test specific RL properties without the complexities of realistic language, while others focus on complex language. Algorithms developers would be expected to evaluate their methods on the totality of all the tasks. We've expanded Section 4.3 to clarify this. We do not believe that this is a weakness, as our benchmark does include a number of tasks with realistic language, and it seems valuable to test a variety of RL algorithm capabilities in isolation (and comparatively less valuable to have redundant tasks that all test similar things). However, we hope that the revised discussion in Section 4.3 more clearly explains this rationale.

“To what extent does the policy's actions on the environments like 20Qs, car dealer, and guess my city look like reasonable sentences?”

We find that all of the policy’s actions on the environment appear to be reasonable sentences. We have verified this by looking at the generated datasets and interacting with the environments ourselves, as well as performing a user study of having humans evaluate if the text used to train the simulator from GPT and from one of our models (MC returns) was natural on a scale of 1-5 (where 5 is most natural). On average, we found humans to say the text from GPT-4 was 55.56% natural, and from our model was 58.51% natural. We provide detailed statistics and explanations of the user study evaluating naturalistic language in our Appendix, Section J. We are also currently conducting a larger human evaluation study of humans interacting with our models as a baseline, and provide partial results in to Table 2 in our paper with some discussion. We also show sample interactions of our BC and MC policy model with the oracle in Appendix, Section H for the Car Dealer Task.

“Are there other LLM-specific policy learning algorithms that can be assessed?”

Yes, since our submission, we have performed a new evaluation experiment by few-shot prompting GPT-4. This is a LLM-specific policy learning algorithm because only LLMs can be few-shot prompted in this way. In addition, we have run an ablation with Online Filtered BC, which is an algorithm that collects data using the current policy and selects the most successful trajectories for fine-tuning. This is an algorithm that is often used in RLHF pipelines. In general, we have found that these evaluations did not outperform our previous baselines. We include these results in our updated results section.

“For some of the environments the specific skills addressed are called out. It would be great to highlight the assessed skills for each environment.”

We agree that it is important to highlight the capabilities enabled by every task. As per your suggestion, we have updated our draft with Section 4.2.1 and an additional Figure 3 to more clearly highlight which tasks enable which RL capability and why. For example, the Maze and Text-Nav tasks contain both partially observed and fully observed versions to highlight the impact of partial observability. In addition, the Text-Nav task is very similar to the Maze task, but has the added complexity of learning to generate realistic text.>

We thank the reviewer for their feedback. We've addressed the main issues raised in your review by: (1) including a comparison between few-shot prompting GPT-4 and our RL baselines; (2) rewriting our paper to more clearly articulate the capabilities being tested; (3) providing evidence in the form of a user study that our tasks reflect naturalistic language. We address your comments point by point below:

“An alternative to improve interaction and performance at tasks is to use prompting and factoring of LLM calls. I found a discussion on prompting vs RL training as alternatives to each other missing.”

As per your suggestion, we have included an additional experiment for few-shot prompted GPT-4 against all of the tasks. We have found that prompted GPT-4 was not able to outperform our baselines on the tasks. We have updated Section 6 of our paper with further details and discussion.

“The reasoning and generalization required on each task needs to be formalized more clearly. What inferences must the model make?”

In Section 4, we delineate the capabilities enabled by RL, including strategic decision making, complex language, credit assignment, partial observability, and trajectory stitching. We have added Appendix A to formalize the reasoning required for each task. To address your question, we have added Section 4.2.1 and an additional Figure 3 to more clearly highlight which tasks enable which RL capability and why.

“For complex tests, 20 questions, guess the city, car dealership where the model relies on GPT-3.5 for the initial test and GPT-2 for the env, there need to be evaluations of the text produced by the models.”

To address the concerns regarding the evaluation of text produced by the models, we have performed an additional evaluation of having humans evaluate if the text from both GPT and from one of our models (MC returns) was natural on a scale of 1-5 (where 5 is most natural). On average, we found humans to say the text from GPT-4 was 55.56% natural, and from our model was 58.51% natural. We provide detailed statistics and explanations of the user study evaluating naturalistic language in our Appendix, Section J. We are also currently conducting a larger human evaluation study of humans interacting with our models as a baseline, and provide partial results in our updated paper.

Additionally, we would like to clarify the methodology with which we generated our datasets to train our simulators. As shown in Figure 2, we train simulators that serve as an “oracle” for the task, which defines the ground truth environment. In most cases, our simulator is performing an easier task than the agent, which does not require strategic reasoning. Additionally, our tasks are designed such that any noise in our environment will not necessarily break the task, but merely add an additional inference challenge for the agent. For example, the role of the oracle in the Twenty Questions task is to provide objective yes/no answers to questions about the object, and in Guess My City, to provide more open ended information about a query on the city. OpenAI’s GPT-3.5 has been shown to be able to generate reasonable questions and answers when used out of the box, which is why we leveraged it to collect our initial dataset. We have provided prompts that we use to generate the data to train our oracle models in our Appendix, Section C.6, and snippets below to show our thought process to maintain high accuracy.

With respect to the Car Dealer task, we spent a considerable effort to ensure diversity in the responses of sellers, by providing different desired brands, features, classifications (i.e. car or truck), and budgets in our prompting to generate the datasets. We provide a sample conversation between the oracle model and MC returns vs. oracle and the BC model to illustrate in our Appendix, Section H.

“Why did the authors not choose to start with human data in domains where that was available? Eg: Craigslist dataset, Deal or No Deal”

Regarding why we did not choose to start with human data in domains that were available for the following reasons: 1) These datasets do not come with simulators that are faithful and produce data that is similar to the dataset 2) These datasets are not designed for RL and do not have clear and intuitive reward functions. Past works required various special choices to use them, for example, Verma et. al replaced the prices in the Craiglist dataset dialogue and used GPT to generate templates to substitute such prices, as they are not very realistic. Overall, we have found that these datasets do not encourage strong bargaining capabilities.

Works Cited:

[1] Verma, Siddharth, et al. "Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning." arXiv preprint arXiv:2204.08426 (2022).

We thank the reviewer for their feedback. We've addressed the main issues raised in your review by: (1) rewriting Section 4.1 of the paper to improve the definitions of the RL capabilities and how they apply to each task; (2) conducting a human evaluation study to test the naturalness of conversation from our simulators; (3) providing further examples of conversations from our trained models against the LLM simulator. We address your comments point by point below:

“The benchmark is artificial and quite less natural. Only two of the eight tasks (Car Dealer and Guess My City) is multi-turn conversation or conversational QA with more unbounded state and action space. All other tasks have obvious restrictions, especially, the Chess and Endgames tasks are not natural language generation to me.”

It's definitely a valid point that not all of our tasks have realistic natural language, though some do. However, each of the tasks in the benchmarks serves a different purpose. Some tasks (Guess My City, Car Dealer) aim to evaluate tasks with realistic natural language. Some tasks aim to test specific RL properties without the complexities of realistic language, while others focus on complex language. Algorithm developers would be expected to evaluate their methods on the totality of all the tasks. We've expanded Section 4.3 to clarify this. We do not believe that this is a weakness, as our benchmark does include a number of tasks with realistic language, and it seems valuable to test a variety of RL algorithm capabilities in isolation (and comparatively less valuable to have redundant tasks that all test similar things). However, we hope that the revised discussion in Section 4.2.1 more clearly explains this rationale.

“Generated examples by the trained agents can be added. Although the authors have put emphasis on “our goal is not to utilize this approach to benchmark whether LLMs are good at talking to humans, but rather as a way to test RL algorithms with datasets that are sufficiently difficult and complex so as to gauge how effective they might be if they were then trained on data from real humans.“ in Introduction, it is good to see how relevant the obtained rewards and the conversation's naturalness are.”

Thank you for this question. Rewards for the tasks are simple and intuitive, because they are based on success of selling the car or guessing the word. To address your concern regarding the naturalness of the conversation, we have provided a few examples in the Appendix, Section H. In addition, we have evaluated all of our tasks with human evaluators playing the games to assess how natural they are, and show an example in Appendix, Section I.

“While the paper claims to focus on LLM, the experiments are conducted on GPT2-small and medium (according to Table3), which may be controversial. I’m also confused by the contradiction in Section 7 and Table3. In Section 7, the authors say they “primarily trained and evaluated models with a maximum 1.5B parameters”, but Table3 shows the experiments are conducted on less than 355M parameters. Therefore, I’m wondering how large is the trained agent in Table2 actually?”

Thank you for your question. This is a valid concern. We trained one model (Car Dealer) with GPT2-XL, and missed putting this into Table 2, which is now updated. We were limited by compute and simply cannot train 7B+ models on all these settings. However, for the models we use as the environment, we use GPT2-XL.

“The writing of section 4.1 can be further improved. For now, I can guess the meaning from the terminologies themselves, but I will get confused from the content in each paragraph trying to explain the corresponding terminology. … Some details can be moved from Appendix B to section 4.3. For example, the desired capability for every task. For now some are left in the appendix.”

In order to make Section 4.1 clearer, we have improved the definitions of the RL capabilities. We have added Section 4.2.1 and an additional Figure 3 to more clearly highlight which tasks enable which RL capability and why.