LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models

We thank the reviewer for their feedback and questions. We would like to answer the questions in your review as follows:

1. "How does the dialogue tasks in the proposed benchmark differ from classical multi-turn task-oriented dialog dataset, such as MultiWOZ?"

The goal of our paper is to present a benchmark that applies RL algorithms to multi-turn tasks, specifically to perform goal-directed dialogue. To this end, we provide (1) online simulators and offline datasets for a suite of 7 text-based strategy games and dialogue tasks (2) methodology to create simulators for offline evaluation, online RL training, and computing rewards (3) a research framework and toolkit for researchers and practitioners to get started with multi-turn RL for LLMs (focusing on both online & offline RL), which includes implementations of PPO, ILQL, and several baseline methods. MultiWOZ primarily focuses on dialog and does not provide the ability to test algorithms on specific RL capabilities including trajectory stitching, credit assignment and partial observability. Additionally, they do not focus on both online and offline reinforcement training algorithms for LLMs. Please refer to response to Reviewer 7KJe for detailed discussion on other related works.

2. "Experiments are only performed on GPT-2 models, casting doubts on the scalability of the results and the validity of the proposed benchmarks on larger models" We focus primarily on GPT2-models as there is a good understanding of its capabilities and broad usage in research for establishing baselines [3] compared to newer, similar sized models. We would like to highlight several recent works [1, 2, 4, 5] that have also used GPT2 for fine-tuning. Our paper focuses on providing a framework that can be easily adapted to various models and the development of further algorithms for RL fine tuning for LLMs, and we hope to see further works that iterate upon other models.

[1] Hicke, Y., Masand, A., Guo, W., & Gangavarapu, T. (2024). Assessing the efficacy of large language models in generating accurate teacher responses. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). arXiv preprint arXiv:2401.12345.

[2] Hong, J., Dragan, A. D., & Levine, S. (2024). Q-SFT: Q-learning for language models via supervised fine-tuning. Proceedings of the 42nd International Conference on Machine Learning (ICML 2024). arXiv preprint arXiv:2403.01512.

[3] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9. 4o mini

[4] Zhou, R., Du, S. S., & Li, B. (2024). Reflect-RL: Two-player online RL fine-tuning for LMs. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). arXiv preprint arXiv:2401.12345.

[5] Zhou, Y., Zanette, A., Pan, J., Levine, S., & Kumar, A. (2024). ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL. Proceedings of the 41st International Conference on Machine Learning (ICML 2024), 235, 62178–62209. arXiv preprint arXiv:2402.19446.

3. "What is the benefit of formulating language generation tasks as a partially observable Markov decision process rather than the standard fully observable MDP?"

We formulate language as a POMDP, as the true state of the world is not completely represented in text form. In language tasks, the state consists of the entire history of tokens, and an agent may need to examine this entire context to infer the correct state. The mental states of a speaker in a dialogue (e.g., whether the buyer is impatient in a selling task), previously observed facts in a guessing game, and other hidden variables might induce partial observability.

4. "What are the benefits of the proposed tasks over math problem solving or code generation in testing RL training on LLM agents?"

We acknowledge that math problems and code generation are interesting tasks to test the capabilities of RL-LLM algorithms. However, we chose our text game tasks as they are similar in nature to traditional RL tasks, but with a twist of including language to test how this impacts the performance of RL algorithms for LLMs. This allows us to isolate traditional RL issues such as credit assignment and trajectory stitching but in a language-based setting. To that end we have designed five tasks as RL Capability Tests, which are text games designed to isolate specific capabilities of RL training as shown in Figure 4. As seen, these text-games do not test all of the capabilities of RL together, which is only possible through the dialogue-based tasks. Please refer to response to Reviewer RuRY on more clarification on our choice of tasks for the benchmark.

We thank the reviewer for their feedback and suggestions for related works. We will be sure to the cite the works that you have noted, and move material from our paper to the Appendix / supplementary section. To address the questions in your review, we would like to provide responses to the following:

1. "Task Design Issues": We would like to clarify that each of the tasks in the benchmarks serve a different purpose, and we would like each task to isolate different RL capabilities to develop better algorithms. Please refer to #2 response to Reviewer RuRY
2. "Related Work": We will be sure to cite the related works you have noted. Please refer to #1 response to Reviewer gYws on discussion of related works.
3. "Model Selection": Regarding model selection and usage of GPT-2, please refer to #1 response of Reviewer RuRY.

Regarding your question on analysis of failure cases and our models, we would like to clarify the methodology with which we generated our datasets to train our simulators, and how we ensured high quality and consistency for these datasets. As shown in Figure 2, we train a simulator that serves as an “oracle” for the task, and hence does not require any capabilities of strategic reasoning, but provides signals to help the agent model learn. For example, the role of the oracle in the Twenty Questions task is to provide objective yes/no answers to questions about the object, and in Guess My City, to provide more open ended information about a query on the city. OpenAI’s GPT-3.5 has been shown to be able to generate reasonable questions and answers when used out of the box, which is why we leveraged it to collect our initial dataset. We have provided prompts that we use to generate the data to train our oracle models in our Appendix, and snippets below to show our thought process to maintain high accuracy.

The method for collecting the dataset is as follows. For each conversation, we select uniformly at random from the above list the word that the oracle is answering question about. The oracle is an LLM (OpenAI’s GPT3.5) given the following prompt. In our prompts, we denote variables that we fill in with variable data with {{variable}}.

Prompt: You are a question answering oracle. You will answer each question about an object with Yes or No. If the answer could be both, answer with the most typical scenario. Here’s a few examples:

example 1:

object: Computer

question: Does the object use electricity?

answer: Yes.

explanation of answer: Computers need electricity to function. [...]

Additionally, we have also validated the data from trained oracle models through human evaluation. We have also provide generated examples by both oracle models and trained agents in our Appendix. With respect to the Car Dealer task, we spent a considerable effort to ensure diversity in the responses of sellers, by providing different desired brands, features, classifications (i.e. car or truck), and budgets. We have provided samples of conversation between the oracle model and MC returns vs. oracle and the BC model in the Appendix.

We thank the reviewer for their feedback and suggestions for clarity on the work. We will revise our figure and term accordingly as per your suggestion. We've addressed the main questions raised in your review by: (1) providing an extensive literature review of other popular benchmark papers (2) clarifying our methodology to generate data to train our simulators (discussed in Reviewer gYws response) (3) clarifying your question on use of FLAN-T5-XL and GPT2-XL (discussed in Reviewer RuRY and Reviewer MQ7V response).

Related Works: In order to clarify the contribution of the paper, we provide comparison to popular related works in text games, interactive dialog tasks and offline RL.

[1] Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., & Bengio, Y. (2018). Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272.

• It is not a text-based representation, and instead a state is passed as a vector, and RL is trained on the state
• This task cannot be easily used to evaluate RL/LLM tasks

[2] Gontier, N., Rodriguez, P., Laradji, I., Vazquez, D., & Pal, C. (2023). Language Decision Transformers with Exponential Tilt for Interactive Text Environments. arXiv preprint arXiv:2302.05507.

• Results indicate they may not have collected enough data for offline RL algorithms, as offline RL performs poorly [17, 18,19,20]

[3] Hausknecht, Matthew, et al. "Interactive fiction games: A colossal adventure." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 05. 2020. Introduces the Jericho Benchmark

• Our smallest task includes a dataset of 1.25k trajectories. This dataset contains 590 trajectories. A large, diverse dataset is critical for testing offline RL [17, 18]
• Our benchmark is not only text-games and using templates for interaction, we utilize free-form text generation and simulate human-AI interaction

[4] Shridhar, M., Yuan, X., Côté, M. A., Bisk, Y., Trischler, A., & Hausknecht, M. (2020). Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768.

• The work is similar to the TextWorld benchmark, but LMRL-Gym benchmark is a lot more than Text-Nav, and this is our simplest task mainly meant to test implementation and correctness (e.g. “unit test”)
• LMRL-Gym has other text-games and dialogue tasks that are more complex and test a variety of RL Capabilities such as credit assignment, trajectory stitching, partial observability, amongst others.

[5] Wang, R., Jansen, P., Côté, M. A., & Ammanabrolu, P. (2022). Scienceworld: Is your agent smarter than a 5th grader?. arXiv preprint arXiv:2203.07540.

• They benchmark both online and offline RL algorithms, but focused on completing tasks related to scientific reasoning
• No focus on interactive communication with humans/more stochastic environments, or partial observability as LMRL-Gym

[6] Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022). Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35, 20744-20757.

• LMRL Gym has longer interactions and simulate dialog, whereas this is focuses on searching through the web

[7] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.

• Uses AlfWorld and Webshop which are limited, refer to [4,6]

On Offline RL: We would like to note that most of the benchmarks for RL finetuning of LLMs are focused on online RL. Our benchmark focuses on providing an optimal testbed for both offline RL and online RL, by providing large datasets for training offline RL algorithms for LLMs, simulators for online RL training and offline evaluation, and several offline RL implementations including MC Returns, Filtered BC, and ILQL. We created the Car Dealer task to address the issues in [20] including dataset diversity. [16-19] list a series of related works in offline RL for LLMs, primarily focusing on either one task or one algorithm. Our work expands upon these works and provides a suite of both text game and dialog tasks.

[8] Kumar, Aviral, et al. "When should we prefer offline reinforcement learning over behavioral cloning?." arXiv preprint arXiv:2204.05618 (2022).

[9] Prudencio, Rafael Figueiredo, Marcos ROA Maximo, and Esther Luna Colombini. "A survey on offline reinforcement learning: Taxonomy, review, and open problems." IEEE Transactions on Neural Networks and Learning Systems (2023).

[10] Snell, C., Kostrikov, I., Su, Y., Yang, M., & Levine, S. (2022). Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871.

[11] Verma, S., Fu, J., Yang, M., & Levine, S. (2022). Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning. arXiv preprint arXiv:2204.08426.

We thank the reviewer for their feedback. We've addressed the main questions raised in your review by: (1) providing a justification for our choice of models (2) clarifying why we chose to have both interactive dialog tasks as well as text game tasks (3) answering your question regarding CoT for GPT-4.

1. “GPT2 model is now quite old and baselines using similar sized newer models should have been reported...”

While it's true that GPT-2 is relatively older compared to more recent language models, the choice to use it in LMRL Gym was driven by its well-understood capabilities and broad usage in research for establishing baselines [3] compared to newer, similar sized models. We would like to highlight several recent works [1, 2, 4, 5] that have also used GPT2 as a baseline. Due to space limitations, we could not cite more. Our paper focuses on providing a framework that can be easily adapted to various models and the development of further algorithms for RL fine tuning for LLMs, and we hope to see further works that iterate upon other models.

[1] Hicke, Y., Masand, A., Guo, W., & Gangavarapu, T. (2024). Assessing the efficacy of large language models in generating accurate teacher responses. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). arXiv preprint arXiv:2401.12345. [2] Hong, J., Dragan, A. D., & Levine, S. (2024). Q-SFT: Q-learning for language models via supervised fine-tuning. Proceedings of the 42nd International Conference on Machine Learning (ICML 2024). arXiv preprint arXiv:2403.01512. [3] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9. 4o mini [4] Zhou, R., Du, S. S., & Li, B. (2024). Reflect-RL: Two-player online RL fine-tuning for LMs. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). arXiv preprint arXiv:2401.12345. [5] Zhou, Y., Zanette, A., Pan, J., Levine, S., & Kumar, A. (2024). ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL. Proceedings of the 41st International Conference on Machine Learning (ICML 2024), 235, 62178–62209. arXiv preprint arXiv:2402.19446.

2. “The interactive dialogue tasks lack complexity - especially the 20Q and guess (only yes or no reply). More tasks on the lines of car dealer would strengthen the benchmark.”

We would like to clarify that each of the tasks in the benchmarks serve a different purpose. Our objective in creating this benchmark is to present tasks that apply RL algorithms for multi-turn tasks in the domain of goal-directed dialogue, where agents must learn from interaction with a conversation partner. However, to enable such a large undertaking, we require tasks that can first test capabilities of RL algorithms that are essential for multi-turn dialogue, including trajectory stitching, credit assignment, and dealing with complex language. Hence, we have designed five tasks as RL Capability Tests, which are text games designed to isolate specific capabilities of RL training as shown in Figure 4. As seen, these text-games do not test all of the capabilities of RL together, which is only possible through the dialogue-based tasks. Our benchmark includes tasks that involve free-form text generation and a longer turn length. We challenge the agents in our tasks to not only follow instructions and understand the world, but plan over long trajectories, generate complex text, trajectory stitch, and resolve partial observability. For example, for the Maze and Text-Nav we test both partially observed and fully observed versions to highlight the impact of partial observability. In addition, the Text-Nav task is very similar to the Maze task, but places more emphasis on realistic text.

Lastly, the dialogue tasks have been designed with increasing levels of difficulty, with twenty questions testing the ability of RL algorithms to perform information gathering, guess my city testing the ability to ask questions beyond just yes/no and with free form feedback, and the Car Dealer task to test more strategic decision making and persuasive capabilities of RL algorithms for LLMs. As shown, some tasks aim to test specific RL properties without the complexities of realistic language, while others focus on complex language. We wanted our tasks to cover a range of RL capabilities to isolate issues with algorithms, as a group of complicated and difficult tasks may not provide such understanding and insight.

3. “Did you evaluate any chain-of-thought settings with frontier models?”

We appreciate the suggestion to evaluate GPT-4 and other models using CoT reasoning. While CoT has demonstrated strong performance in reasoning-based tasks, our focus was on evaluating baseline RL capabilities without extensive prompt engineering. and we wanted a fair comparison across all settings, including human evaluation and testing on our algorithms.