The Generalization Gap in Offline Reinforcement Learning

We are grateful for your valuable feedback and hope our answers below will address your remaining concerns. We are pleased to see that you found our paper to be “well-written, easy-to-follow” and the topic important.

I have concerns that some of the offline RL baselines might not be tuned appropriately.

We have included thorough details on all the parameters that we had swept for tuning all our baselines in Appendix G.2

To summarize, for Procgen:

    “For BC, we performed a sweep over the batch size ∈ {64,128,256,512} transitions and learning rate ∈ {5e − 3, 1e − 4, 5e − 4, 6e − 5}. For BCQ, CQL, and IQL, which use a DQN-style training setup (i.e., they have a base model and a frozen target model) (Mnih et al., 2013), in addition to the hyperparameters mentioned for BC, we swept over whether to use polyak moving average or directly copy weights, in the latter case, the target model update frequency ∈ {1, 100, 1000} and in the former case, the polyak moving average constant τ ∈ {0.005, 0.5, 0.99}. For BCQ, we also swept over the threshold value for action selection ∈ {0.3, 0.5, 0.7}. For CQL, we swept over the CQL loss coefficient, which we refer to as cql alpha in our codebase, ∈ {0.5, 1.0, 4.0, 8.0}. Finally, for IQL, we sweep over the temperature ∈ {3.0, 7.0, 10.0} and the expectile weight ∈ {0.7, 0.8, 0.9}.		
	
    For sequence modelling algorithms, DT and BCT, we sweep over the learning rate and batch size mentioned above, as well as the context length size ∈ {5, 10, 30, 50}. For DT, we also sweep over the return-to-go (rtg) multiplier ∈ {1, 5}. We follow similar approach in (Chen et al., 2021) to set the maximum return-to-go at inference time by finding the maximum return in the training dataset for a particular game and then multiplying by either 1 or 5 depending on the rtg multiplier value. We also use the default value of 0.1 for dropout in DT and BCT from (Chen et al., 2021).”

For CQL, we swept over 4 (batch size) * 5 (learning rate) * 3 (update freq.) * 3 (tau) * 4 (cql alpha)=720 different combinations, with each combination trained and evaluated thrice for different random seed, amounting to a total of 2160 training runs for tuning CQL. Similar is the case for other baselines.

We have also run a more comprehensive sweep for 10M expert dataset and have updated Table 2 in the Appendix accordingly. In WebShop too we ran a similar comprehensive sweep which is mentioned in Appendix G.2.2.

In conclusion, we believe we’ve done extensive HP tuning with similar budgets for all our baselines.

D4RL (https://arxiv.org/abs/2004.07219), so authors should explain why they did not conduct experiments in the setting of continuous control. For example, the different test environments can be attained by modifying the environment parameters such as gravity.

Thank you for such an insightful question. First of all, we would like to emphasize that D4RL is not a standard benchmark for testing generalization. Procgen was chosen because it is one of the most commonly used benchmarks for this purpose in the online RL setting, and therefore we wanted to advocate its use for offline setting as well. Similarly, WebShop is an equally challenging generalization benchmark in a different domain which is relevant for real-world applications (particularly of language models interacting with the web, which is a timely subject of wide interest to the community).

Moreover, the offline RL methods that we evaluated are advertised for both, continuous as well as discrete domains. BCQ, CQL and DT especially have achieved state-of-the-art results on Atari at the time of their papers’ acceptance.

Our research, on the other hand, emphasizes the importance of testing offline RL methods across a diverse range of tasks. While these methods are often presented as universally applicable, identifying scenarios where they falter is crucial. This approach not only highlights the limitations of current methods but also encourages the development of more robust and versatile solutions.

We would like to thank the reviewer for providing us with valuable feedback. We are glad that the reviewer found our paper’s results to be “important and interesting” and our experiments to be “solid and abundant”.

The algorithms included are a bit limited. Methods like model-based learning [1-2], curriculum imitation [3], and other methods are not involved. More useful conclusions can be made when the benchmarking algorithms are expanded.

Our paper aims to evaluate the most commonly used offline RL algorithms on a new benchmark that probes their generalization abilities (which is an understudied problem in the literature). We believe we have accomplished this by benchmarking BCQ, CQL, IQL, DT, BCT, BC etc., which have a combined 3000+ citations and are competitive on a wide range of commonly used offline RL benchmarks [1]. In contrast, the methods you referenced are rather specialized and don’t seem to be standard baselines in common offline RL papers. There are a lot of offline RL algorithms and we cannot possibly implement, tune, and evaluate all of them on a new benchmark (especially if we want to do justice to all these methods via extensive HP tuning using similarly large budgets to ensure fair comparisons—which we do for our current baselines in Appendix G). Since we had to draw the line somewhere, we decided to include only the most versatile, effective, and widely used offline learning methods according to the literature.

Our goal was to create a benchmark that is accessible and easy to use by the community with simple and unified implementations in order to speed up progress. Therefore, we have open-sourced our code and datasets, and we welcome contributions from everyone to integrate and tune their baselines on our benchmark.

[1]: Tarasov, D., Nikulin, A., Akimov, D., Kurenkov, V., & Kolesnikov, S. (2022). CORL: Research-oriented deep offline reinforcement learning library. arXiv preprint arXiv:2210.07105.

Most of the results are concluded from a simulated benchmark ProcGen, only a small part of experiments are conducted on the real-world dataset "WebShop". The author may consider expanding their tested benchmark to more real-world problems as introduced in [4].

While we covered a significant portion of our experiments on Procgen in the main paper, we covered similar experiments conducted with the WebShop dataset in Appendix N of our submission. Here we present a comprehensive assessment on various sizes of the WebShop dataset (i.e. 100, 500, 1000, 1500, 5000, 10000 episodes), as well as study the impact of suboptimal (collected from IL policy) versus expert data (collected from humans) in WebShop. These experiments are also crucial in demonstrating the robustness and adaptability of our approach in a real-world context.

Furthermore, we want to emphasize that our research is highly relevant and timely, particularly with the increasing importance of LLM-based agents interacting with the internet. The Procgen benchmark, which learns directly from high-level images in an applied domain (games), and the WebShop environment, which is a realistic simulator of e-commerce websites, both are highly relevant and practical. Specifically, our experiments with WebShop showcase generalization to real-world scenarios, including interactions with actual websites like Amazon.

While NeoRL [4] does not assess generalization to new levels/instructions, the WebShop simulator on the other hand allows a realistic emulation of web interactions, thus enabling the agents to generalize to real-world scenarios like the Amazon website. We argue that this is a substantial step towards real-world applicability.

We do not currently plan to expand the benchmark to other domains for the purpose of this submission. However, we are open to exploring this in the future and will aim to make it easy for others to contribute additional datasets since we will be open-sourcing our codebase. Both our repositories can be adapted to other similar environments i.e. image-based/state-based environments in the Procgen repository, and text-based environments in the WebShop repository.

Will the author open-source their benchmark and evaluation codes?

We would like to clarify that both, the code and the datasets, are already uploaded as zip files as part of the supplementary material here and we will definitely open-source the code upon acceptance.

We answer the remaining questions in the next comment.

Thank you for providing valuable and constructive feedback which greatly improves our work. We were glad you found our “experiments thorough” and “writing clear and easy-to-follow”. Below we address the points raised in the same order:

novelty is restricted. Given that many existing offline RL algorithms do not inherently prioritize generalization ability in their design, so the current experimental results are mostly within expected outcomes.

We respectfully disagree with this assessment of our contributions. As far as we know, our paper is the first to introduce a benchmark specifically focused on evaluating the generalization of offline RL algorithms. The absence of such a benchmark limits our understanding of these algorithms' real-world applicability, and our work strives to bridge this gap. In addition, we believe our results are novel in that this is the first paper to demonstrate that online RL > behavioral cloning > sequence modeling > offline RL approaches when it comes to generalization to new tasks. Even if our results weren’t surprising, we believe it is important to show this empirically via a thorough comparison study that others can later reproduce and build on since, as we show in the paper, details such as the data diversity or size can lead to different results. Moreover, our findings emphasize the need for more robust algorithms suitable for practical applications and establish a standardized benchmark for the equitable assessment of these methods across diverse scales and qualities. Hence, our contributions consist of: the creation of multiple datasets from two different domains, along with the extensive evaluation of multiple algorithms, and a clear evaluation protocol.

many duplicate references

Thank you for pointing this out! We have now rectified the references.

paper primarily serves as a summary of empirical observations, and no specific solutions are proposed to address the generalization issue

While we agree that specific techniques (either new or existing ones such as regularization) could increase performance of these offline RL, the purpose of our work isn’t to improve existing offline RL algorithms but rather to thoroughly evaluate existing ones as they are typically used in the literature (and in practice).

Consequently, it would be prudent to avoid overly definitive statements in this paper, given these successful examples…… Moreover, I would like to highlight a gap in the current work—there is a lack of more in-depth analysis to elucidate the key question: "why certain offline RL agents [1][2] exhibit superior generalization skills while others do not?"

We acknowledge the points you raised and we will revise our manuscript to address them. We agree that our conclusions should be more cautiously phrased, emphasizing that they are based on evaluations within specific domains. Indeed, there's a need for further research to study the reason behind the good/bad generalizability of these methods.

Furthermore, we would like to re-emphesize the following:

1. Paper [1] doesn't have any experiments on generalization to new environments so it is not a relevant comparison to our work.
2. This is also mentioned in the “Extended Related Works” section in Appendix D of our submission, i.e. methods like [2] study how to generalize skills across different Atari games, using a dataset of about 40 games (and [1] only studies online generalization to the same game). In contrast, our research is about how well different levels or instructions within the same game can be handled, and we do this within the Procgen as well as the WebShop framework. Moreover, our dataset and benchmark is designed to be memory-efficient, making it easier for the academic community to use without needing many GPUs. This is unlike the approaches in [2], which require a large number of TPUs and are practically infeasible to reproduce in most of the academic settings.

We hope our responses above have adequately addressed your queries and that you will consider increasing your score. Please let us know if there is anything else preventing you from recommending acceptance of our paper.

We are grateful for your consideration of our rebuttal and thank you for your decision to raise the score. If there are any further issues or questions, we would be happy to provide additional information.

We are grateful for your valuable feedback and hope our answers below will address your remaining concerns. We are pleased to see that you found our “results to be impactful” and “our benchmark an important contribution”.

some thoughts on why offline RL does not generalize well?

We believe one reason for which offline RL methods lag behind BC is that all the offline RL approaches employ a risk-averse strategy whereby if they are encouraged to not take actions that they haven’t seen during training. However, given that we are testing generalization to new environments, all the states the agent encounters at test time are new so offline RL methods will be averse to taking any of those actions, likely defaulting to a rather suboptimal policy. In contrast, BC doesn’t have any such constraints so it simply uses its learned representations to decide what action is best to take by “looking at the observation closest to the test state and picking the best action”. If BC is able to learn good enough state representations from its training data, it should be able to also learn a good enough policy that generalizes to new environments reasonably well.

On the question of why all offline learning approaches lag behind online RL ones, we believe the reason lies in the fact that online RL collects and learns from its own data, thus seeing a much more diverse set of states than BC or offline RL approaches where the set of states is fixed by the dataset. Given that at test time, all the states are new for the agent, having seen a wider range of states can help the agent learn better representations and have a better idea of what’s the best action in new situations. As our experiments in section 4.4 show, training on more diverse data can greatly improve the generalization of offline learning methods. Yet, as our results in section 4.5 show, simply training on data from multiple PPO checkpoints is not enough since these are sparsely sampled throughout training so they cannot span the entire space (it’s also difficult to know exactly how training dynamics play into the diversity of the data collected which could be a good direction for future work to further investigate).

We’ve updated the paper with a more in-depth discussion of this topic – see Appendix E “Discussion”, however, we believe more work is needed to test these hypotheses in order to reach a better understanding of these results.

It's important to note that our findings, which may seem surprising, underscore the significance of our benchmark. The divergence in results emphasizes the need for further research in this domain. Our results highlight the complexity of offline RL generalization and cautions against assuming that insights from Atari datasets (which is one of the most commonly used offline RL datasets) will directly transfer. Moreover, we would like to emphasize that our results also align with other literature like [1], [2],[3] that show similar results i.e. BC > offline RL wrt generalization.

[1]: Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., ... & de Freitas, N. (2023). Reinforced Self-Training (ReST) for Language Modeling. arXiv preprint arXiv:2308.08998. [2]: Piterbarg, U., Pinto, L., & Fergus, R. (2023). NetHack is Hard to Hack. arXiv preprint arXiv:2305.19240. [3]: Hambro, E., Raileanu, R., Rothermel, D., Mella, V., Rocktäschel, T., Küttler, H., & Murray, N. (2022). Dungeons and Data: A Large-Scale NetHack Dataset. Advances in Neural Information Processing Systems, 35, 24864-24878.

There are minor problems: The color of the lines in Figure 2(a) is wrong. The results on Leaper are missing in Figure 11 and Figure 12.

We thank the reviewer for pointing out these errors. We have now fixed Figure 2(a). In Figures 11 and 12, our underlying dataset had 0 return, since the PPO policy itself did not perform well on that level. Hence, all the offline RL baselines too had 0 return which is the reason why the plots look empty as if the results are missing. To clarify this, we have added a note in the captions of Figures 11 and 12

We hope our responses above have successfully addressed your concerns. Please let us know if you have any outstanding questions.