Replay across Experiments: A Natural Extension of Off-Policy RL

Thank you for your helpful and positive feedback. We have updated the paper with suggested changes highlighted in red and are confident that the presentation is stronger thanks to them. We address specific questions below and have added a general comment for points shared across reviews. Please do not hesitate to ask for further clarifications.

• What motivated the reuse of previous experimental data

We demonstrate how reusing previous data improves performance across a range of diverse domains and algorithms. Related work has demonstrated a similar effect to some degree albeit with additional algorithmic complexities that render them difficult to implement. Our focus in this work is on analyzing an underlying simple core idea more directly. With the increasing proliferation of offline datasets for robotics domains, RaE is an effective tool that can be incorporated into existing algorithms and fits naturally into existing RL experimentation workflows.

• “..would it be more valuable to add older trajectories to the data buffer or relatively newer ones?”

The results of Table 1 indicate that the best way to use RaE with limited data is to combine trajectories sampled randomly throughout training (Mixed return) with a higher ratio of online to offline data (70-80 % works quite well). We have emphasized this point in the text.

• Benefits of RaE on Atari games

The `Random resets’ baseline we consider in our work is closely related to the work of Nikishin et al. [1]. They demonstrate an improvement of performance on Atari domains albeit with different underlying algorithms. We demonstrate RaE outperforms random resets across a range of diverse environments and algorithms with continuous action spaces; a finding which is likely to carry over to the Atari domain.

• Reasons for results in Table 1.

Table 1 presents an analysis of choosing different mixing ratios when running RaE with smaller datasets (10,000 and 100,000 episodes). Importantly, for all the main results shown in Figure 3 and 4 a fixed 50-50 ratio works well across domains. However, when the amount of data is restricted, using more online data is beneficial and prevents overfitting to a narrow distribution of offline trajectories. We have clarified this point in the description of Table 1.

• “..exaggerate the increase in block size to bring a reader’s attention to the data accumulation mechanism.”

Thank you for the suggestion. We have updated the paper with a new version of the figure that highlights the increasing block size of the data on each iteration. Please let us know if further improvements can be made.

[1] Nikishin, Evgenii, et al. "The primacy bias in deep reinforcement learning." International conference on machine learning. PMLR, 2022.

Thank you for your helpful and constructive feedback. We have updated the paper with suggested changes highlighted in red and are confident that the presentation is stronger thanks to them. We address specific questions below and have added a general comment for points shared across reviews. Please do not hesitate to ask for further clarifications.

• Total online steps

We have updated Figure 4 to include the data from the offline dataset. The relative performance of RaE against the baselines remains unchanged and importantly, RaE continues to show improved asymptotic performance on challenging domains. We hope this addresses your original comment and will help improve your original score. If there are further questions, please let us know.

• Difference between Random resets and RaE

When restarting an experiment with RaE, in addition to resetting the weights of the network, the data distribution throughout learning utilizes a fixed ratio of mixing between offline and online data. In contrast, `Random resets’ only reload network weights and then use data from the replay buffer for learning in which older data is overwritten with time. In addition, RaE is more practical to implement since a period to reset does not have to be chosen and fixed a-priori; experiments can be restarted and data mixed naturally during the course of experimentation with repeated iteration.

• “I'm curious if re-initializing vs not re-initializing at the start of each training run makes a difference in performance?

This is a very good point. The choice of re-initializing weights is orthogonal to RaE and can be incorporated on top of RaE. Figure 7 of Appendix C.2 highlights this by comparing the performance of RaE with and without reloading weights on the `Locomotion’ domains. We find reinitializing weights can improve learning speed without hurting asymptotic performance.

• “Have the authors actually tried experimenting with changing dynamics across training runs.”

Our comment regarding changing dynamics alludes to a theoretical limitation where an extreme change in the dynamics could render the transitions in the offline data invalid. However, as you note, this may not be much of an issue in practice. To test this, we conducted an experiment presented in Appendix C.4 where we transfer data from the `Locomotion Soccer (State)’ task to a new setting where a mass (of 300g) is randomly attached to the legs of the walker to perturb its motion. Surprisingly we find RaE works well in practice in this setting and can learn a partial solution to the task while learning online from scratch fails to learn any solution.

Thank you for your helpful and constructive feedback. We have updated the paper with suggested changes highlighted in red and are confident that the presentation is stronger thanks to them. We address specific questions below and have added a general comment for points shared across reviews. Please do not hesitate to ask for further clarifications.

• “The main insight of the work, while interesting, is a small contribution.”

We respectfully disagree on the scale of contribution of our work. As you note, we present a highly-effective, simple method that can easily be introduced to RL methods and provide a comprehensive evaluation showing good performance compared to several strong baselines across a breadth of domains. As we discuss in the main text, other works that use a similar idea do so in an incidental manner, and commonly focus on additional algorithmic changes that add considerable complexity. By focusing on the core approach and combining its simplicity with a strong empirical contribution we believe our work can translate to practical real world impact across a range of RL domains.

• “Storing all previous data is memory intensive”

We agree that memory is not free, but as Reviewer wVcc points out, and as witnessed by the increasing availability of large datasets in RL and robotics [1], memory is relatively cheap. Large, shared datasets for robotics are becoming more widely available and are likely to be integrated into existing RL workflows. As we show, RaE improves asymptotic performance on many domains: in many cases it may be cheaper to scale up data storage rather than compute to achieve a similar final performance. And as illustrated by Table 1, we can achieve strong results by storing relatively small subsets of data randomly throughout training.

• Confidence intervals instead of standard deviations.

We agree that the visual presentation of data is important. We have replaced the error bars for the plots in Figure 3 (and Figure 6 in the Appendix) to instead show a 95% confidence interval over the mean after smoothing over 1000 episodes and have updated the description in Section 3.3 to reflect this.

• 90% online data works best in Table 1

The results in Table 1 discuss how best to adapt RaE to regimes where data is sparse. As you point out, using more online data is a reasonable strategy in this setting. However note that mixing some offline data is preferable to learning purely offline and all the results in Figure 3 and 4 show an improvement in asymptotic performance with a simple 50/50 ratio of mixing data. Additionally, we have updated the paper to include an additional result in the Appendix C.3 that illustrates the effect of mixing different data ratios when using the full dataset (of 4e5 episodes). This result represents many project settings and shows that when more data is available, a mixing ratio of 50 to 70 percent of online data works better than a mix with too much online data.

• Accumulated Reward

Yes, by accumulated reward we mean the undiscounted episode return achieved by the agent. We have updated the captions of Figures 3 and 4 to reflect this.

• Table 1 caption ‘from scratch to convergence’

The results in Table 1 present the performance of RaE with small subsets of data as a percentage of learning from scratch. By ‘from scratch to convergence’, we want to highlight that the agent performance being compared against is not at the 10,000 or 100,000 episode mark but with an online agent trained until it converges.

• Minor edits

Thank you for pointing these out to us. We have made corrections to the text.

[1] Open X-Embodiment: Robotic learning datasets and RT-X models. https://robotics-transformer-x.github.io.

Thank you for your helpful and constructive feedback. We have updated the paper with suggested changes highlighted in red and are confident that the presentation is stronger thanks to them. We address specific questions below and have added a general comment for points shared across reviews. Please do not hesitate to ask for further clarifications.

• “What about training on the data throughout the optimization process?”

RaE mixes online data and offline data in a 50/50 ratio for each minibatch sampled throughout training (and not just at the beginning). We have updated the text (page 3) to clarify this.

• RaE in the context of the scientific method for reproducible and comparable results.

This is a very good point; thank you for highlighting it. Reproducibility in RL is already a challenging open problem [1]. If RaE is more widely adopted, it may add another dimension to this issue: subtle changes in the ordering of data when mixing may lead to changes in final performance. One solution to this could be to standardize and open source offline data sources for benchmark domains (e.g. RL Unplugged) and define deterministic orderings when reading from the data using, for example, fixed random seeds. We have included a brief discussion on this point in the Discussion section (Section 5).

• “Why does offline training work so poorly?”

In Figure 4, fine-tuning and AWAC learn on the Humanoid Run domain but perform poorly on the Manipulator domains of RL Unplugged. Humanoid run uses a dense reward signal that incentivizes the agent for moving forward. In contrast, the Manipulator domains provide sparse rewards that are triggered in stages when a robotic arm moves to a peg (or ball) and inserts it into a hole. Learning offline from limited data in sparse reward settings is particularly challenging which is why we see a strong benefit of using RaE to mix in online data to improve asymptotic performance. Additionally offline learning may also overfit to the limited set of trajectories in the dataset, making further improvement through fine-tuning difficult.We have added a sentence to Section 3.3 to highlight this.

• “Additional baseline: BC and distillation to previous agent.”

When fine-tuning, we consider two baseline agents trained with the offline data: one that uses CRR and another that uses BC. We then select the best performing variant among these to finetune. We believe this addresses your point but are happy to clarify and run another baseline (time permitting) if required.

• “.. more easily identify which hyperparameter settings are promising by giving the agent additional data.”

Yes, the faster iteration time when using RaE could be useful to search for better hyperparameters to use in new settings from scratch. However, it is important to note that hyperparameters that are tuned in a setting that reuses data may not directly transfer over to settings where data is not reused. For example, learning rates and optimizer parameters may not be directly transferable.

[1] Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018). Deep Reinforcement Learning That Matters. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai.v32i1.11694

I noticed that Reviewer wVcc's comments are slightly misrepresented here. The reviewer's comment was that it is great that this paper studied an understudied topic, and not that this is a great paper that discusses an understudied topic. It is just a subtle difference in wording, but I believe it misrepresents the original comment. I assume this was unintentional, but I just wanted to point it out to give the authors an opportunity to correct it.