We would like to thank the reviewer for their kind comments! We’re quite pleased to see that you found our work to be well-presented and that you appreciate the insights we draw from our empirical analysis.

Below, we address your questions regarding how and why we generate “valid” augmented data. We would also like to point the reviewer to Appendix C in the supplemental material for a more in-depth description of all DAFs in our analysis.

Ensuring we generate valid augmented data:

Ensuring we generate valid augmented data requires domain knowledge; we know that the dynamics of the robot and objects in Panda tasks are independent of the goal state, and we know that translating and rotating the agent in Goal2D will generate data that respects the task’s dynamics. We want to note that while domain knowledge may seem like a limitation, we observe in the literature and real world RL applications that valid data augmentation functions are incredibly common and often require very little prior knowledge to specify. For example:

1. Transition dynamics are often independent of the agent’s goal state [1].
2. Objects often have independent dynamics if they are physically separated [2,3], which implies translational invariance conditioned on physical separation.
3. Several works focus on rotational symmetry of 3D scenes in robotics tasks [4,5], and many real-world robots are symmetric in design and thus have symmetries in their transition dynamics [6,7].

We chose to focus on dynamics-invariant augmentations because they have already appeared so widely in the literature. Furthermore, as RL becomes an increasingly widely used tool, we anticipate that domain experts will be able to identify new domain-specific augmentations and use them to further lower the data requirements of RL. Hence, the importance of identifying when and why different general properties of data augmentation will benefit RL.

What if we generate non-valid augmented data?

Non-valid data, i.e. data that does disagrees with the task’s dynamics and/or reward function, can bias learning and reduce data efficiency (similar to how model uncertainty biases learning in model-based algorithms [8, 9]). Since our analysis aims to understand which aspects of DA contribute to observed improvements in data efficiency, our focus on dynamics-invariant DAFs removes a potential confounding factor in our analysis. An interesting future research question to consider might be “How much non-valid data can an agent tolerate without harming data efficiency?”

Several prior works have used non-valid augmentations – especially those focusing on visual augmentations [e.g. 10, 11] – and we do think it is worth studying this class of DAFs. However, visual augmentations primarily aid representation learning, so such a study will likely need to focus on different aspects of DA than the ones we considered in our work and is thus beyond the scope of our analysis.

Please let us know if our response clarifies your comments! If you have follow-up questions or comments, we are more than happy to discuss them with you.

References

[1] Adrychowicz et. al. "Hindsight Experience Replay." NeurIPS 2017.

[2] Pitis et. al. Counterfactual Data Augmentation using Locally Factored Dynamics." NeurIPS 2020.

[3] Pitis et. al. "MoCoDA: Model-based Counterfactual Data Augmentation." NeuIPS 2022.

[4] Wang et. al. "On-Robot Learning with Equivariant Models." CoRL 2022.

[5] Wang et. al. "The Surprising effectiveness of Equivariant Models in Domains with Latent Symmetry." ICLR 2023.

[6] Pavlov et. al. "Run, Skeleton, Run: Skeletal Model in a Physics-Based Simulation." AAAI 2018.

[7] Abdolhosseini et. al. "On Learning Symmetric Locomotion." ACM SIGGRAPH 2019.

[8] Moerland, Thomas M., et al. "Model-based reinforcement learning: A survey." Foundations and Trends in Machine Learning 16.1, 2023.

[9] Polydoros and Nalpantidis. "Survey of model-based reinforcement learning: Applications on robotics." Journal of Intelligent & Robotic Systems 86.2, 2017.

[10] Raileanu et al. “Automatic data augmentation for generalization in deep reinforcement learning.” arXiv:2006.12862, 2020.

[11] Laskin et. al. "Reinforcement Learning with Augmented Data." NeurIPS 2020.

First, we would like to thank the reviewer for their positive review. We’re pleased to see that you found our analysis to be well-written and clever, and our empirical findings to be useful to the research community. Below, we address your comments.

Clarification on limitations:

Thank you for pointing out the additional limitation with respect to our RL algorithm choices – we’ll include it in our revisions.

Different algorithms may be more or less capable of learning from augmented experience. We do think that it would be interesting to conduct another empirical study investigating how different algorithmic advances (e.g. multiple target networks [1], entropy regularization [2], n-step returns [3], etc.) may affect learning from augmented data. Several recent works have shown that n-step returns are critical to data efficient RL when learning with only observed data [4,5], and we hypothesize this may also be the case when learning with augmented data.

How did we choose our algorithms?

We initially wanted to use TD3 throughout the paper; TD3 often performs much better than DDPG, and is much less expensive to run than SAC. However, we found that TD3 performed noticeably worse than DDPG on all Panda tasks. In fact, Gallouedec et. al [6] (the creators of Panda robot tasks) made this same observation. Thus, we decided to use DDPG for Panda tasks and TD3 for Goal2D as well as all MuJoCo tasks in Appendix F.

Why does data augmentation improve data efficiency more in Figure 1 than in Figure 3?

In the Goal2D task, we do see that additional augmented data improves data efficiency more than additional policy-generated data. In Panda tasks, augmented data is either just as good or slightly worse. We believe this difference arises because Goal2D is much simpler than the Panda tasks.

The absolute improvement to data efficiency provided by DA is not a critical point in our analysis; we sought to understand how much different aspects of DA contribute to the observed relative improvements in data efficiency.

Please let us know if we’ve addressed all of your comments. We are more than happy to discuss any follow-up questions or comments you might have!

References

[1] Fujimoto et. al. "Addressing Function Approximation Error in Actor-Critic Methods." ICML 2018.

[2] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." ICML, 2018.

[3] Sutton and Barto. "Reinforcement Learning: An Introduction." 2018.

[4] Fedus et. al. “Revisiting fundamentals of experience replay.” ICML 2020.

[5] Obando-Ceron et. al. "Small batch deep reinforcement learning." To appear in NeurIPS 2023.

[6] Gallouedec et. al. “panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning” 4th Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurIPS 2021

Do the findings hold when DA is used for policy distillation or representation learning?

Our work focuses on understanding the benefit of integrating dynamics-invariant augmented data into model-free RL updates. Analogous studies focusing on policy distillation and representation learning are interesting directions for future work that we intend to investigate, but are beyond the scope of this analysis. We note that such studies will likely need to focus on different aspects of DA than the ones we considered in our work.

In particular, prior DA works focusing on policy distillation [4] and representation learning [3] are designed to work with DAFs that generate augmented data with the same semantic meaning as the original data, such as visual augmentations. This point has three implications:

1. Because these augmentations produce data which could never be observed through environment interaction (e.g. an agent would never receive a cropped, recolored, and rotated image from the environment), they primarily aid representation learning rather than exploration.
2. It may be imprecise to say that these augmentations increase coverage, because the observed and augmented data have the same semantic meaning.
3. Because the original and augmented observation have the same semantic meaning, the augmented reward will often be the same as the original reward. Thus, the concept of “reward density” does not apply to these augmentations.

Having said this, the role of the augmented replay ratio can and should be studied for visual augmentation. Researchers have become increasingly interested in how the replay ratio affects learning [9-12], and lowering the replay ratio via augmentation may yield substantial improvements in data efficiency on visual tasks.

Hyperparameter tuning:

Appendix G in the supplemental material lists hyperparameters and training details for all experiments. For Panda tasks, we use the same hyperparameters used by Plappert et. al [13], the creators of the Fetch robot tasks analogous to the Panda robot tasks we consider in our work. Gallouedec et. al [14] (creators of Panda robot tasks) also use these hyperparameters. Plappert et. al found these hyperparameters by performing a sweep over many different hyperparameter combinations described in Appendix B of [13].

We modified two hyperparameters in our replay ratio experiments (also noted in Appendix G). These experiments use a larger split of augmented data in each update, and we observed that these modifications improved performance in all tasks for both augmented and non-augmented agents.

Again, please let us know if our response clarifies your comments. We would be happy to discuss any follow-up questions you may have!

References:

[3, from previous comment] Raileanu et al. “Automatic data augmentation for generalization in deep reinforcement learning.” arXiv:2006.12862, 2020.

[4, from previous comment] Ko, Byungchan, and Jungseul Ok. "Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning." NeurIPS, 2022.

[9] Fedus et. al. “Revisiting fundamentals of experience replay.” ICML 2020.

[10] Chen et. al. “Randomized Ensembled Double Q-Learning.” Arxiv, 2021.

[11] Nikishin et. al. “The primacy bias in deep reinforcement learning.” ICML 2022.

[12] D'Oro et. al. “Sample-efficient reinforcement learning by breaking the replay ratio barrier.” ICLR 2023.

[13] Plappert et. al. “Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research.” Arxiv, 2018.

[14] Gallouedec et. al. “panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning” 4th Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurIPS 2021

We would first like to thank you for your review! We’re pleased that you found our work well written and noted the importance of our findings regarding the augmented replay ratio. We believe we can address your comments and questions with a few minor clarifications in our revisions, and we hope to discuss these clarifications with you.

Core messages of our empirical analysis:

In our submission, we provide two practical guidelines:

1. Data augmentation (DA) should focus less on generating reward signal and more on increasing state-action coverage. In our experiments, most – and sometimes all – of the benefits of a DAF can be explained by an increase in state-action coverage alone, and increasing reward density beyond a small threshold typically provides no further benefit. This guideline contrasts with the core motivation of HER and its follow-up works [1, 2] which focus on generating augmented reward signal to improve data efficiency.
2. When it is possible to generate a large amount of augmented data, use this data to decrease the replay ratio (rather than to increase the batch size used for updates.)

We’re pleased that our second guideline was clear, and we hope our response clarifies the first guideline. In our revisions, we will better emphasize these guidelines.

Differences between our work and prior works studying visual augmentations:

This is a great question that we can clarify with a few minor additions to the related work section. There are a few fundamental differences between our work and the related works you’ve provided:

1. Most prior works – including all three you have listed – focused on introducing new types of data augmentation functions (DAFs) or frameworks and demonstrating that they can boost the data efficiency of RL. To the best of our knowledge, our work is the first to systematically investigate which aspects of DA are most responsible for observed improvements in data efficiency.

2. Raileanu et. al [3] and Ko & Ok [4] use augmented data for auxiliary tasks, while we focus on using augmented data for model-free updates. Moreover, these auxiliary tasks aim to make the policy and value function invariant to augmentation, i.e. $\pi( \cdot | s) = \pi( \cdot | \tilde s)$ and $V(s) = V(\tilde s)$, and cannot be used with the dynamics-invariant DAFs we considered. For example, the optimal action generally changes if we translate the agent or goal, so it is undesirable to have $\pi( \cdot | s) = \pi( \cdot | \tilde s)$.

3. As you noted, all three works you listed consider visual augmentations that only transform the agent’s state and generate augmented data with the same semantic meaning as the original data. As such, they study DA for representation learning. In contrast, the dynamics-invariant DAFs we consider generate augmented data with a new semantic meaning and thus study DA to improve exploration.

Novelty of our framework:

The core novelty of our framework is that it permits fine-grained control over many hyperparameters relevant to DA, allowing us to systematically study different aspects of DA.

Crucially, our framework guarantees that the agent uses the same ratio of augmented to observed data (or update ratio) in each update by sampling observed and augmented data from separate replay buffers. Existing DA frameworks [e.g., 5,6] and applications of DA [e.g., 7, 8] sample data from a shared replay buffer containing both augmented and observed data, so the update ratio can change across updates. Moreover, with a shared replay buffer, the update ratio and the augmentation ratio (the number of augmented samples generated per observed sample) are entangled; increasing the augmentation ratio increases the update ratio.

Given that DA is so widely studied and that most prior work focuses on improving over baselines rather than understanding why certain methods perform well, we hope the analysis enabled by our framework is the first of many which move us towards the development of further practical DA guidelines.

We hope our response clarifies our work's novelties with respect to the existing DA literature. If you have further questions, we are more than happy to discuss them!

[1] Adrychowicz et. al. “Hindsight Experience Replay.” NeurIPS 2017

[2] Li et. al. “Generalized hindsight for reinforcement learning.” NeurIPS 2020

[3] Raileanu et al. “Automatic data augmentation for generalization in deep reinforcement learning.” arXiv:2006.12862, 2020.

[4] Ko, Byungchan, and Jungseul Ok. "Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning." NeurIPS, 2022.

[5] Pitis et. al. “Counterfactual Data Augmentation using Locally Factored Dynamics.” NeurIPS 2020.

[6] Adrychowicz et. al. “Hindsight Experience Replay.” NeurIPS 2017.

[7] Fawzi et. al. “Discovering faster matrix multiplication algorithms with reinforcement learning.” Nature 2022.

[8] Abdolhosseini et. al. “On Learning Symmetric Locomotion.” ACM SIGGRAPH 2019.

Thank you for your kind comments! We’re glad you found our work interesting, clear, and easy to follow. Below, we’ve addressed your comments and answered the question you posed. We hope to discuss these clarifications with you.

The purpose of Definition 3 (dynamics-invariant DAF):

We use definition 3 to define the class of data augmentation functions (DAFs) we focus on in this work. At a high-level, a dynamics-invariant DAF only generates augmented data that the agent could in principle observe through task interaction (i.e. data that agrees with the task’s dynamics and has the correct reward). Learning from augmented data that does not match the task’s dynamics can harm learning, similar to how model uncertainty biases learning in model-based algorithms [1, 2] . Since our analysis aims to understand how different aspects of DA affect the data efficiency of RL, our focus on dynamics-invariant DAFs eliminates a potential confounding factor.

Upon consideration, we do not require the second half of this definition (“and if $f$ respects the stochasticity of $p$”) for our analysis and will remove it to prevent confusion. Definition 3 will become: “A DAF is dynamics-invariant if it is closed under valid transitions.” Intuitively, a DAF is dynamics-invariant if it takes valid transitions and produces new valid transitions.

[1] Moerland, Thomas M., et al. "Model-based reinforcement learning: A survey." Foundations and Trends in Machine Learning 16.1, 2023.

[2] Polydoros, Athanasios S., and Lazaros Nalpantidis. "Survey of model-based reinforcement learning: Applications on robotics." Journal of Intelligent & Robotic Systems 86.2, 2017.

Our choice of data augmentation functions (DAFs) in our experiments:

Thank you for this comment! Our decision to focus on a few DAFs comes from two practical considerations that we will clarify in the main paper.

While our experiments focus on 3 dynamics-invariant DAFs – agent translation, agent rotation, and goal relabeling – these DAFs are extremely general and apply to many tasks. For instance, translation and rotation can be applied to most navigation tasks. In the Goal2D environment in Fig. 1, translation and rotation would still be valid DAFs if the agent were a more complex locomotor (e.g. a quadruped robot). Moreover, since task dynamics rarely depend on the goal state, goal relabeling can be applied to most goal-conditioned tasks.

Other popular DAFs such as visual augmentations generate non-valid augmented data and are thus beyond the scope of our analysis. A study of visual augmentations would need to focus on different aspects of DA than the ones we considered in our work. For example, visual augmentations generally do not change a transition’s reward, so we cannot use them to study the effect of increasing reward density.

We can emphasize both of these points in the paragraph just before section 5.1.1. We hope this clarifies our empirical design. Please let us know if you have further questions or comments; we would be more than happy to discuss them!

Regarding theoretical analysis:

We agree that theoretical analysis is important, however, like many prior DA works in RL, our work is empirical in nature. We can make suggestions for future theoretical analysis in the paper. Are there types of analysis that the reviewer thinks are particularly important for this line of work?