Revisiting Data Augmentation in Deep Reinforcement Learning

Thank you for noting the generality of our proposition and providing constructive suggestions. Below, we provide our answers to the weaknesses and questions.

1. W1: Oversimplified Unified Framework

Please check point 3 in our Meta Reply.

2. W2: Excessive Reference to Appendix.

We thank the reviewer to have read our paper so carefully. We admit that it is distracting to frequently check the appendix and we apologize for this. As the reviewer may understand, due to the page limit, all the information cannot be included in the main text. We tried to maintain the coherence of our presentation as much as possible. The references to the appendix mainly provides the theoretical proofs, further empirical evidence, additional experiments, and other minor details. We believe that they can be skipped in a first reading.

3. W3: Unverified Claims.

Note that the expectation of the target is actually very large compared to the variance of the target. At the end of the training (steps 500K), the mean of the target is more than 100 while the variance of the target is just about 1 for both cases of using complex image transformations and not. From Table 5, although we can observe a small increase in the variance of the target due to using complex image transformation, it is still reasonable to state that this variance increase is not the dominant issue for the performance drop.

We thank the reviewer for raising this point. We updated Table 5 to include the mean of the target.

4. W4: Comparison with RAD and SVEA

Before we compare RAD and SVEA, we need to clarify that SVEA actually uses random shift in calculating the targets and only avoids those complex image transformations in calculating the targets. Although it is removed from their public repository in the github commit (https://github.com/nicklashansen/dmcontrol-generalization-benchmark/commit/35a1eed87f72d3c9881c425fd83604c756849e56), we checked with the authors by email to confirm that the results in the paper were generated by applying random shift in calculating the targets.

Now, if we compare the generalization ability evaluated in color-hard background, we should observe that RAD (random shift) $<$ RAD (random conv) $<$ SVEA (random conv). The first inequality is decided by using random conv as the image transformation. The second inequality is caused by avoiding using random conv in calculating the target.

5. W5: Unclear Table Presentation

Thank you for raising this point. We apologize for the confusion and clarified the formatting of Table 4. In this table, each column corresponds to SVEA trained with one specified image transformation (DA) and each row corresponds to the cosine similarity recorded at 100k steps between the features of augmented states. Here, the cosine similarities between latent features of randomly shifted images are used as the baseline for comparisons. By comparing elements in one column, we can observe a smaller cosine similarity between the augmented features when training SVEA with complex image transformation which indicates the invariance with respect to complex image transformation is harder to learn.

6. W6: Novelty and Contribution

Please check point 1 in our Meta Reply.

7. W7: Minor comments Bolding in Tables.

Thank you for your constructive suggestions. We initially used bold to highlight the values that are commented in the text. We understand this formatting may lead to some confusion. We have modified our paper according to the suggestions:

• For a clear presentation, we removed bolding in Tables 1, 4, and 5 and added more explanations in the captions.
• We included more discussions in the Limitation section and explicitly referred to it in the Conclusion section in the main text.
• We added the missing legends for Figure 8.

I acknowledge the author's responses to my questions. Given this information and the reviews provided by my colleagues, I am willing to raise my score from 5 to 6.

Thank you for noting the quality of our work especially in originality and significance. We are happy that you recognize our paper being logically clear. Below, we provide an answer to the weaknesses.

1. W1: However, I believe the experimental section of the paper is somewhat lacking.

To the best of our knowledge, most recently-proposed techniques for improving generalization ability are actually orthogonal to our analysis and could be combined with our proposed algorithm. However, if we have missed any relevant work, we would appreciate if you could provide some specific references and we would try our best to add some additional comparative experimental results in our final version. As for the limitations of our proposed algorithm, please check point 2 in our Meta Reply.

2. W2: However, throughout the entire text, there is no discussion about the shortcomings and limitations of the algorithm, nor is there any mention of how to extend the algorithm or areas that need further exploration in the future.

Please check point 2 in our Meta Reply.

3. W3: However, the paper does not address whether there might be potential overfitting issues under these specific conditions and how to handle them.

We did not observe this overfitting issue when we updated more. To verify this point, we ran some additional experiments to provide further evidence for it. First, we can see that the training scores are similar to the evaluation scores when we update more, as shown in the figure (evaluation_training_score_for_reviewer_9PSY.png) in our new supplementary material. We also ran the experiments of training the agent with more iterations when more updates are applied per iteration and the result is shown in the figure (More_iterations_and_more_updates_for_reviewer_9PSY.jpeg) in our new supplementary material. In this experiment, we trained the agent with more iterations and more updates per iterations while we did not observe an overfitting issue.

Please let us know if you have any other comments or concerns!

5. Q3: How to show that "even using complex image transformations such as random convolution in the target". We did not find an experiment in the paper that verifies this conclusion.

We apologize for the ambiguity here. The experimental results for this point are shown in Appendix F.
We modified our paper to explicitly indicate that all observations about using complex image transformations in calculating the targets are presented in Appendix F. Moreover, we clarified the descriptions of Tables 4 and 5.

6. Q4: In the experimental part, the grid search for $\alpha_{KL}$ and $\alpha_{tp}$ is too sparse, and the final choice of 0.1 leads to curiosity about the results within [0,0.1]. A similar situation occurs with the SVEA comparison experiment of 0.5 selected from {0.1, 0.5}.

Considering that our main goal was to provide some justifications and recommendations on techniques used in data augmentation methods in DRL, we did not focus on getting the best possible score on this DMControl benchmark. However, we can indeed perform a more thorough grid search. Due to the time constraint in this rebuttal phase, we will update the results in the final version.

7. Q5: I'm curious about the method of determining $\alpha_i$, which seems to be missing the reason(s) in the paper. Does it have a specific value for each augmentation in the set, or does it choose an average weight?

We simply use an average weight for different image transformations, as it was done in SVEA.

8. Q6: In Figure 2, the batch size of DrQ reproduced in the results is 256 instead of 512 in the original DrQ.

We ran the experiments of DrQ and our method in the environments listed in the question with batch size of 512:

We can see that

1. Our method is insensitive to the choice of batch size.
2. Without retuning the hyperparameter, our method always matches or outperforms DrQ in these environments with these two choices of batch size.
3. Considering the modest computational resources we have and the huge increase in computational cost due to usage of batch size = 512, it is fair to compare sample efficiency of all the methods with the batch size of 256.

Please let us know if you have any other comments or concerns!

Thank you for summarizing our work and highlighting that our theoretical analysis is rare and valuable compared to many similar work. Below, we provide our answer to the weaknesses and questions.

1. W1: The work in this paper seems to be equivalent to adding an explicit regularization to implicit regularization, is it equivalent to DrQ combined with DrAC in terms of functionality?

Please check point 1 in our Meta Reply.

Roughly speaking, our method follows the idea from DrQ to apply invariant image transformations in both the training of the actor and critic, complements it with a KL regularization term in the actor loss and introduces a new regularization term called tangent prop in the critic loss. However, there are three key differences between our method and the simple combination of DrQ and DrAC:

• Our method introduces a new regularization term called tangent prop which is novel in DRL.
• The KL regularization term in our method is not exactly the KL regularization term used in DrAC. The KL regularization term $D_{KL}[\pi_\theta(\cdot|s),\pi_\theta(\cdot|f_\mu(s))]$ used in DrAC only augments the state on one side while the KL regularization term $D_{KL}[\pi_\theta(\cdot|f_\eta(s),\pi_\theta(\cdot|f_\mu(s))]$ in our method augments the states on both sides. The regularization term in our method is somehow equivalent to using an averaged policy as the target, which may be preferred. The detailed analysis is presented at the end of Section 4.2 and the experimental comparisons between these two KL regularizations are shown in Figure 2. The two lines which are respectively labeled as KL_fix_target and KL correspond to the KL from DrAC and the KL in our method.
• Our method is compatible with using more than one type of image transformation in the training which is required to achieve state-of-the-art performance in both sample efficiency and generalization ability. Moreover, we also provide an analysis and suggest a criterion for the usage of complex image transformations.

2. W2: How the proposed algorithm addresses the initial question "Although they empirically demonstrate the effectiveness of data augmentation for improving sample efficiency or generalization, which technique should be preferred is not always clear.".

The goal of this work was to analyze existing methods and make recommendations on how to use data augmentations in a more principled way in DRL. We start our analysis by connecting the implicit with explicit regularization and indicate some preferences about them. For example, the actor loss in implicit regularization can only enforce the invariance in the actor under the condition that the invariance in the critic has been learned. This motivates our inclusion of an explicit KL regularization term in the final actor loss of our method. Note that we justify most of our recommendations in Section 5.2. For example, we recommend applying complex image transformations in calculating the target Q values and provide a criterion for judging if an image transformation should be considered as complex or not. All in all, we tried to judge different techniques with our theoretical analysis and validate our propositions empirically.

3. Q1: Please check that the two terms in the KL divergence in Eq. 8, and there seems to be an ambiguity with Eq. 4 and Eq. 9.

There is actually no ambiguity here. The KL term in Equation 8 is different from the KL terms in Equation 4 and Equation 9, as listed below:

• Only the state in one side of the KL term in Equation 8 is augmented while the states in both sides of the KL term in Equation 4/Equation 9 are augmented. The differences between them are explained in our answer 1(b) to W1 above.
• The directions of the KL regularization terms are different. In Equation 8, the stop gradient is attached to the $\theta$ on the right hand side. Considering that KL is asymmetric, it is necessary to discuss the difference which is presented in the second paragraph below Proposition 4.1. In conclusion, we found that using the stop gradient in the first term is better and including an explicit KL term in the actor loss is preferred.

4. Q2: How to prove that the distributions of $\nu$ and $\mu$ in Eq. 11 and Eq. 12 are the distributions of $\hat\nu$ and $\hat\mu$ satisfying Lemma 1?

   Note that in Eqns. 11 and 12, the distributions of $\nu$ and $\mu$ are not restricted. They are usually set to be uniform distributions, but other distributions could be chosen depending on the application domain.

   Lemma 1 simply states that for any distributions $\nu$, implicit regularization in the critic with specifically chosen $\hat\nu$ and $\hat\mu$ (see Proof of Lemma 1 for details) is equivalent to explicit regularization in the critic with $\nu$.

Thank you for indicating our contributions of comprehensive theoretical analysis and concise experimental validations. Below, we provide an answer to the points corresponding to the weaknesses.

1. W1: More insights into the limitations and potential failures of the proposed method should be discussed.

Please check point 2 in our Meta Reply.

2. W2: Further analysis and comparisons with a wider range of existing techniques should be conducted to showcase the advantages and limitations of the proposed method in different scenarios.

Please check point 3 in our Meta Reply.

Please let us know if you have any other comments or concerns!