A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations

We thank the reviewer for your thoughtful comments. We address individual comments in the following.

[Weakness]

[W1] Need for an explicit articulation of its motivation and contributions

(A) We will add a paragraph to clarify the motivation and contribution of our paper.

---

[W2] Lack of in-depth analysis of the 'Video Easy' benchmark

(A) As for the `video easy’ benchmark, it is much more generalizable than the 'Video Hard' setting, as it features simpler backgrounds and scene structures. Therefore, given that the issues in video hard are resolved, we assume that video easy, which shows high test performance, is also addressed and did not analyze it separately. In addition, the performances for Video Easy of existing methods already are sufficiently saturated performance, and thus there is no significant performance difference between our method and existing methods. Therefore, we did not particularly emphasize the performance on Video Easy. The significance of the results for Video Easy in Table 1 is that our method demonstrates consistently excellent performance across both Video Easy and Video Hard environments, without being specifically favored to any particular environment.

---

[Question]

[Q1] Need for a more detailed explanation of the results of 'Video Easy' in Table 1

(A) As discussed in W2, the significance of the results for Video Easy in Table 1 is that our method demonstrates consistently excellent performance across both Video Easy and Video Hard environments, without being specifically favored to any particular environment.

---

[Q2] Does the strong performance in 'Cartpole, Swingup' and 'Cheetah, Run' indicate that the impact of the study might be constrained?

(A) Existing methods show limited performance on 'Cartpole, Swingup' and 'Cheetah, Run' tasks compared to other relatively easy tasks where the performances are achieved to some extent. In contrast, our method provides consistently sufficient performance across all tasks, regardless of the type and difficulty of the task. Namely, while we see relatively greater performance improvements in the previously underperforming 'Cartpole, Swingup' and 'Cheetah, Run' tasks, we also achieve relatively small yet meaningful gains in other tasks. This is confirmed by the TID metric analysis for all tasks presented in Appendix D.3. In conclusion, we would like to emphasize that the impact of our research is not limited.

Dear Reviewer t66L,

We sincerely appreciate your thoughtful response and the time you've dedicated to reviewing our paper. We are strongly encouraged by your recognition of our work. Thank you once again for your thorough review and your positive evaluation.

Sincerely,

Authors

We thank the reviewer for your thoughtful comments. We address individual comments in the following. Additionally, we sincerely appreciate your eagerness to engage in further discussions regarding our work!

[Weakness]

[W1] Clarity w.r.t the attribution masking argument

(A) To avoid confusion, we clarify that our paper does not claim that an RL model must make a decision based on the ‘whole’ of task-relevant objects for better policy generalization. Rather, we emphasize that it is important for the RL model not to be disturbed by task-irrelevant backgrounds. To do this, we assumed that the RL agent should at least focus more on the task object than on the background, and employed the attribution masks to analyze whether the trained RL models correctly distinguish the task objects and backgrounds. Additionally, to quantitatively measure and analyze the ability to distinguish, we proposed the TID metric. We clarify that this metric is not designed to measure whether the models are making decisions based on the entire object.

To investigate whether the ability to distinguish between task objects and backgrounds also helps in identifying the key points that influence decisions, we provide attention maps obtained from the softmax of the attribution maps $M(Q_\theta, s_t, a_t)$ before thresholding for the masks $M_\rho(Q_\theta, s_t, a_t)$ in the attached PDF. Figure 3 of the PDF shows that the identification ability for the task objects of SimGRL also leads to focus on the most important key point parts such as the pole and body of the 'Cartpole’ or legs of the 'Cheetah’ in the attention maps. On the other hand, the SVEA baseline incorrectly considers the background parts as key points.

---

[W2] TID Metric when a few parts of the agent are responsible for achieving reward

(A) As addressed in W1, we do not assume that the entire object is responsible for the policy's decision; instead, we assume it is important for the agent not to be disturbed by task-irrelevant backgrounds. Therefore, TID metrics are designed to measure how well the task-relevant object and background are distinguished, without considering which parts within the task-relevant object are more important.

---

[W3] Need for improvement of Figure 4

(A) We will improve Figure 4 by highlighting a distinction between the existing pipeline of the SVEA baseline. The modified figure is presented in Figure 2(b) in the PDF.

---

[W4] Additional experiments for RL-ViGen benchmark

(A) Due to the time constraints of the rebuttal period, we were only able to conduct a limited number of additional experiments on the RL-ViGen benchmark task. Specifically, we present the experimental results on the 'Door' task in the Robosuite environment.

This table demonstrates that SimGRL achieves test performance gain against the baseline method in different simulation environments as well.

---

[W5] Measurement using IQM metric

(A) We evaluated the IQM (interquartile mean) scores, where we discarded the top and bottom values over the 5 seeds, and averaged the middle 3 seeds.

The table shows that SimGRL also achieved robust performance in IQM evaluation.

---

[Question]

[Q1] How does a baseline where we have different data augmentations for every frame in the history of stacked frames perform?

(A) To augment the background, it is better to apply the augmentation non-uniformly. However, if the augmentation directly affects the agent, using uniform augmentation would be more appropriate. For example, if random shifts are applied differently, the agent's position may change for each frame, which may distort temporal information. Therefore, the random shift (referred to as soft augmentation in the paper) applied by default, like in existing methods, was applied uniformly. On the other hand, we confirmed that shifting the random overlay, which has a greater influence on the background than the agent, is effective for generalizing to environments including dynamic perturbations.

---

[Q2] In the Robotic manipulation results what does Test {1, 2, 3} mean? Does it mean running an evaluation on a single episode for 5 seeds?

(A) Robot manipulation experiments were conducted by testing in each of the test environments—test 1, test 2, and test 3—after training in the training environment. Each test environment was evaluated with 30 episodes, and this process was repeated across a total of 5 different seeds for training.

Dear Reviewer wvnF,

We sincerely appreciate your thoughtful response and the time you've dedicated to reviewing our paper. We are strongly encouraged by your recognition of our work. We will incorporate your suggestions and insights into the revised manuscript. Specifically, we will add some sentences to avoid confusion and enhance the clarity of our claims. Thank you once again for your thorough review and your positive evaluation.

Sincerely,

Authors

We thank the reviewer for your thoughtful comments. We address individual comments in the following.

[Weakness]

[W1] Comparison with robust RL algorithms

(A) We clarify that our work aims to address the generalization issue in a purely model-free vision-based RL setting, which requires the frame stack to encode temporal information. Due to the limitations of the rebuttal period, we present comparative results for DBC[1] and DRIBO[2] among the robust RL algorithms to investigate whether these methods also suffer from imbalanced saliency and compare them with the proposed method.

• DBC: Image-level frame stacking of 3 images, CNN encoder, Representation learning using a dynamics model, No strong data augmentation (e.g., Random overlay).

• DRIBO: No frame stack (using single images), Recurrent State-Space Model (RSSM) encoder, Representation learning using a dynamics model, No strong data augmentation (e.g., Random overlay).

• SimGRL (Ours): Feature-level frame stack of 3 images, CNN encoder, No representation learning and a dynamics model, Strong data augmentation (i.e., Shifted Random overlay).

To investigate whether these methods suffer from imbalanced saliency, we investigated TID metrics on 'Cartpole, Swingup', and compared them with SimGRL

This table indicates that 1) DBC suffers from imbalanced saliency (a lower TID Score and higher TID Variance) and 2) DRIBO does not suffer from imbalanced saliency and can correctly recognize individual images (a higher TID Score and lower TID variance). Additionally, for better understanding, we provide the qualitative results of the masks and masked images for DBC and DRIBO in Fig. 1(a) of the attached PDF. These results imply that, when using an image-level frame stack as in DBC, imbalanced saliency persists even when the model-based RL setting is integrated. In contrast, when encoding single frames individually as in DRIBO, imbalanced saliency can be avoided, similar to feature-level frame stacking of the proposed method.

However, our SimGRL has the advantage of being a simple yet effective structure, as it uses neither complex RNN-based networks like RSSM nor any additional representation learning strategies. In Figure 1(b) of the attached PDF, the training curves indicate the high sample efficiency in the training stage and superior test time performance of SimGRL.

---

[W2] Discussion on attribution masks in different settings

(A) In Section 5.2 and Appendix D.3, we have conducted an in-depth analysis of the pitfalls regarding clean videos and video hard scenarios. Additionally, in Appendix D.4, we have included rich example images of various attribution masks to demonstrate how effectively SimGRL addresses these pitfalls compared to existing methods. We would like to request you to refer to these sections. As for the 'Video Easy’ setting, it is much more generalizable than the 'Video Hard' setting, as it features simpler backgrounds and scene structures. Therefore, considering that the issues in the 'Video Hard’ setting are resolved, we assume that in the `Video Easy’ setting, which shows high test performance, the attribution mask is also well predicted, so we did not analyze it separately.

---

[W3] Figure 5 that may be misleading

(A) For better clarity, we will add the augmentations used during training in Figure 5. We provide the modified figure in Figure 2(a) in the attached PDF.

---

[Question]

[Q1] Whether the pitfalls also apply to robust RL algorithms

(A) As explained in W1, even robust RL algorithms can experience pitfalls when using image-level frame stacks, as seen with DBC, but these pitfalls will be resolved when sequentially embedding single images, as done in DRIBO.

---

[Q2] How different attribution methods may affect the identified pitfalls and TID metrics

(A) We clarify that the attribution masks are used solely for analysis and not for training, so they do not impact whether the RL models fall into pitfalls during training. Therefore, even if different methods are used to obtain attribution masks, as long as they correctly measure the influence of pixels in input images on the RL model, the identified pitfalls and TID metric values will not change significantly.

---

[1] "Learning invariant representations for reinforcement learning without reconstruction." ICLR (2021).

[2] "Dribo: Robust deep reinforcement learning via multi-view information bottleneck." ICML (2022).

Thank you for the further clarification!

The results on applying random overlay augmentation on DBC and DRIBO are particularly helpful. I would rather disagree with the setting where DBC and DRIBO are compared with SimGL but trained under the clean setting. Both DBC and DRIBO did not claim transferability of robust RL agents trained under the clean setting. I would recommend author to include the second part of results in the revised version.

In addition, I would strongly recommend authors to compare with more recent robust RL methods in the revised version.

I would like to raise my score to 6 and this is a borderline paper to me. I think the major contribution of the paper lies on the random overlay augmentation and the TID metric. However, more comparison with other robust RL methods can further strengthen the paper.

We thank you for your constructive feedback and your eagerness to engage in further discussions regarding our work!

We address your comments in the following.

---

[Q1] Are DBC and DRIBO trained with the same video augmentation as SimGL or trained in the clean setting?

(A) To clarify whether robust RL methods suffer from the imbalanced saliency problem, and since both DRIBO and DBC did not originally use strong augmentation such as random overlay, clean images were used during training. This implies that even when methods like DRIBO, which learn robust representations, do not suffer from issues such as imbalanced saliency, training on diverse backgrounds remains important for generalization to unseen environments.

---

[Q2] Can existing robust RL methods also benefit from the random overlay augmentation?

(A) To investigate whether existing robust RL methods also benefit from augmentations like random overlay, we provide experimental results on the 'Cartpole, Swingup' task in the 'Video Hard' test environment for DBC and DRIBO when using random overlay augmentations. Following SODA [1], we applied random overlay augmentations solely to the representation learning losses of the encoder for stability in training.

The above table indicates that DBC showed marginal effects even with the application of random overlay, while DRIBO demonstrated clear improvements. Additionally, DRIBO was able to obtain additional performance gain when using the shifted random overlay augmentation. For DRIBO, these results clearly suggest that robust RL algorithms can benefit from data augmentation techniques such as random overlay, further strengthening our contribution. On the other hand, DBC's results suggest that a specific augmentation method may not always be well-suited for a particular learning algorithm. Alternatively, this could be due to issues with hyperparameter tuning in methods with complex representation learning processes, where the marginal performance might be attributed to not finding the optimal hyperparameters for applying the augmentation. Our approach, with its simple structure and minimal hyperparameter tuning, could be a solution to address these drawbacks.

---

References

[1] "Generalization in reinforcement learning by soft data augmentation." ICRA (2021).

We thank the reviewer for your thoughtful comments. We address individual comments in the following.

[Weakness]

---

[W1] Dependency on augmentation quality

(A) The effectiveness of the shifted random overlay augmentation may depend on the quality and diversity of the natural images, and the performance may be limited in specific real-world scenarios where such images are unavailable or unsuitable. Nevertheless, we demonstrated in Q1 that even with fewer and less diverse augmentation data, robust performance can be maintained.

---

[W2] Ad Hoc approach

(A) SimGRL consistently demonstrates robust generalization performance across various distraction environments and task scenarios, including DMControl-GB (Section 5.1 and Appendix B.6), DistractingCS (Appendix B.7), and robotic manipulation (Appendix B.8). Therefore, we do not consider this to be overfitting to specific tasks.

---

[W3] Applicability to real-world scenarios

(A) Because it is very difficult to build a safe real-world RL experimental environment using actual robots, existing studies have primarily verified algorithms in simulation environments. Similarly, we conducted experiments in simulation environments to compare our method with existing methods. Nonetheless, applying the algorithm to more realistic real-world situations is crucial for practical use, and we also consider this an important area for future research.

---

[Question]

[Q1] Impact on the diversity of natural images

(A) We provide the average results of SimGRL for the training set and validation set of the Places dataset used for augmentation on the DM-Control-GB benchmark.

This table implies that even with fewer types of natural images, the performance does not significantly degrade, implying that the diversity of natural images does not have a major impact on generalization performance. More effective types of images for augmentation would be those that resemble the distribution of the test environment. However, since we do not have prior information about the test environment, we used arbitrary natural images for augmentation and confirmed through various experiments that this approach is effective.

---

[Q2] Challenges anticipated in real-world scenarios

(A) A potential challenge when testing in real environments is the distribution shift, where the distribution of natural images used for augmentation during training differs from the distribution in the actual environment. However, our approach trains the model to focus on task-relevant objects while ignoring the background. As these task-relevant objects should consistently exist even in new environments with different backgrounds, we expect SimGRL to perform well in such cases.

---

[Q3] Perturbations that are not present in simulated benchmarks

(A) The robust performance of our method in DistractingCS in Appendix B.7, which includes perturbations such as camera pose distractions, suggests that our approach can handle types of disturbances not present in the training environment. We speculate that SimGRL’s robustness to distractions in this benchmark stems from its excellent ability to identify salient objects in the input, which helps in generalizing against distractions such as dynamic backgrounds or changes in camera pose.

---

[Q4] Effects of numbers of frames in the feature-level frame stack

(A) The number of stacked frames involves a trade-off between the amount of temporal information and the computational cost for encoding. If the number of stacked frames is reduced to less than three, the performance will degrade due to insufficient temporal information. On the other hand, since a feature-level frame stack encodes individual images separately in the image encoder, the number of encodings required increases in proportion to the number of frames, resulting in higher computational costs. Therefore, we adopted stacking three frames as in the previous methods and experimentally confirmed that this is the most appropriate frame stack.

---

[Q5] Sensitivity of the performance to the choice of hyperparameters

(A) As analyzed in Appendix B.3, the sensitivity to the number of layers in the image encoder was not highly critical. Even with just one layer, using a feature-level frame stack demonstrates significantly better generalization performance compared to an image-level frame stack.

Additionally, to show the impact of varying the maximum shift length (L) for the shifted random overlay, we present the test results on the "Cartpole, Swingup" task in 'Video Hard' for SimGRL-S, which uses only shifted random overlay augmentation without a feature-level frame stack.

In this case, sensitivity to the maximum shift length is not very high, but there is a significant difference compared to the case of not using any shifting (L=0).