References

[1] Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations (Lu et, al. TMLR 2023)

[2] Principled Offline RL in the Presence of Rich Exogenous Information (Islam et, al. ICML 2023)

[3] "Information prioritization through empowerment in visual model-based RL." Bharadhwaj et al. (ICLR 2022)

[4] “RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictability” Zhu et al. (NeurIPS 2023)

Response to Reviewer $\text{\textcolor{green}{UDWw}}$

Thank you for taking the time to review our paper. Please let us know if there are further questions or necessary clarifications and we would be happy to address them during the discussion period.

W1. Although the authors' claim primarily in the visual offline RL, the conducted experiments are strictly limited to MuJoCo variants. The authors are strongly encouraged to strengthen the experiment section by including (pure) vision-based RL problems with distractions.

Regarding the benchmarks that we have considered, we closely follow previous work in visual offline RL [1, 2] where DeepMind Control Suite is considered to be the standard benchmark. Our setting is purely vision-based RL problems, using only image observations without access to ground-truth states, with added distractions. We recommend the reviewer refer to Section 2.1 and the Datasets subsection in Section 5. Additionally, we have included results from other environments.

The table shows the result on an additional environment on DeepMind Control Suite called Finger-Spin and Cartpole-Swingup, which were used in previous works [3, 4]. We refer the reviewer to take a look at Appendix F for detailed analysis.

W2. Extensibility. I think the adaptation ability of CLEAR representation for different visual distractions from the training environments is only partially validated by multiple videos. The authors are encouraged to analyze general extensibility of CLEAR in these aspects for real-world scenarios.

We strongly believe that the multiple video distraction scenario well covers the generalization of CLEAR representation. In Appendix E, we have shown that unseen distractions can be viewed as a problem of distribution shift, which can be mitigated by, for instance, random convolutions. Thus, we view any alternative approach which can mitigate this problem of distribution shift as compatible with our framework. We note that one of the underlying assumptions with our method as well as baselines is that the presentation of the agent remains the same while the distractions can change. If the reviewer is asking, for example, to test the CLEAR encoder on 2x2 while trained on Single Video, this breaks the aforementioned assumption since the presentation of the agent is different in those two scenarios (agent size and location in images). We ask the reviewer to provide a detailed suggestion on alternative experimental setups.

W3. It would be better if intuitive motivation or a toy experiment empirically verifies the ExoPOMDP setting.

We have provided an intuitive real world example in which visual observations often contain complex distractions, such as background screens playing video advertisements or birds flying in the sky, which are irrelevant to the control task at hand. Please see Figure 1 in the paper for further illustrations.

Q1. In the graphical model in Fig 2, what happens when $e_t$ is affected by $s_{t-1}$? Does this modification change Eqs. (2) and (3)?

We characterize distraction as a factor that 1) does not affect the reward function, 2) is unaffected by action, 3) is independent of the agent state, and 4) is present in the observation. When $e_t$ is affected by $s_{t-1}$, we can no longer call the distraction $E$ independent of agent state $S$, violating the definition of distraction. Thus, we can no longer call it distraction $E$ but rather part of the agent state $S$ and the ExoPOMDP reduces to a POMDP.

To give an example, consider the floor part of the visual observation (Figure 4). Although the floor is uncontrollable, the inferred state captures the floor as a controllable part. This is because how the floor moves depends on how the agent moves. We also make an additional example where the floor does not correlate with the agent state in the Finger environment (Appendix F in the updated manuscript). In such a case, the floor is inferred as part of the exogenous variables.

S1. I think most of the results could be implemented and reproduced even though source code is not provided.

We would like to correct the reviewer that we have provided the source code in https://anonymous.4open.science/r/anonymous-clear-EFFC as we put in footnote 2 (Line 375)

Q4. How can this method handle other forms of nuisance information, such as color perturbations of task-relevant parts, where the assumption in L278 does not hold?

In L278, we assume the state variables and exogenous variables occupy different parts of the visual observation thus we employ a compositional decoder commonly used in object-centric representation learning as an architectural choice which does not affect the main proposed objective.

Following along with the example of color perturbations, the answer depends on how the color perturbs the visual observation. We characterize distraction as a factor that 1) does not affect the reward function, 2) is unaffected by action, 3) is independent of the agent state, and 4) is present in the observation. If the color perturbation only changes the color of the agent, then the color is dependent on the agent states which violates the definition of distraction that we have established in the first paragraph of section 2.1. Thus, the color can no longer be called a distraction rather as part of the agent state.

As an example, consider the floor part of the visual observation (Figure 4). Although the floor is uncontrollable, the inferred state captures the floor as a controllable part. This is because how the floor moves depends on how the agent moves.

We also make an additional example where the floor does not correlate with the agent state in the Finger environment (Appendix F in the updated manuscript). In such a case, the floor is inferred as part of the exogenous variables.

Q5. Including more detailed descriptions of each column in the caption of Table 3 would make it much easier to interpret the results.

Thank you for the feedback. We have modified the caption of Table 3 in the updated manuscript.

References

[1] "Information prioritization through empowerment in visual model-based RL." Bharadhwaj et al. (ICLR 2022)

[2] “Dream to control: Learning behaviors by latent imagination.” Hafner et al. (ICLR 2020)

[3] “RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictability” Zhu et al. (NeurIPS 2023)

Response to Reviewer $\text{\textcolor{blue}{UCLi}}$

Thank you for taking the time to review our submission and providing constructive feedback. We address each concern one-by-one below. Please let us know if there are further questions or necessary clarifications and we would be happy to address them during the discussion period.

W1. Three tasks from a single domain seem too few, particularly since some baselines are comparable to the proposed method in some tasks (e.g. InfoGating on Hopper Hop). Expanding the evaluation to include more tasks (e.g., 5-6 tasks) would significantly strengthen the paper. A larger set of tasks would provide a more robust assessment of the method's performance and demonstrate its versatility across different scenarios.

We have provided results on additional environment(s) on Appendix F in the updated manuscript.

The table shows the result on an additional environment on DeepMind Control Suite called Finger-Spin and Cartpole-Swingup, which were used in previous works [1, 3]. We refer the reviewer to take a look at Appendix F for detailed analysis.

W2. Although a large set of prior works is presented as baselines, some important and closely related studies are missing. Notable examples include [1] and [2]. [1] also employs information theory to address controllability. A detailed comparison between [1] and CLEAR is necessary, both methodologically and experimentally. [2] leverages causality to learn disentangled representations across four different categories, focusing on controllability and reward relevance.

We have incorporated the suggested baseline, InfoPower [1], in the updated manuscript (Table 1 and Table 2). Since its source code is unavailable, we reimplemented the method to the best of our effort given the limited details on the paper. We verified our implementation by checking that the ground-truth state regression yielded values within a reasonable range. Despite having relatively low MSE on ground-truth state regression (Table 2), the offline RL result for all environments and distractions are significantly lower than that of CLEAR as well as other baselines (Table 1). We argue that this is due to the contrastive loss which tends to perform poorly in practice when compared to the reconstruction loss [2].

Q1. Which representation is used for offline RL (i.e., policy learning)? I assume $\hat s$ is used and not $\hat e$, but I would like to clarify this.

Yes, $\hat s$ is used for offline RL. Please see L272 in the paper.

Q2. The method requires extensive hyperparameter search over the variance and balancing factors of the KL divergence for each task, domain, and distraction type (Table 9). It would be much more convenient to eliminate this necessity. Do you have any ideas on how to achieve this?

Our method requires only three hyperparameters. In our experiments, we found that (1) the KL weights $c_E$ and $c_S$ are the most important and (2) the qualitative results (e.g. Figure 4) can be helpful in tuning hyperparameters efficiently.

For the easy level, we can simply set $c_E = c_S = 1$ and only search over $\sigma^2 \in \{0.1, 0.02, 0.04 \}$. For the medium level, we searched only over $\sigma^2$ and $c_E$ while $c_S=1.0$ remains fixed. We found that without setting $c_E < c_S$, the model tends to put all representations inside $S$. Finally, for the hard level, we found that additional tuning of $c_S$ was necessary due to the fact that the size of the controllable agent now occupies a smaller segment of the image.

We note that complete elimination of hyperparameter search is infeasible for both our method and other baselines since it is not possible to evaluate the policy offline.

Q3. Since the method aims to improve representation learning, it could potentially be applied within an MBRL framework. How do you think it would perform when integrated with an MBRL framework?

We thank the reviewer for pointing out the potential applicability of our representation learning to (offline) MBRL framework. In principle, we can use the variational distribution $q_\phi(\hat s_t|\hat s_{t-1},a_{t-1})$ as the dynamics model for the MBRL framework and then train an additional reward predictor. While it can be leveraged, offline MBRL challenges still remain on the implementation side e.g. which pessimism heuristics to be used, the weight for the pessimism heuristics, as well as the percentage between the real trajectory and “imagined” trajectory.

Response to Reviewer $\text{\textcolor{red}{7K7j}}$

Thank you for taking the time to review our submission. Please let us know if there are further questions or necessary clarifications and we would be happy to address them during the discussion period.