DEXR: A Unified Approach Towards Environment Agnostic Exploration

We want to thank the reviewer NQDk for the effort in reviewing the paper. We address the questions and respond to the concerns in the following.

Q1: "Comparing the results ... does not outperform TD3."

A1: In our PointMaze experiments, DEXR consistently outperforms TD3 across various layouts, achieving significantly better results. Specifically, we observed that TD3 requires at least twice as long to reach the goal point compared to DEXR. This demonstrates the superior efficiency and effectiveness of DEXR in these scenarios.

Regarding the MuJoCo Locomotion tasks, our results indicate a more nuanced performance comparison. While DEXR does not show a substantial advantage over TD3 in environments with dense rewards, it notably surpasses TD3's performance in most environments characterized by sparse rewards. This distinction highlights DEXR's enhanced adaptability and proficiency in more challenging settings with limited feedback.

Q2: Furthermore, the baseline exploration methods reported all perform worse than TD3. Why does this happen? Since these methods are implemented on top of TD3 it would be strange if they had worse performance than the backbone RL method without any extrinsic rewards.

A2: As discussed extensively in Sections 1 and 2, methods like Exp, which rely on intrinsic rewards, tend to be distracted and exhibit unstable performance, failing to exploit efficiently when the hyperparameter is not carefully tuned. This is the precise challenge DEXR aims to address.

As detailed in Section 5, all algorithms, including TD3, are trained using extrinsic rewards. Notably, the TD3 agent is trained without the addition of intrinsic rewards.

Concern 1: The method is incremental compared to work like Go-Explore.

Response 1: We regret any misunderstanding regarding the relationship between DEXR and Go-Explore. While there is a superficial similarity, DEXR embodies a fundamentally different design philosophy and operates within a markedly distinct setting.

Firstly, Go-Explore's methodology hinges on the environment being deterministic or the availability of a local model (resettable environment) that can relocate the agent to a previously visited state. Contrarily, DEXR operates independently of these requirements.

Secondly, Go-Explore relocates the agent to regions that are rarely visited previously, whereas DEXR relocates the agent to regions that are believed to be fruitful according to the extra exploitation policy. Despite these two paradigms being similar in form, DEXR and Go-Explore hold drastically different ideas and intuition. To be specific, Go-Explore enables more efficient over-exploration, by leveraging the special assumptions and privilege accesses to accurately relocate the agent to “regions not well covered”, so that the agent can fully cover the whole MDP, including regions that might not be worth exploring. In contrast, DEXR prevents over-exploration and does not need any special assumptions, by employing an extra exploitation policy to relocate the agent to regions that are “worth exploring”, which largely mitigates the hyperparameter sensitivity, as discussed in Section 1 and Section 4 in detail.

The effectiveness of DEXR, while straightforward and intuitive, is underscored by our comprehensive empirical analysis. This analysis examines the behaviors of the exploitation and exploration policies in navigation and evaluates the capacity of the exploitation policy to effectively guide the exploration policy.

We have already put a discussion on the relationship between DEXR and the Go-Explore algorithm in the related work section, the reviewer can refer to the revised manuscript for a clearer context.

Concern 2: "There are missing baselines ... TD3 baseline."

Weakness 2: We acknowledge the inquiry regarding the comparison with TD3. For your reference, this comparison can be found in the context of the PointMaze experiment. Additionally, we provide visual representations of the behavior of various algorithms, including TD3, within the PointMaze environment, detailed on page 8 of our document.

We apologize for the omission of the split plots and the comparative plot against TD3 in the appendix. We have since rectified this oversight, and these plots are now included in the updated version of the manuscript and can be found in Appendix A.

We want to thank the reviewer CdTY for the effort in reviewing our paper, we address the questions and concerns in the following.

Q1: How does $\beta$ influence the performance of each algorithm?

A1: We apologize for the oversight concerning the absence of split plots in the appendix of our manuscript. We have already put the split results in Appendix A, which describes the influence of $\beta$ on each algorithm.

Q2: Are there any justifications for setting $p$ to $1-\gamma$? Also, why is $p$ annealed to $\frac{1}{H}$ during the course of the training?

A2: Let me walk through the key choices in detail:

Using $p=1-\gamma$ for truncation probability initially:

• The discount factor $\gamma$ determines the effective horizon $H^e=1/(1-\gamma)$
• This horizon suggests the steps needed to propagate rewards back to early states
• Therefore, letting the exploitation policy run for H steps on average via p=1-γ allows it to reach promising states
• Formally, if $T \sim Geom(p)$, then $\mathbb E[T]=1/p=1/(1-γ)=H^e$

Decaying $p$ over time:

• As exploitation policy improves, less exploration is needed near the initial point
• Reducing p increases $\mathbb E[T]$ since $\mathbb E[T]=1/p$
• This lets exploitation run longer to fine-tune on higher-value regions
• $p$ is decayed gradually to $1/H$, to gradually increase $\mathbb E[T]$ to $H$

Avoiding excessive pointless exploitation:

• Despite higher $\mathbb E[T]$ later, $T$ still follows a geometric distribution
• So there are always probabilistic chances of truncating early
• Formally, $\mathbb P(T=t)=(1-p)^{t-1}p$ still enables occasional exploration in region near the initial point
• Experiments confirm this prevents issues of over-exploitation

Per your feedback, we have already added some discussion on this design choice in Section 4 for more information.

Q3: Typos and minor suggestions.

A3: We extend our apologies for any inconvenience caused by these errors. Additionally, we want to clarify that in Equations 3 and 4, the term $\mathbb{P}V(s, a)$ is defined as $\mathbb{E}_{s’ \sim \mathbb{P}(\cdot|s, a)} V(s')$, as elaborated in the paragraph subsequent to Equation 2.

Concern 1: First of all, Appendix B is still missing, which contains important empirical results to validate the paper's central claim. The paper claims to find an algorithm that has a reduced sensitivity to the exploration hyperparameter. Then, how the algorithm (and baselines) reacts to different values of the exploration hyperparameter should be shown. Also, I think it would be better if it’s in the main text.

Response 1: We again apologize for the oversight concerning the absence of split plots in the appendix of our manuscript. To rectify this, we have now included a separate plot in Appendix A.

Concern 2: The effect of the extra hyperparameter, the truncation probability, is not discussed adequately. The paper only mentions that it is set to $1-\gamma$. In addition, there is an annealing of this parameter, which is not mentioned in the main text nor investigated thoroughly. Are there any justifications for these design choices?

Response 2: We have addressed this concern in Q2, the reviewer can refer to the details in our answer A2.

Concern 3: There are still a lot of typos and reference format issues in the paper. Please carefully fix them. See Questions for an unexhausted list.

Response 3: We appreciate your suggestion and would like to inform you that the necessary corrections have been made.

We would like to thank the reviewer Nqr6 for the suggestions and the questions. We address the concerns and the questions in the following:

Q1: "Regarding Figure 2 ... not help in reward-free exploration."

A1: Thanks for raising this point. We would like to clarify that DEXR is primarily designed to enhance stability across hyperparameters,  and thus enable efficient exploration in an environment-agnostic manner, rather than to maximize exploration in environments without reward. Nevertheless, despite it is not designed for the reward-free setting, our theoretical analysis shows that it would perform on par with standard intrinsic reward exploration algorithms, which indicates that DEXR can also explore efficiently under the reward-free setting per the findings in [1].

Q2: "Since off-policy approaches can be more sample-efficient ... PPO need to be used with EIPO?"

A2: Thank you for raising the thoughtful question about using TD3 as an alternative backbone for EIPO. You make an excellent point that off-policy algorithms could provide advantages. There were two key factors in preserving the PPO backbone in EIPO:

1. EIPO relies on accurate policy value estimates to switch between exploration/exploitation. As an on-policy method, PPO better fits this need.
2. Significantly changing the original PPO-based EIPO risks invalidating the baseline.

Q3: "Regarding the results presented in Figure 6 ... reward coefficients $\beta$?"

A3: We apologize for missing the split plots, we have already restored the split plots in Appendix A to provide more detailed information on the experimental results.

Concern 1: "While the proposed approach shows robustness ... curiosity-based approaches succeed."

Response 1: We would like to acknowledge that harder exploration problems exist, such as Montezuma’s Revenge. However, we believe our experiment, along with our theoretical analysis, supports our technical claim about DEXR well, as our focus is to enhance to stability of curiosity-driven exploration methods under different reward sparsity settings.

Concern 2: "The environments considered ... the goal sometimes in the largest maze."

Response 2: The reason why we chose to work with MuJoCo is two-fold:

1. The aim of DEXR is not to outperform the state-of-the-art in the most popular benchmark but to enhance the stability of existing curiosity-driven exploration algorithms.
2. As Suggested in [3], vision representation can affect performance heavily in pixel-based environments. We evaluate DEXR on MuJoCo to eliminate other factors.

PointMaze is a good testbed for exploration:

1. Its observation space contains clear location information, which allows for monitoring visualizing, and analyzing the behavior of the agents.
2. The hardness of an environment should not be defined by whether the random exploration can get the reward, but defined as “how many samples are needed to find the near-optimal policy”. For reference, without structured exploration, PPO can obtain a considerable amount of reward in Montezuma’s Revenge.

Despite TD3 reaching the goal location a few times in the training process in Large-Maze, still fails to escape the suboptimality. And TD3 gets stuck with its sub-optimal policy, it takes at least 2x longer for TD3 to reach the goal compared to DEXR in all tasks.

Concern 3: "To properly evaluate ... why lower values of $\beta$ were not considered?"

Response 3: For PointMaze, the intention was to investigate the impact of small v.s. large intrinsic reward coefficients on the performance and the behaviors of different algorithms.

For MuJoCo locomotion, to evaluate the robustness of DEXR in MuJoCo locomotion tasks, we test various hyperparameters. The exploration & exploitation balance with a higher intrinsic reward coefficient is crucial: higher values encourage exploration in unknown environments but can lead to distractions and unstable optimization. We assess DEXR's effectiveness with coefficients ranging from the most natural choice 1.0 and increase up to 1000.0, examining its ability to handle these challenges.

Concern 4: "It would also be useful ... why p is chosen as 1- gamma?"

Response 4: We have also added some discussion on this design choice in Section 4 for more information, and provide a set of experiments on evaluating how DEXR would respond to different initial truncation probability $p$ and different decay schedules, which can be found in Appendix A.

Concern 5: "The paper misses .. variants (e.g., [4])".

Response 5: We have already added some discussion on the Bayesian RL approach in the related work section.

Reference:

[1]Wang et al. (2020) On Reward-Free Reinforcement Learning with Linear Function Approximation.

[2] Bellemare et al. (2016) Unifying count-based exploration and intrinsic motivation.

[3]Kostrikov et al. (2020) Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.

Thank you for your answers/clarifications. I appreciate the time and effort spent in crafting the reponse. I will maintain my score as some of my main concerns still remain open.

• Thanks for the clarification regarding Q1, it was slightly confusing since I was trying to see how DEXR better explored in that setting.

• I still think applying the idea of EIPO with TD3 could be useful, but I can accept that it is probably not essential due to the reasons you mention.

• Regarding responses 1 and 2, I still feel that harder exploration environments should also have been considered to understand DEXR comprehensively. It is important to know how much of a potential downside (or not) there could be from DEXR in such environments. Harder environments have been used by the considered baselines for stably training with intrinsic rewards. EIPO evaluates in the atari suite, DERL studies DeepSea. About the large point-maze, I agree that obtaining rewards should not be a measure of hardness in general, but I think it is not unreasonable for a sparse reward navigation task. Even if we consider it a relatively hard exploration task, it is still only one example on the simpler side.

• Regarding response 3, could the authors clarify why $\beta=1$ is a natural choice on the lower side? Many (if not most) approaches to intrinsic exploration often use a far smaller value of $\beta$, for eg. see appendix A2 of RIDE (https://arxiv.org/pdf/2002.12292.pdf), which uses coefficients of 0.005 in some cases. The range over which the approaches are evaluated should definitely contain the best values of the intrinsic bonus coefficient. Is that the case in these experiments? Of course, the best value could differ across environments, and that would be a strong motivation for DEXR.

We would like to thank the reviewer CCrp for the suggestions and for bringing up insightful questions. We address the concerns and the questions in the following:

Q1: If you have the truncation probability, why do you need the horizon in the algorithm? Is it an implementation detail to ensure that the exploitation phase does not cover a majority of the episode?

A1: Note that H is an environment parameter of the finite horizon setting. The algorithm takes this parameter and splits the horizon into two parts randomly (via the truncation probability). The algorithm performs exploitation in the first part and exploration in the second part. Indeed, if the environment is infinite-horizon, we can set another truncation probability $1-\gamma$ for the exploration policy.

Q2: In Figure 6, the results are averaged over 5 random seeds and 5 values of $\beta$. It is not standard practice to average over hyperparameters, so I am curious to know why the authors went with this particular decision. Also, the paper mentions that the split over is in the appendix but I was unable to find it. Could the authors clarify where I can find these results?

A2: We have already restored the split plots in Appendix A to provide more detailed information on the experimental results, we apologize for missing these plots in the previous version. Regarding the averaging performance across both hyperparameters and seeds, our rationale was to use such averaged results over the hyperparameters to show the robustness of our method, as the performance is promising, and the standard deviation is much lower than Exp and DeRL. Per your feedback, we move the aggregated plots to Appendix A, and in the main text, we only keep the results with Disagreement intrinsic reward with $\beta=1.0$ and $\beta=1000.0$ for better clarity.

Q3: Bonus (not a request for experiments): I'm curious to know if the authors experimented with iterative exploitation-exploration within an episode.

A3: The paradigm of alternating exploitation and exploitation within every episode is precisely the approach taken by EIPO: EIPO utilizes PPO as the backbone and toggles between exploration and exploitation based on state-value estimates. However, as evidenced by our visualizations in Section 5, this approach is suboptimal. As we have discussed in Section 5 of the paper, the alternative policy switching makes the exploitation policy unable to identify successful trajectories and hence hampers overall performance.

Concern 1: The related work section covers important papers, but I felt a key section was missing, the Go-explore family of algorithms.

Response 1: Thanks for your suggestion to include a discussion on the Go-Explore work in the Related Work section. In response, we have added a discussion on the connection between our method and the Go-Explore method in the related work section.

Concern 2: (First-Explore, then Exploit: Meta-Learning Intelligent Exploration) actually has quite a similar proposition to this paper; there are two policies learned, one that learns only to exploit and one that learns only to explore. While that is in the meta-RL setup, I think it should be mentioned that there is existing work that has suggested similar ideas, but not from an intrinsic motivation perspective.

Response 2: Thank you for the insightful point. Indeed this is quite relevant to our paper, and it shares some similar approach with DEXR. We have added a discussion concerning this work to our related work section.

Concern 3: In the results, the paper mentions that DEXR enjoys notably smaller variances compared to DeRL and Exp but that does not seem to be the case (or at least does not seem to be obvious) in Figure 6. Could the authors provide cleaner plots (with only one DEXR, one Exp, and one DeRL variant, instead of the full set of plots)?

Response 3: We have put cleaner results in the main texts, and added the detailed split plots in Appendix A.

I thank the authors for answering the questions I had and addressing some of the concerns particularly in the related work section. I have gone through the revised version and also concur with the concerns raised by other reviewers regarding the inclusion of another hard exploration task, and I am thus keeping my current score.