Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus

We gratefully thank the reviewer for recognizing our contributions, writing and novelty. The additional ablation experiments have been included in the PDF file and all the references mentioned in the response can be found in the reference list in the global comment. We provide our response below:

W1: To my knowledge other methods that try to aggregate states in high dimensions are not considered. Wouldn’t this mitigate or solve the issue with unique counts in high-dimensional spaces?

Response: Thank you for the insightful question. Indeed, one alternative to address the issue of unique counts in high-dimensional spaces is to design a function that extracts relevant features from each state and feeds them into the count-based episodic bonus. Specifically, instead of defining a bonus using $N_e(s_t)$, we could define the bonus using $N_e(\phi(s_t))$, where $\phi$ is a hand-designed feature extractor. In our experiments, following the RIDE implementation in [G], in the Habitat environment, the RIDE and NovelD implementations use $\phi(s_t) = (x_t, y_t)$, where $(x_t, y_t)$ is the spatial location of the agent at time t, so the unique state count issue is solved. However, despite this, the performance of RIDE and NovelD still lags behind in Habitat and the state counts is still very sparse. Additionally, the aggregation of states using a feature extractor relies heavily on task-specific knowledge, which significantly restricts scalability. Another approach involves using pseudo-counts [H], which estimate counts in high-dimensional state spaces using a density model. This method is adopted by RIDE and NovelD in our MiniGrid experiments and the performance still falls behind ours. Consequently, even with pseudo-counts, the training efficacy and accuracy of the estimated counts remain challenging issues. Overall, even if we solve the unique count issue, the performance of count-based bonuses still struggles in high-dimensional environments due to the sparsity and limited accuracy of counts.

W2: The maximum reward scaling seems to be an important parameter, however I do not see an ablation for it. The ablations in the appendix ablate the nature of the scaling factor, but not M

Response: Thank you for the valuable suggestions. Additional experiments can be found in the uploaded pdf. We provide an ablation study on the value of M in Figure 2 for continuous control tasks in Robosuite. M sets an upper limit on the bonus. We set M=10 as the default setting. As we can see, if M=1, the scaling factor is fixed to 1, resulting in a significant performance decline. Higher M encourages more extensive exploration. The performance with M=5 slightly lags behind the default setting. The performance with M=20 and M=40 is comparable and almost the same, indicating that the performance stabilizes as M increases. Practically, the value of M can be adjusted depending on the specific task and environment to determine the intensity of exploration. We are glad to include the detailed ablation experiment in the revision.

W3:To my knowledge there is no talk about limitations of the method. Could you provide insight on where possible limitations to your method lie?

Response: Thank you for the observation. We want clarify that we have provided the cross-reference to limitations in the paper checklist (see Appendix E). This discussion was acknowledged by reviewers 7tZv, TUTk, and tp7G. To summarize briefly, our method does not consistently achieve top performance in environments with less sparse rewards with untested performance in RL domains with very large action spaces.

Q1: In practice, how do you train the ensemble of reward models in the reward free setting? Is there any relation to successor representations? What do you do in settings where the reward is not a good indicator for the environment structure? I'm a bit surprised that the method works well given there is no reward to learn the ensemble models. Does the KL-divergence of the policies take the overhand here?

Response:Thanks for the insightful question! First, we want to emphasize that in the reward-free setting of Habitat, since there is no reward, we compare the exploration capability of these methods by evaluating the portion of explored space. Therefore, the exploration bonus takes the lead here. The performance can be analyzed from two perspectives:

• First, regarding the scaling factor, in the reward-free setting for training the ensemble models, each model’s parameters are initialized differently, and we train each model on a subset of data randomly sampled with replacement. Without the external reward, most labels of the training samples are zero during the training process, leading to larger variance between predictions during the training phase. The increased variance requires more time or a life-long learning approach for the ensemble models to converge, allowing for a larger scaling factor in such scenarios to promote more extensive exploration.

• Second, from the point of metric, referring to EME metric definition in Equation (4), when the reward is zero, the KL-divergence of the policies indeed plays a significant role. It encourages agents to take more diverse actions, compensating for the lack of a reward signal.

Overall, based on our bonus Equation (10), the EME bonus can promote more extensive exploration compared to other methods when the reward signal is sparse or not a good indicator of the environment structure. The effectiveness is also evidenced in Figure 8, which demonstrates the performance of EME in reward-free experiments. Thus, while the KL-divergence does take a more prominent role in these settings, it effectively drives the exploration process combined with diversity-enhanced scaling factor, ensuring the method remains robust and efficient. We are glad to include the discussion in the main paper to further improve the clarity.

We would like to thanks to the reviewer for their insightful feedback. Additional exps have been included in the uploaded pdf, for the references mentioned in the response, please find the reference list in the global comment and we provide our response below:

W1:(1)evaluating on more Atari games..(2)does not include a ablation..

Response: Thank you for the valuable suggestions. The results of more Atari games is in Fig 1 of PDF. We selected 10 widely used Atari games based on [C,D]. EME still achieves top performance and achieves the best performance in 8 out of 10 games. For the ablations, we have already provided results of scaling factor in Appendix C.4, as shown in Fig 8, 10, and 11. For metric comparison, we also include the performance of other metric-based methods with our scaling factor to isolate the impact of our scaling factor, denoted as LIBERTY with DSF and RIDE with DSF, and the results are shown in Fig 1 of the PDF. EME still achieves the best performance, indicating the superiority of our metric. Additionally, we provide further ablation studies on the ensemble size, maximum reward scaling M, and different RL algorithms in Fig 2, 3, and 4 of the PDF. We will include more detailed ablation results of other envs in the revision

Q1:..line99-102 is unclear..how EME overcomes the issue?

Response: We are afraid that there is some mistake in the line number 99-102. We assume that you may be unclear about the statement "breaking theoretical integrity of bisimulation metric", and our response are provided as follows. First we have clarified the statement in Propostion 1. The relaxation of Wasserstein distance $W_1$ to $W_2$ will break the theoretical guarantee of bisimulation metric when the transition dynamic model or the policy is stochastic, e.g. the fixed-point guarantee is broken under the circumstance, the metric may be not able to converge in the training process. EME overcomes the issue by eliminating the need of calculating $W_1$ distance and preserves the guaranteed fixed-point and value difference bound theoretical property, as stated in Theorem 1 and 2. The detailed proof can be found in Appendix B.

Q2:..L130..differentiate EME from such methods?

Response: Again we are afraid that there is some mistake in the line number 130. We assume that you are referring to line 91-92 that state discrepancy-based exploration bonus methods can be defined as Equation (1). These methods' bonuses can generally be defined as the product of a measure of state discrepancy and a scaling factor. What differentiates EME from such methods is twofold. First, EME uses a novel metric that more effectively and robustly captures state similarity backed by thm 1 and 2. Second, we propose a dynamic diversity-enhanced scaling factor, whereas the scaling factors of previous methods are either not scalable or are hand-crafted parameters. This combination allows EME to adaptively balance exploration and exploitation, leading to improved performance. For a detailed comparison, please refer to Table 1.

Q3:Why is Habitat a more realistic scenario?

Q4:..lacks citations..[1,2]

Response: Thanks for pointing out the citations! we will include them in the revision. We had included a detail introduction of Habitat in Appendix D.1.3. Habitat is a platform designed for embodied AI research, where agents navigate and act within photorealistic simulations of real indoor environments. It is recognized as a more realistic indoor scenario in the RL research community, as noted in [E,F].

Q5: Is the feature encoder jointly learned.. random encoder shows good results

Response: We are sorry for the confusion. The feature encoder is pre-trained to study how EME performs under different representations. When using bisimulation and inverse dynamic encoders, the exploration bonus is restricted because their representations are very compact, making state differences subtle and thus harming exploration efficacy. In contrast, as also oberseverd in experiments of [I], random embedded states introduce randomness into the calculation of the exploration bonus, sometimes resulting in higher bonuses in sparse reward environments. This can occasionally provide good results but also introduces higher variance, as shown in Fig 6 and 7.

Q6:How does number of reward models affect performance..

Response: Thank you for the valuable question. We carry the ablation study on ensemble size in Figure 3 in pdf. From the study, we observe that as the ensemble size (ES) increases, the performance of ES = 6 and ES = 9 surpasses that of ES = 3. The performance of ES = 6 and ES = 9 is comparable, with no significant difference. However, when the size is further increased to ES = 12, there is a decline in performance, particularly in the Table Wiping and Pick and Place tasks. The performance change can be analyzed from two perspectives. First, the number of reward models is related to the accuracy of the reward variance prediction used to calculate the loss function. As the ensemble size increases, the overall prediction error decreases because the models can average out individual errors more effectively. This leads to a more accurate approximation of the variance $\zeta(s_t)$. Second, with a larger ensemble size, the variance of reward predictions decreases, resulting in a lower scale of the exploration bonus. Thus, there is a balance or trade-off between estimation accuracy and exploration, explaining why the performance of ES = 12 lags behind ES = 9. The optimal number of ensemble models may vary depending on the specific tasks and environments. Regarding computational cost, it increases with the ensemble size. Therefore, we set ES = 6 as our default setting, where the performance is nearly the same as ES = 9 but with lower computational overhead. It is also noteworthy that even with an ensemble size of 3, EME still outperforms the best baseline methods, further demonstrating the robustness of EME.

We gratefully thank the reviewer for recognizing our contributions! The additional ablation experiments have been included in the uploaded pdf, we provide our response below:

W1:The performance of EME with feature encoder is mixed; this may hamper the scalability of EME to realistic scenarios, where pre-trained feature encoder is commonly used

Response: Thanks for the valuable question. Yes, we agree that the performance of EME can be influenced by the choice of pre-trained feature encoders, which is also a common issue for existing state difference-based exploration approaches like RIDE where the bonus heavily relies on the design of encoders. However, as demonstrated in Figures 6 and 7, the impact on EME's performance is not significant and can sometimes be positive. Even when different pre-trained encoders are used, EME consistently outperforms other baseline methods, which suggests that despite the inherent variability introduced by different feature encoders, the performance of EME remains at a relatively high level, providing stronger results compared to other methods.

W2:Ablations are limited. It would be good to know how sensitive EME is to the choice of RL algorithm, the value of M, and other hyperparameters. Only scaling factor is studied right now

Response: Thank you for the valuable suggestions. Additional experiments can be found in the uploaded pdf. We conducted an ablation study on the value of M, shown in Figure 2, for continuous control tasks in Robosuite. M sets an upper limit on the bonus. We set M=10 as the default setting. As we can see, if M=1, the scaling factor is fixed to 1, resulting in a significant performance decline. Higher M encourages more extensive exploration. The performance with M=5 slightly lags behind the default setting. The performance with M=20 and M=40 is comparable and almost the same, indicating that the performance stabilizes as M increases. We also provide the performance of EME with different RL algorithms. We chose widely used benchmark algorithms for continuous control tasks: PPO, A2C, SAC, and DDPG. As shown in Figure 4, EME acts as a plug-in algorithm and demonstrates considerable improvement across all baseline methods. The performance of EME with different RL algorithms is almost at the same level, indicating EME's wide adaptability. Additionally, we conducted an ablation study on ensemble size, presented in Figure 3, and included the performance of other metric-based methods integrated with our proposed scaling factor in Figure 1. These studies show that EME is robust to different parameter choices and consistently demonstrates superior performance compared to other baselines. Due to limited time and space, we will include the detailed ablation experiment of other environments in the revision.

W3:On the Robosuite environments, EME do not seem much better than the baselines. How do we characterize when we expect EME to do much better than existing methods?

Response: Thanks for the insightful question. We have discussed the point of EME's performance lead in the limitations section in Appendix E. Specifically, we expect EME to outperform existing methods more significantly in environments with sparse rewards and challenging exploration tasks requiring extensive exploration and diverse actions. Examples include navigation tasks in Habitat, hard exploration tasks in Atari and the gym-maze environment. In these settings, where external rewards from the environment are very sparse or zero during the learning process, the advantage of EME's bonus becomes more pronounced compared to other bonuses due to the superiority of our proposed metric and the diversity-enhanced scaling factor.

We appreciate the reviewer for valuable suggestions, the requested exp have been included in the uploaded pdf, for all the references mentioned in the response, please find the reference list in the global comment, and we provide our responseas follows.

W1:The ultimate method seems incredibly complex..moving Algorithm 1 to main paper could help..

Q1:..more details on the computational complexity and runtime performance..how does EME scale with size of the state space..

Response: Thank you for your valuable suggestions. We would like to move Algorithm 1 to the main text in the revision. Regarding the runtime performance, we direct you to Fig 2 and 3 in the paper, which demonstrate that EME achieves higher returns in the same timesteps and faster convergence compared to other baselines, indicating reduced running time during training. In terms of computational complexity comparison, RIDE utilize calculating state difference in representation space (see Table 1), which is learned using both a forward and an inverse dynamics model. While for the loss function of LIBERTY, please refer to Eq (3), we can see that the calculation of LIBERTY requires the approximation of Wasserstein distance of transition distributions and the difference of inverse dynamic output, which requires additional training of transition model and inverse dynamic model. In contrast, as shown in Equation (9), EME simplifies the process by only training additional reward models to calculate the bonus. So our method is more computationally friendly and we require less training procedures comparing to RIDE and LIBERTY.

The performance of EME is scalable because EME provides a more robust and effective metric to evaluate state difference within different environments, and we provide analysis in line 200-212 and 218-226. The scalabity of EME is also evidenced in the experiments where EME outperforms other methods in different environments with different state size. Specifically, when EME scales with environments with different state size or complexity, we only need to adjust the size of network accordingly to fit in the state size for calculation of bonus. Because the objective of metric learning only considers the sum of reward differences, the distance between sampled subsequent states, and the KL divergence between policy distributions, which means that the size of states has minimal impact on the metric learning.

Q2:Could you provide ablation studies to isolate the components of EME?

Response: Yes, we had already provided the ablation study on scaling factor in Appendix C.4 (line 667-676), where we compared EME with two variants: EME-EP, which incorporates episodic counts, and EME-Static, which uses a static scaling factor. The experimental results are shown in Figure 8, 10 and 11. For metric comparison, we also include the performance of other metric-based methods with diversity-enhanced scaling factor to isolate the impact of our scaling factor, the result is shown in Figure 1 in uploaded pdf, where EME beats other metric-based methods equipped with diversity-enhanced scaling factor, indicating the superiority of our metric comparing to bisimulation metric and $L_p$ norms. Also we provide additional ablation study on the ensemble size, max reward scaling M in Figure 2, 3 and 4 of uploaded pdf. Due to limited space and time, we will include the ablation results with other environments in the revised version.

Q3:..Could you provide more intuitive explanations..on the theory and its practical implications?

Response: Thank you for your insightful question. Proposition 1 and 2 illustrate the impact of the approximation gap on the bisimulation metric-based method. Fundamentally, the distance derived from the inverse dynamic bisimulation metric could break the fixed-point guarantee, resulting in a less stringent bound on the value difference. This suggests that the metric may not be able to converge and not be closely relate to the value function. Consequently, the learned metric could diverge and fails to precisely reflect state similarity within the bisimulation metric space, leading to a decrease in performance.

Theorem 1 provides a convergence guarantee, stating that our learned metric will converge to a fixed point during training, which ensures that the metric becomes stable and reliable as training progresses. Theorem 2 explains that the EME distance between states serves as an upper bound on the value difference between those states. Practically, this means the agent is encouraged to visit states with both higher state differences and higher value differences. Visiting states with higher value differences enhances the value diversity between collected transition samples, leading to a more diverse policy during training. Furthermore, an increase in bonus reward leads to larger value differences and thus larger TD errors (see lines 218-226), incentivizing the agent to prioritize transitions with large TD errors, so the training efficiency will be improved.

W2:..a real-life indoor environment..is rather misleading..

Response:Thank you for pointing out the potential for misunderstanding. The detailed description of Habitat is included in Appendix D.1.3. To clarify, we use the Habitat platform [A], which is designed for embodied AI research and provides an interface for agents to navigate and act within photorealistic simulations of real indoor environments. Specifically, we utilize the MP3D dataset[B], which includes high-quality renditions of 1,000 different indoor spaces. This platform offers a practical application for RL in embodied AI, providing a highly realistic and diverse set of environments for testing and development. So we call it a real-life indoor environment. To avoid confusion, we will change the term to "a photorealistic simulator of indoor environments" in our revised manuscript.