MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization

We thank the reviewer for their feedback and for acknowledging the significance of our work, the empirical evaluation, and our intuitive methodology. Below we address the concerns raised by the reviewer.

Weaknesses

Clarity on the Importance of Information Gain: As we discuss in Section 2.4, information gain is commonly used for exploration for experiment design in bandits and RL literature. In equation (8) in the paper, we show that our approach also results in maximizing entropy in both the state and action space as opposed to just the action space which is what standard Boltzmann exploration does. In Figure 4, we empirically show this on the Pendulum, where we plot the states visited during exploration by MaxInfoRL and compare it to SAC. Lastly, in Figures 12 and 15, we combine our method with different intrinsic rewards and show that generally the information gain/model uncertainty performs better. However, we are happy to consider additional illustrations that the reviewer thinks might add more to the clarity of the reader.

Limited Novelty in the Trade-off Mechanism: We agree that our trade-off mechanism builds on existing strategies in the RL literature. However, we think this is a strength of our work and part of the reason why our approach works so robustly across different benchmarks and tasks. In particular, these auto-tuning mechanisms are widely applied for the action entropy (SAC, DrQ, REDQ, DrO etc., all use this mechanism and have achieved strong empirical performance). More importantly, they are simple to integrate into an RL algorithm, which in itself is an important attribute for RL practitioners. Therefore, we specifically catered our approach to benefit from these existing mechanisms and thus make it flexible, robust, and also easily accessible for RL researchers. To the best of our knowledge, we are the first to establish this link between intrinsic exploration and the trade-off mechanisms discussed above.

We once again thank the reviewer for their active engagement in the review process and hope our responses enhance the confidence of the reviewer about our work and would be happy if they prompt the reviewer to improve our score

We thank the reviewer for their feedback and for acknowledging our thorough empirical evaluation. Below we provide a detailed response to the questions raised by the reviewer.

Weaknesses

Regret bound: We provide an upper bound for the regret and we think comparing a loose lower bound with the upper bound is not informative since the two do not match. To the best of our knowledge, the regret bound even for UCB or TS approaches with GPs, i.e., continuous input and output spaces, scales with $\beta_T \sqrt{\gamma_T T}$ and [1] also provide a tighter lower bound that scales similar to the upper one, i.e., is not $log(T)$. Moreover, it is widely believed that the minimax optimal rate is polynomial (see https://proceedings.mlr.press/v178/open-problem-vakili22a/open-problem-vakili22a.pdf and references therein). However, we acknowledge that our upper bound is weaker than TS and UCB methods. This is not surprising as $\epsilon$--greedy generally has a weaker regret bound than UCB-like algorithms such as UCRL [2, 8, 9]. Yet, these methods are widely applied in deep RL due to their simplicity and scalability.

Regret analysis in the bandit setting: We agree that it is not straightforward to claim sublinear regret in the MDP setting with analysis for the bandit case. However, our intention for the bandit analysis is not to make this claim. Rather we used the bandit analysis as inspiration for our deep RL algorithm. This is partially motivated by looking at other works such as [2]. To further clarify this, we changed the line 210 “In Appendix A, we provide a theoretical justification for our approach and study $\epsilon$–MAXINFORL in the simplified setting of multi-armed bandit (MAB).” to “In Section A, to give a theoretical intuition of our approach, we study $\epsilon$-MAXINFORL in the simplified setting of multi-armed bandit (MAB).” In addition, while we think deriving a regret bound for our algorithm is an interesting direction for future work, we focus the scope of this paper more on the wide applicability of our approach, which is why we evaluate it across several standard deep RL benchmarks on both state-based and visual control tasks, as also acknowledged by the reviewer.

Choice of baselines: The main objective of our method is to boost existing RL algorithms by combining intrinsic exploration and extrinsic rewards in a principled manner. Our work allows us to extend several RL algorithms, such as SAC, DrQ, DrQv2, benefiting from their simplicity while also having principled exploration from intrinsic reward which are widely studied in the RL literature. Therefore, besides considering the naive exploration algorithms such as SAC, we use intrinsic exploration baselines such as curiosity and disagreement for our comparison. In addition, we also compare our algorithm with [3, 4] which also try to combine extrinsic and intrinsic rewards. To the best of our knowledge, these are state-of-the-art approaches for combining intrinsic and extrinsic exploration. In addition, we also compare our method to REDQ (see Figure 9) which is one of the most competitive baselines in the paper [5] proposed by the reviewer. However, following the reviewer’s feedback we have also added experiments with the algorithm OAC [6] and our MaxInfoRL version of OAC to the paper (see Figure 17). We chose OAC as we think it fits our set of baselines the best since it is also a soft-actor critic algorithm that is based on principled exploration via optimism. Furthermore, we could find an official open-source implementation of the algorithm as a reference.

For our visual control tasks, we compare with DrQv2 and now DrM (Figure 18) which are SOTA model-free RL algorithms known to solve the challenging problem of humanoid control from visual observations [7]. We show that our method outperforms it. Given our collection of baselines, and now OAC in addition, we believe our empirical evaluation makes a strong case for our method, as also acknowledged by the other reviewers.

Related Works: Thanks for listing the related work. We will include them in the revised/final version of our paper. The provided references are interesting and relevant for exploration in RL. However, as we mentioned in the previous point the goal of our work is to combine extrinsic and intrinsic rewards in a principled manner, which results in a simple, scalable and flexible algorithm, which is what we also show in our experiments where we thoroughly evaluate our approach on state-based and visual-control benchmarks while also combining it with several model-free RL base algorithms such as SAC, REDQ, DrQv2 etc.

I think [6] is a great example to rebut the authors' points.

The authors of [6] implemented BEN from scratch (the original paper is for discrete domains, this work extends it to continuous). There is at least one BootPriorDQN+ implementation on Github that I found with a quick Google https://github.com/johannah/bootstrap_dqn. From past experience, it's very easy to reach out to authors to get implementations as most are willing to share them.

Figure 4 from [6] shows Bayesian methods can solve incredibly challenging high dimensional domains. Most impressive is sparse humanoid. I've never seen another approach do this before. Moreover, Verse Sparse Ant can be solved by several methods. The only sparse environment I can find in the authors' paper is Cartpole, which from experience is relatively simple.

Most Bayesian methods can be added on top of existing algorithms as they just affect exploration. I have added Bayesian exploration to many approaches, including PPO, DQN, RedQ etc.

Thanks for your feedback on our work and also for acknowledging our algorithmic insights and experimental evaluation.

Weaknesses

Convergence result: We suppose the reviewer is referring to the convergence result in Theorem 3.1. This result is inspired by the SAC paper [1], where they also show that the modified critic and policy updates give convergence. This is a crucial result as in our setting it means that our proposed policy and critic update rules give convergence. However, as also in the SAC paper, it does not take the learning aspect into account and is a consistency result for the Bellman updates.

Baselines: As highlighted by the reviewer, the main objective of our method is to combine intrinsic and extrinsic exploration in a principled manner. To this end, we use intrinsic exploration baselines such as curiosity and disagreement for our comparison. In addition, we also compare our algorithm with [2, 3] which also try to combine extrinsic and intrinsic rewards. Furthermore, we also consider widely applied naive exploration baselines such SAC and REDQ in our experiments. Across all our experiments, we show that our approach outperforms the aforementioned methods. In addition, we show that our method also scales to complex high dimensional settings from visual control tasks, where we also compare it with SOTA (model-free) baselines: DrQ and DrQv2. All our experiments are evaluated on well-established deep RL benchmarks, including the recent HumanoidBench benchmark [4]. However, following the feedback from the reviewer, we have added additional experiments, where we compare our approach with DrM [5] on the challenging humanoid and dog tasks from the deepmind benchmark suite (Figure 18). Furthermore, we have also added another experiment comparing OAC [6] (a baseline suggested by reviewer 8tDw) to the paper (Figure 17). We hope through this we can convince the reviewer of our empirical evaluation.

Regret bound in the tabular setting: We agree that it would be interesting to derive a regret bound for the algorithm, even in the simplified tabular setting. In fact, this is a direction we are actively pursuing for future work, where we aim to theoretically study the algorithm further. We think results from works such as [7] are a positive sign and we are hopeful of obtaining a regret bound. However, the focus of this work was mostly on the deep RL algorithm and showing the diversity and scalability of the method, therefore we decided to keep the scope of our paper focused more on those aspects.

We hope our responses adequately addressed the reviewer’s concerns and would be happy if they prompt the reviewer to improve our score. Thanks once again for your insightful feedback on our work.

References:

1. Haarnoja, Tuomas, et al. "Soft actor-critic algorithms and applications." arXiv preprint arXiv:1812.05905 (2018).

2. Chen, Eric, et al. "Redeeming intrinsic rewards via constrained optimization." Advances in Neural Information Processing Systems 35 (2022)

3. Burda, Yuri, et al. "Exploration by random network distillation." arXiv preprint arXiv:1810.12894 (2018).

4. Sferrazza, Carmelo, et al. "Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation." arXiv preprint arXiv:2403.10506 (2024).

5. Xu, Guowei, et al. "Drm: Mastering visual reinforcement learning through dormant ratio minimization." ICLR (2024).

6. Ciosek, Kamil, et al. "Better exploration with optimistic actor critic." Advances in Neural Information Processing Systems 32 (2019).

7. Sukhija, Bhavya, et al. "Optimistic active exploration of dynamical systems." Advances in Neural Information Processing Systems 36 (2023)

We thank the reviewer for their feedback. Particularly, we are happy that the reviewer acknowledged the simplicity of our approach and the strong empirical performance. This is a central strength we wanted to communicate with our work: well-established and simple findings from naive exploration methods can be easily integrated with intrinsic exploration objectives to achieve state-of-the-art performance. Below we address the points raised by the reviewer.

Weaknesses

Originality – paper combines ingredients of existing approaches: While we understand the concern raised by the reviewer, we think this is in fact a strength of our approach. Namely, Boltzmann exploration, soft-Q learning, and automatic coefficient tuning are well-studied and widely applied in the RL community. Similarly, several works that we discuss in the paper use information gain/intrinsic rewards for unsupervised exploration in RL. However, to the best of our knowledge, we are the first to establish this link between the two, which allows us to easily combine extrinsic rewards and intrinsic exploration objectives, thus proposing algorithms that benefit from the simplicity of naive exploration schemes but enjoy principled exploration from intrinsic rewards. Compared to other works such as [1, 2] that combine extrinsic and intrinsic rewards our approach performs much better while also being easy to implement, flexible (can be combined with other RL methods as we show), and scalable. We thoroughly evaluate our approach in the experiments as also acknowledged by the reviewer.

Connection between bandits and RL: Thanks for your feedback. In section A of the appendix, we tried to establish this connection and also highlighted other works such as [10–13] which leverage ideas/algorithms from multi-arm bandits for RL. Particularly, [11-13] leverage and test the algorithms for RL on real-world robotic platforms. Similar to [10], the MAB setting serves as an inspiration for our algorithm and the proposed approximations for the general deep RL case. We are happy to receive suggestions from the reviewer on how to further clarify this connection. Furthermore, we also want to make it clear that while we do the analysis for the MAB case, the analysis for the deep RL method would be much more challenging and we believe also out of scope for this work.

Weaknesses/Limitations/Computational cost: We agree that learning an ensemble of NNs has additional computational costs. We highlight this in the conclusion of the paper. However, to further underline this, we have added the following table comparing the computational cost of SAC and MaxInfoSAC to the paper (see Table 1 in Appendix E).

Time to train for 100k environment steps on NVIDIA GeForce RTX 2080 Ti: SAC (SB3): 16.96 min +/- 1.64317 min, MaxInfoSAC (SB3): 39 min +/- 1 min, SAC (JAX): 5.6 min +/- 0.2min, MaxInfoSAC (JAX): 7.3 min +/- 0.75

As the number suggests MaxInfoSac requires more time to train due to the additional computational overhead. However, our JAX implementation already bridges the computational gap and is much faster. To further reduce the computational cost, we can consider cheaper methods for uncertainty quantification such as [9] or different/cheaper methods to evaluate intrinsic rewards, e.g., [2]. We have added this discussion to the paper.

Use of upper (Gaussian) bound for the information gain: We have adapted the paper (see Section 2.4) and put more emphasis on the fact that we approximate the information gain with this upper bound. We would also like to highlight that this approximation is commonly used by several RL algorithms [3 - 6] and they show that it works well in practice. Effectively, the objective encourages exploration in areas where we have less data/high uncertainty. Particularly, [6] gives a theoretical justification for this choice of the upper bound for the model-based (active learning) setting. We will highlight this further in the paper.

Trade-off between information gain and reward: We agree that this is generally a nontrivial trade-off due to the different scales of the objectives. However, Boltzmann exploration methods such as SAC also result in a similar trade-off and the auto-tuning approach used in these methods achieves strong performance and is widely applied. To this end, as highlighted by the reviewer, we leverage ideas from them in MaxInfoRL. Furthermore, note that we take a similar approach as in [2] of normalizing the intrinsic rewards during training, which also helps with the scaling (see section E in the appendix).

Questions

Importance of naive exploration: Thanks, this is a very interesting question. One reason we believe having a stochastic policy helps empirically is that it introduces diversity to the collected data if we repeat the rollouts between different episodes or across several parallel environments. Moreover, usually, actor-critics are not aggressively updated in RL to maintain stability during training. Thus, without a stochastic policy, we might have the risk of collecting very similar samples from one episode to another. Furthermore, we believe it also introduces additional stochasticity during training which could also be beneficial. Following the reviewer’s suggestion, we have adapted Figure 11 in the paper. In Figure 11, we compare a version of Maxinforl (with DrQv2) which uses a deterministic policy with the standard version (stochastic policy). We run this comparison on the challenging, high-dimensional, pixel-based exploration task: Humanoid-walk (also noted by [7] as being a hard exploration task). The figure shows that Maxinforl without noise performs better than DrQv2 with noise. However, including the noise during exploration has an additional significant gain in performance.

Clarification on $\epsilon$–greedy: Thanks for this insightful question. We think the strategy would still make sense since the $\epsilon$ parameter is gradually decreased from 1.0 to 0.0. Therefore, during the initial stages of training the intrinsic exploration policy is sampled much more frequently effectively collecting trajectories that focus much more on intrinsic exploration. Towards the end of training, we focus more on the extrinsic reward. A similar strategy of greedy in the limit with infinite exploration (GLIE) is also used in [8]. However, we agree that perhaps a better approach would be to switch between the exploration-exploitation policies after several steps or an episode. Empirically, this is what did for our experiments in Figures 14-15, where we switched between the two strategies every 32 steps. However, to illustrate that a switch at every step also could work, we have added an additional experiment (see Figure 16) which switches between the two policies at every step. We compare this to the one where we have a switching frequency of 32 steps and show that while both approaches work, the 32-step frequency performs better.

Tightness of the constraint: What we meant with tight is that asymptotically when the policy converges, the target policy would give similar information gain as the true policy and thus the constraint will be satisfied. However, we see how this comment can be misleading and will remove it from the paper to avoid confusion.

Code: Yes, we will make the submitted code public.

Minor comments and suggestions: Thanks a lot for pointing these out. We have already started addressing them and will adapt the paper according to the proposed suggestions. To have an interactive rebuttal period, we have already added some of the changes discussed above to the manuscript.

We thank the reviewer once again for their feedback and hope our responses adequately address the raised concerns. We’d be happy if the reviewer would consider revising our score and are happy to answer any other open questions. Thanks again for your feedback and active engagement in the review process.

References

1. Chen, Eric, et al. "Redeeming intrinsic rewards via constrained optimization." NeurIPS (2022)

2. Burda, Yuri, et al. "Exploration by random network distillation." arXiv preprint arXiv:1810.12894 (2018).

3. Sekar, Ramanan, et al. "Planning to explore via self-supervised world models." ICML, 2020.

4. Mendonca, Russell, et al. "Discovering and achieving goals via world models." NeurIPS (2021): 24379-24391.

5. Sancaktar, Cansu, Sebastian Blaes, and Georg Martius. "Curious exploration via structured world models yields zero-shot object manipulation." NeurIPS (2022)

6. Sukhija, Bhavya, et al. "Optimistic active exploration of dynamical systems." NeurIPS (2023)

7. Yarats, Denis, et al. "Mastering visual continuous control: Improved data-augmented reinforcement learning." ICLR (2022)

8. Parisi, Simone, Alireza Kazemipour, and Michael Bowling. "Beyond Optimism: Exploration With Partially Observable Rewards.", NeurIPS (2024)

9. Osband, Ian, et al. "Epistemic neural networks." NeurIPS (2023)

10. Cesa-Bianchi, Nicolò, et al. "Boltzmann exploration done right." NeurIPS (2017).

11. Calandra, Roberto, et al. "Bayesian optimization for learning gaits under uncertainty: An experimental comparison on a dynamic bipedal walker." Annals of Mathematics and Artificial Intelligence 76 (2016).

12. Berkenkamp, Felix, Andreas Krause, and Angela P. Schoellig. "Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics." Machine Learning 112.10 (2023)

13. Widmer, Daniel, et al. "Tuning legged locomotion controllers via safe bayesian optimization." CORL, 2023

We thank the reviewer for their feedback and active engagement in the rebuttal period. We are glad to see that our rebuttal has addressed most of the reviewer's concerns. Below is our response to the remaining points.

Originality: As highlighted in our initial response, we think our method precisely proposes a new simple idea that connects two existing RL approaches and results in a simple, flexible, and scalable class of algorithms. Therefore, we still think our approach itself for combining and auto-tuning of intrinsic and extrinsic rewards is original and we have also not seen prior work taking a similar approach as ours.

Epsilon–greedy: Thanks for your feedback. We would like to clarify this point further. In particular, the epsilon–greedy extension we propose is very flexible. Effectively, we can decide on a schedule for epsilon and based on the schedule, the agent will alternate between (intrinsic) exploration and exploitation actions. Therefore, the case where the agent switches after multiple steps is also part of our proposed formulation. In the extreme case, we can use a schedule for epsilon, which at the beginning of each episode decides to either explore or exploit and then maintains this strategy for the whole episode, i.e., $\epsilon = 1$ for the remainder of the episode. We will add this explanation as a further clarification to the paper. Furthermore, we would like to highlight that the comparison we made between the two different schedules for epsilon is purely empirical and we cannot draw any theoretical conclusions from it. Moreover, we acknowledge that regret analysis for the bandit setting does not transfer directly to the RL case. However, given results from prior works [6, 8], we have confidence that for an appropriate schedule, our epsilon–greedy strategy could also result in sublinear regret for the RL case. We leave this extension for future work and will also further clarify in the paper that our analysis is for the bandit setting and extending it to the RL case is an interesting direction for future work.

Code: We agree that providing code to reproduce our results is important and we want our code to be easily accessible for all RL researchers and practitioners. Therefore, we plan to release all our code and instructions to ensure good reproducibility. We are working on it already and we will have the release ready together with the camera-ready version.

We hope our response addresses the reviewer’s remaining concerns and given that we could address all the other concerns raised in the review, we would appreciate it if the reviewer would consider increasing our score and are happy to answer any other open questions. Thanks again for your active engagement.

Dear Reviewer,

Thanks for your response. We are glad we could address the remaining open concerns. Since you do not have any further questions, we would be very happy if you would consider increasing our score.

Thanks again for your active engagement in the rebuttal period.