Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling

Thank you very much for taking the time to review our paper. We appreciate your comments and suggestions to improve our evaluation. Please find our responses below.

Weaknesses

1. The experiments are insufficient. For environments with sparse rewards and difficult exploration, tasks like Maze and Navigation, such as Sparse-Point-Robot, might better illustrate the effectiveness of the approach.

Response: Thank you for the recommendation to consider Sparse Point Maze environments. We have taken your suggestion to evaluate HOT-GP and baseline algorithms on two additional Sparse Point Maze environments: U Maze and Medium Maze. Currently, we have finished gathering the results on U Maze and are still running experiments on the Medium Maze. Our results on the U Maze are consistent with our findings on the Sparse Reacher task and demonstrate the superior sample efficiency of HOT-GP in sparse reward settings. Furthermore, these results highlight the stability of our approach against MBPO and optimistic MBPO.

Additionally, we have strengthened our experiments by increasing the number of seeds to 10 and evaluated on the Sparse Reacher task with an intermediate action penalty of 0.3. We have uploaded a new paper pdf with all these new results except for the Medium Maze, which will be included when the experiments finish.

2. In some environments, the training curves presented in the paper do not appear to have converged, leaving the convergence of the algorithm unclear and making it difficult to judge whether an optimal policy has been achieved.

Response: While it is true that some training curves do not converge within the maximum number of environment steps, this is secondary to the main motivation of our evaluation. Our primary objective is to evaluate the sample efficiency of HOT-GP relative to existing approaches by comparing algorithms’ performance after a fixed number of environment interactions, as done in H-UCRL (and many other papers). The question of whether algorithms converge within the set number of steps is less relevant and involves balancing computational costs across multiple tasks while ensuring useful results. We have chosen the maximum number of environment steps that allows most algorithms to converge whenever possible, given our computational limitations. It is important to note that, in all experiments apart from the Sparse Reacher, at least one HOT-GP variant (of the three considered $r_{\text{min}}$ schedules) achieves the SAC-best performance after training for a fraction of the environment steps.

3. The paper seems to lack ablation studies on different modules and tasks, which fails to adequately demonstrate the independent contributions of various components to overall performance. Without detailed ablation studies, it becomes challenging to evaluate the individual effects of joint uncertainty modeling and the optimistic strategy across different tasks.

Response: We would like to point out that our paper includes an ablation study in Figure 3 (the old Figure 2) and is discussed in Section 6.2. In this study, we ablate over values for the $r_{\text{min}}$ threshold ($r_{\text{min}}$ ranges from 01 to 0.3, 0.5, and 0.7) and the number of steps K in a model rollout (K = 1, 5, T). We also consider HOT-GP with component-wise independent predictions (no coregionalization) and HOT-GP with sampling the next state (Thompson sampling) rather than selecting the expected next state. Our ablation study provides insights into the importance of each element of our proposed approach. Please let us know if there are specific ablations that you believe are missing.

Questions

1. According to the paper, the HOT-GP algorithm is inspired by H-UCRL. Why, then, was H-UCRL not used as a comparative algorithm in the MuJoCo tasks?

Response: This is a good question. While we are inspired by the notion of optimistic exploration that is central to H-UCRL, our work considers a different setting where the reward function is unknown. Specifically, H-UCRL assumes access to the ground truth reward function, making it not directly comparable with HOT-GP. One way to circumvent this problem is to estimate the reward function and perform H-UCRL using the learned reward as a surrogate for the ground truth. We tested this approximation of H-UCRL on the MuJoCo Half-Cheetah task and observed subpar performance relative to HOT-GP and MBPO, as shown in Figure 5 in Appendix D. Since we do not have access to the true reward function during training, it remains unclear whether the poor performance is due to the algorithm itself or errors in reward estimation. This uncertainty complicates a fair comparison to H-UCRL, so we opted not to do so.

Theoretically, Thompson sampling has a much lower sampling efficiency than Bayes-optimal. The policies learned within the BAMDP (Bayes-adaptive Markov decision processes) framework are closer to the optimal exploration strategy. However, the methods compared by the authors did not include approaches based on the Bayes-optimal policy. Additionally, the authors' approach is not the first to 'reason about joint uncertainty over transitions and reward.' Please refer to [1], which uses VAE-based methods to model the transition and reward models, allowing for better learning of the agent's uncertainty about the environment. This method has already been shown to learn policies closer to the optimal compared to posterior sampling(Thompson sampling)-based methods [2]. I believe this paper only modifies H-UCRL and compares it with some basic algorithms, with limited contributions. Thank you to the authors for the experiments, which have made the paper more thorough. However, I will maintain my score.

[1]Zintgraf et al, Varibad: A very good method for bayes-adaptive deep rl via meta-learning. ICLR, 2020.

[2]Rakelly et al. Efficient off-policymeta-reinforcement learning via probabilistic context variables. ICML, 2019.

We believe there may also be a misunderstanding of our approach and contributions. HOT-GP is not a modification of H-UCRL. HOT-GP is based on Thompson sampling rather than Upper Confidence Bounds and operates in a fundamentally different manner. HOT-GP learns both the dynamics and reward functions with joint uncertainty while H-UCRL only learns the dynamics model and assumes access to the ground truth reward function. Learning the dynamics and reward functions allows us to reason about the joint uncertainty over possible outcomes and generate plausible transitions conditioned on optimistic rewards. In contrast, H-UCRL learns an uncertainty-aware model of the dynamics that captures the mean $\mu(s,a)$ and standard deviation $\Sigma(s,a)$ for each input $(s,a)$. They then select the next state $s’ = \tilde{f}(s,a) = \mu(s,a) + \beta\Sigma(s,a)\eta$ using $\beta$ as a fixed hyperparameter and hallucinated $\eta$ value that leads to the highest transition reward. This can result in infeasible optimism if $\mu(s,a) + \beta\Sigma(s,a)\eta$ is not reachable from the current state $s$. We do take high level inspiration from the motivation behind H-UCRL to direct exploration in an optimistic manner with respect to the single-step reward value. We also compare HOT-GP with a superset of the baselines in the H-UCRL paper, namely Thompson sampling and greedy exploitation, in addition to MBPO, an optimistic variant of MBPO, and model-free algorithms. We choose these comparison methods to illustrate important aspects of our approach. Specifically:

• Greedy exploitation maximizes the expected reward and ignores uncertainty induced by the reward-dynamics model. We hypothesized this would lead to poorer performance than HOT-GP, demonstrating the importance of uncertainty-aware policy optimization.
• Thompson sampling optimizes the policy with respect to a sampled optimistic reward-dynamics model. We expected similar performance to HOT-GP, with slightly less stability, showing the benefit of optimism only over the rewards.
• MBPO uses an ensemble of reward-dynamics models to sample states and rewards but does not account for joint uncertainty. We anticipated poorer performance than HOT-GP, illustrating the importance of optimism and joint uncertainty for consistent hallucinations.
• Optimistic MBPO is the same as MBPO but incorporates the optimistic reward sampling of HOT-GP. Similarly, we expected better performance than MBPO but poorer performance than HOT-GP due to the lack of joint uncertainty for consistent state-reward hallucinations.
• Model-free algorithms serve as a baseline to contrast model-free and model-based approaches. We hypothesized these results would be less sample efficient than the model-based approaches.

The primary contributions of our work are twofold: 1) the HOT-GP algorithm and 2) establishing the importance of joint uncertainty modeling for optimistic exploration. Our evaluations reveal that modeling and leveraging joint uncertainty over the state and reward distributions is crucial for effective optimistic exploration, particularly in challenging settings with sparse rewards, action penalties, and difficult-to-explore regions. Therefore, we anticipate that the HOT-GP algorithm and broader technique of optimism with joint uncertainty will be valuable tools for advancing exploration strategies and improving sample efficiency–core challenges at the heart of reinforcement learning.

We hope we have addressed your most recent concerns as well as those raised in your original review. To summarize, these are the actions we have taken in response to the issues outlined in your initial feedback:

• Updated the paper with completed results on two Sparse Point Mazes (Figure 2) as suggested
• Ran additional experiments to show asymptotic performance on the Sparse Reacher (Figure 6), asymptotic performance was already clear on the other tasks
• Highlighted our original ablation study results (Figure 3)
• Provided results comparing HOT-GP with an appropriate approximation of H-UCRL (Figure 8) along with a justification for not conducting further comparisons with H-UCRL

We would be grateful if you would consider increasing your score in light of these responses. If you have any further questions or comments, please do not hesitate to ask. Thank you!

[1] Rakelly et al. Efficient off policy meta reinforcement learning via probabilistic context variables. ICML, 2019.
[2] Zintgraf et al. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. ICLR, 2020.
[3] Moradipari et al. Improved bayesian regret bounds for thompson sampling in reinforcement learning. NeurIPS, 2023.
[4] Osband et al. Why is posterior sampling better than optimism in reinforcement learning?. ICML, 2017.
[5] Osband et al. More efficient reinforcement learning via posterior sampling. NeurIPS, 2013.
[6] Bubeck and Sellke. First-order bayesian regret analysis of thompson sampling. Algorithmic Learning Theory, 2020.

We wanted to follow up on our last response as the end of the discussion period is approaching. We believe we have addressed your feedback by (1) providing the additional experiments you suggested and showing asymptotic behavior; (2) highlighting our original ablation study; (3) explaining the relationship between our work and the referenced papers; and (4) clarifying the novelty of our approach and its relationship to H-UCRL. In light of these responses, if you find our work more convincing, we would greatly appreciate it if you would consider updating your score. Also, we would like to check if there are any remaining adjustments to the paper you feel are necessary to resolve your concerns. Thank you!

Thank you very much for taking the time to review our paper. We are so pleased to hear that you enjoyed our work and appreciate the practical recommendations. Please find our responses below.

Weaknesses

1. The evaluation would benefit from more complex scenarios, such as object manipulation or long-horizon tasks, which could further demonstrate the method’s robustness and adaptability.

Response: Thank you for this suggestion. To address this, we are running additional experiments on two Sparse Point Maze environments where the agent must navigate through a maze to a goal location. The sparse reward structure makes these long horizon tasks and our results so far validate the sample efficiency of our approach. We have uploaded a new paper pdf including the new results on one maze and will update the paper pdf again with results on the second maze once our experiments are complete.

2. While the integration of a neural network to learn the GP mean is discussed in the main text, I feel like a small paragraph or additional description in the appendix is required. I understand the general approach and can infer the authors' method, but a concise explanation would be a helpful addition for readers.

Response: Thank you for highlighting this. We agree that a more detailed explanation of how the neural network is integrated to learn the GP mean would improve clarity. To address this, we added a paragraph on this, entitled Gaussian Process with Neural Network Mean Function, to the paper in Appendix B.

Thank you very much for taking the time to review our paper. We appreciate your comments and insightful questions. Please find our responses below.

Main Weaknesses

1. I am not convinced the reward-uncertainty is significantly more useful than the dynamics uncertainty. As the authors describe, the de-facto standard way of formulating GPs is that output dimensions are considered independent predictions. The uncertainty of each dimension (the variance of the GP) of a GP conditioned on $X$ in a query-point $x$ is then given by $K(x,x) - K(x,X)K^-1(X,X)K(X,x)$ and depends only on the kernel choice and the data distribution. Unless one has specific knowledge about an appropriate reward-kernel, I have no intuition why the reward offers a better (or different) information gain signal than the dynamics.

Response: We believe there is some misunderstanding regarding how the GP model is set up. We will try to clarify our model here and have updated the paper to make sure it is better explained. We use a multi-output GP so that our model describes the full covariance structure of the state and the reward as a function of the state and action, i.e. the output dimensions are not conditionally independent given the input. Due to the often high-dimensional state space, we use the linear model of coregionalization to efficiently do computation on this object. As we describe in the paper (lines 270-281), modeling the dependency between predicted state and reward is essential to our approach as this allows us to formulate a distribution that predicts future state based on desired reward. It is correct that we need to make assumptions about the specific relationship between the input (state and action) and the output (next state and reward) for this to be feasible. We use a Matern 5/2 covariance function that encodes a smoothness prior while the coefficients in the linear model of coregionalization are completely free. This means that, if the model is not able to pick up a correlation signal between the output dimensions (reward and state), then its values could be set to zero, implying that the dimensions are independent. In all our experiments, there is a significant correlation between the dimensions, meaning that the reward is informative of the state and vice versa. We believe this is intuitive because the reward is the primary signal that we aim to maximize in RL.

2. It is unclear to me why the proposed sampling distribution is meaningful. The sampling distribution described in Eq. 8 seems to imply a risk-sensitive sampling strategy without motivating this. For example, one may easily construct MDPs, where optimal trajectories are guaranteed to contain a very small reward. Such trajectories could not be sampled under this strategy, despite being optimal. Mathematically speaking, it is unclear to me what the implications of conditioning on an impossible event is here (i.e., what if $r_t$ can't be greater than $r_{min}$). I can unfortunately not see why one would expect this algorithm to explore reliably.

Response: Yes, it is certainly true that these types of problems where optimal trajectories contain very small rewards (i.e., sparse reward settings) are going to be quite challenging for most RL algorithms since exploration is usually done locally. However, we would like to point to the empirical evidence that our approach is doing something sensible and learning effectively even in sparse reward settings, namely the Sparse Reacher variations as well as the new Sparse Maze environments that we have added. We are not making any general theoretical claims about the optimality of our approach and instead show its efficacy on practical, standard benchmarks under difficult learning conditions.

Also, it is important to note that $r_{\text{min}}$ is a percentile value between 0 and 1 (exclusive) so that we draw $r_t$ from the truncated normal distribution $p(r_t \mid s_t, a_t, r_t > r_{\text{min}})$ where $r_t$ is guaranteed to be greater than the value corresponding to the percentile $r_{\text{min}}$. By truncating the distribution, we exclude the region where $r_t$ is less than the value associated with the percentile $r_{\text{min}}$. We are never conditioning on an impossible event where $r_t < r_{\text{min}}$ because there will always be a part of the distribution where $r_t$ is greater than the value corresponding to the percentile $r_{\text{min}}$.

Questions

1. Why do the authors aim to relieve the stationarity assumption in line 285. Since this GP models the transition dynamics of an MDP, the transition kernel should indeed be stationary. Aiming to address non-stationary MDPs, I believe, would entail significant complications for the entire learning algorithm.

Response: We believe there may be some confusion regarding our use of the term "stationary". We refer to non-stationary settings as ones where the dynamics and reward structures vary significantly across different regions of the input space. In GP literature, this is the normal interpretation of the word but we recognize that it can be misleading as this has nothing to do with time. In most GP applications, the mean function is represented in zero or constant form so, if the function is more complex than what the mean can capture, the covariance function has to take responsibility for modeling most of the function's behavior. Instead, our approach uses a neural network as the mean function, which allows the covariance function to focus less on describing the overall behavior of the function. We apologize for the confusion and have removed the word “stationary” from the paper.

2. In most experiments, the asymptotic performance of the shown algorithm seems to be lower than the base-SAC implementation in all of the environments. Is this due to the cutoff in plotting or is the asymptotic performance indeed lower in most environments? Inserting a dashed line as with SAC into the figure might clear this up.

Response: It is true that some training curves do not reach the SAC-best performance level within the maximum number of environment steps. In all experiments apart from the Sparse Reacher, at least one HOT-GP variant (of the three considered $r_{\text{min}}$ schedules) achieves the SAC-best performance after training for a fraction of the environment steps. On the Sparse Reacher task, we believe the HOT-GP performance would continue to increase if we had allowed the algorithm to continue running (i.e., this may be due to the plotting cutoff), however we did not systematically evaluate this due to computational constraints. Note that assessing optimal performance is secondary to the main motivation of our evaluation and we only provide the best-SAC performance to offer an idea of what good performance may be. Our aim in this paper was to evaluate the sample efficiency of HOT-GP relative to existing approaches by comparing algorithms’ performance after a fixed number of environment interactions, as done in H-UCRL (and many other papers).

3. Given that the authors perform k-branching rollouts, the value estimates bootstrapped at the end of model-generated rollouts should play an important role in quantifying uncertainty. Would omitting this uncertainty skew the ability of the algorithm to explore efficiently? It seems unlikely to me that Thompson sampling maintains its theoretical guarantees if one only samples single-step transitions around known trajectories.

Response: This is a really good question that touches on an important avenue worth studying. It is crucial to highlight that our work does not rely on any theoretical guarantees of Thompson sampling, nor do we claim that our algorithm achieves theoretically optimal exploration. Instead, we emphasize the empirical performance of our approach as evidence of its effectiveness.

4. My suggestion would be to analyze the implications of the proposed algorithmic components carefully. What is the implication of performing Thompson sampling with a sampling probability dependent on the expected return of some policy? What is the implication of performing Thompson sampling with a sampling probability dependent on the return and reward distribution of some policy (as suggested here). What is the implication of performing Thompson sampling with limited branched rollouts?

Response: These are extremely interesting questions! We are in complete agreement that modeling the return rather than the reward would allow us to be less myopic in exploration, which would improve sample efficiency even further. However, modeling the full expected return with principled epistemic uncertainty over the dynamics is a very challenging task. This is something that we want to explore in the future, as we believe that some of the concepts around exploration that we have presented in this paper can be useful in that setting.

Thank you for your response. We are glad to hear that our explanations have helped resolve some misunderstandings and points from your original review. We appreciate the additional feedback and aim to address these points individually.

1. As pointed out in my review, I think this paper would benefit greatly from a stronger theoretical justification of the proposed methods and ideas. I generally find the avenue of return-conditioned rollouts with learned models an interesting and important one.

Response: Thank you for the suggestion. We would like to highlight some theories that our work currently builds upon.

For a high-level theoretical justification of HOT-GP, we refer to the thresholded Thompson sampling regret bounds presented in [1]. In a combinatorial semi-bandit setting, the authors show that regret is no longer dependent on $T$, the number of timesteps, when the rare arms are never played. This is similar to how we perform optimistic Thompson sampling by drawing rewards from a truncated normal distribution, where the region where $r_t$ is less than the value associated with the percentile $r_{\text{min}}$ is excluded. For Thompson sampling regret bounds on reinforcement learning problems, we refer to the theoretical analysis in [2].

Regarding the performance implication of using limited k-branched rollouts, we refer to Theorem 4.3 in [3]. This theorem states that, as long as the returns of the policy under the estimated model are lower-bounded by $2r_{\text{max}} \left[\frac{\gamma^{k+1}\epsilon_{\pi}}{(1-\gamma)^2} + \frac{\gamma^{k}\epsilon_{\pi}}{(1-\gamma)} + \frac{k}{1-\gamma}(\epsilon_{m’}) \right]$, performance improvement on the true MDP is guaranteed. Here $\epsilon_{\pi}$ and $\epsilon_{m’}$ denotes the distribution shift of the policy and generalization error between iterations respectively. Notably, since the argmin with respect to $k$ of $\left[\frac{\gamma^{k+1}\epsilon_{\pi}}{(1-\gamma)^2} + \frac{\gamma^{k}\epsilon_{\pi}}{(1-\gamma)} + \frac{k}{1-\gamma}(\epsilon_{m’}) \right] > 0$, this theoretically motivates the use of k-branched rollouts. Going back to one of the questions raised in the original review, the implication of doing this with Thompson sampling is that Thompson sampling may not be able to explore the full range of outcomes, yet we get to benefit from the improvement in performance. We empirically study this tradeoff with an ablation study in Figure 3, showing that performing Thompson sampling over $k=5$ branched rollouts and full rollouts with $k=T$ (task horizon) significantly harms learning, likely due to accumulated model bias that prevents performance improvement in the true MDP.

Regarding return-conditioned rollouts with learned models, we agree that this is an interesting and important avenue. In our work, we treat the single step reward values as a proxy for the return and condition transitions on optimistic and feasible rewards. A natural next step would be to extend the optimistic horizon to match the task horizon and condition entire rollouts on optimistic total returns. In theory, this could enable the discovery of optimal trajectories, where intermediate states yield small rewards and later states provide larger rewards (as pointed out in your original review). Such exploration can be driven by epistemic uncertainty over performance as encoded over multi-step imagined returns or value functions. However, these approaches remain quite challenging due to the accumulation of model biases over multi-step imagined rollouts [3] and the rigidity of overestimated value functions [4, 5]. For instance, the recent method presented in [6] aims to be optimistic over the values yet does not reach SAC-best performance on some standard benchmark tasks. While this line of work is very promising, further research is needed to develop algorithms that can effectively explore using optimistic return-conditioned rollouts. By establishing the importance of joint uncertainty modeling for optimistic exploration, we believe that our work meaningfully contributes to advancing this area and we are eager to extend our work in this direction in the future. However, due to the many challenges involved, directly addressing optimistic return-conditioned rollouts is out of scope for this paper.

2. For a more empirically motivated paper, I think the experimental section would need to be exceptionally strong. In its current form, however, I find the empirical benefit not quite convincing enough (both in the performance on the shown experiments and in terms of the variety of experiments) for the high requirements of this venue.

Response: Regarding the experimental section, we would like to highlight that this has been improved in the newest version of the paper that was updated after your most recent comment. These updates directly respond to both criteria you mentioned: the variety of experiments and the performance on shown experiments.

First, following Reviewer LvwR’s suggestion, we have evaluated HOT-GP on two Sparse Point Mazes of varying difficulty from the D4RL suite (Figure 2). In the simpler U Maze environment, HOT-GP matches the sample efficiency and performance of alternative methods. In the more challenging Medium [sized] Maze environment, HOT-GP with higher levels of optimism ($r_{\text{min}}$ from 0.1 to 0.7) outperforms both HOT-GP with lower levels of optimism and all comparison methods. These results validate the findings from our original experiments and demonstrate HOT-GP’s adaptability to new, complex tasks requiring long-horizon exploration, all without requiring tuning $r_{\text{min}}$ or other parameters. Notably, we observe consistency in these results across diverse test suites. If there are additional specific experiments that you think would highlight particular characteristics of our model or approach, we would be happy to do such evaluations for the final version of the paper.

Secondly, to further address Question 2 in your original review, we reran our experiments on Sparse Reacher with more environment interactions to clarify the asymptotic performance (Figure 6). These results show that HOT-GP converges to SAC-best performance after just 1/10th of the environment steps. Across other tasks, our original results already show that at least one HOT-GP variant achieves performance comparable to the best model-free asymptotic results.

We hope that our response has sufficiently addressed your concerns, but if you have any further comments or questions please do not hesitate to ask. We believe that these additions strengthen the experimental section of the paper as they showcase both optimal policy convergence and robust performance across a diverse range of experiments. We would be grateful if you would consider increasing your score in light of these responses. Thank you!

[1] Bubeck and Sellke. First-order bayesian regret analysis of thompson sampling. Algorithmic Learning Theory, 2020.
[2] Moradipari et al. Improved bayesian regret bounds for thompson sampling in reinforcement learning. NeurIPS, 2023.
[3] Janner et al. When to trust your model: model-based policy optimization. NeurIPS, 2019.
[4] Sims et al. The edge-of-reach problem in offline model-based reinforcement learning. NeurIPS, 2024.
[5] Fujimoto et al. Addressing function approximation error in actor-critic methods. ICML, 2018.
[6] Luis et al. Model-based epistemic variance of values for risk-aware policy optimization. arXiv, 2024.

Thank you very much for taking the time to review our paper. We appreciate your comments and thoughtful feedback. Please find our responses below.

Weaknesses

1. The description for Fig.2 in Section 6.2, references several methods that are not shown in the plots. For instance, SAC and PPO as described in Line 430, “we use … SAC, and the on-policy PPO” and HOT-GP with k-branching (k=5) as described in Line 470.

Response: Thank you for pointing this out and we apologize for the confusion. We have uploaded a new paper pdf in which Figure 3 (the old Figure 2) has been updated with the correct corresponding training curves and algorithms.

2. Learning curves seem to be from the training episodes. It would be better to report the performance in evaluation episodes instead, as done in SAC.

Response: Our learning curves actually show the total reward averaged across evaluation episodes (not training episodes). We agree this was initially unclear and have clarified this metric in the new version of the paper.

3. The method introduces a crucial hyperparameter reward threshold $r_{min}$, which controls optimism during exploration. While the results do not seem to be overly sensitive to it, its effect in other tasks remains unclear. Although future work is mentioned to address this, its current implementation could limit its applicability.

Response: Indeed, we find that $r_{\text{min}}$ is not very challenging to set in practice, although setting this optimally is an open question for future research. Currently, we use a linear scheduler to increase $r_{\text{min}}$ during training. This effectively increases the level of optimism during training, as too much early/naive optimism could be detrimental to the learning process. The most principled approach to setting $r_{\text{min}}$ would be to set a distribution over $r_{\text{min}}$ and integrate it out, however this cannot be done in closed form and hence is computationally expensive. We plan to look into approximations that enable learning $r_{\text{min}}$ adaptively in future work. Note that it is common in the literature to introduce a hyperparameter to balance the trade-off between exploitation and exploration, as seen in H-UCRL with the $\beta$ hyperparameter. However, a key distinction with our approach is that the results are not very sensitive to the hyperparameter's value. In contrast, our experiments with H-UCRL revealed a strong dependence between the performance and the $\beta$ value.

Overall, we completely agree that better understanding the implications of this parameter is important. In this work, we want to convey that a non-zero $r_{\text{min}}$ value improves performance and, crucially, that this effect is robust across different tasks. Our results on the MuJoCo and VMAS tasks already highlight this robustness, and we are currently running additional experiments on two Sparse Point Mazes. Our results on these tasks so far also indicate consistent performance across various $r_{\text{min}}$ values, providing further evidence that our approach is robust to the specific value.

Questions

1. Could you provide any results on computational cost or computation speed for the proposed method?

Response: Certainly, we have added Table 2 into Appendix B comparing the average computation speed across each MuJoCo task and algorithm.

Thank you very much for your thoughtful follow-up review and increasing your score! We are pleased to hear that our revisions addressed your concerns and that you found the manuscript stronger as a result.

Regarding the papers highlighted by Reviewer LvwR, we have clarified their relationship to our work and also restated the contributions and novelty of our submission in our response. We are happy to engage further about this if you have any questions or comments. Additionally, please let us know if there is any other information we can provide to help strengthen the confidence of your recommendation. Thank you!