Thank you for your engagement and constructive review of our manuscript! We would love to address each of your comments and questions.

Response to Weaknesses:

W1: The scope of the paper is rather limited as the authors only tackle Soft Actor-Critic (SAC) in their paper. This paper can be extended to other off-policy algorithms as they all share the issues of premature convergence of the exploration and the influence from different UTD ratios.

Response to W1:

Thank you for highlighting this point!

KEA is indeed designed with SAC in mind, as SAC's ability to dynamically balance exploration and exploitation in off-policy settings provides a suitable foundation for our approach.

For KEA to be applicable to other off-policy algorithms, there must be a similar interaction between the original exploration strategy and curiosity-based exploration. For example, in epsilon-greedy exploration, commonly used in Deep Q-Networks (DQN), actions are selected based on a fixed probability either through exploitation or random sampling. Since this exploration mechanism does not inherently interact with curiosity-based exploration, the balance-shifting behavior observed in SAC may not arise.

Investigating KEA's applicability to other off-policy RL algorithms is an interesting and valuable direction. We have initiated preliminary experiments to explore this possibility, but additional time is required for setup and training. We will provide updated results as soon as they are available before the rebuttal ends.

W2: The experiments are also not very convincing. First, the proposed 2D navigation task is too simple and the performance improvement is also not significant. Second, NovelD and RND both are used as baseline algorithms and NovelD has achieved better performance compared to RND. Yet, the latter experiments are all about RND. Thirdly, in the continuous tasks, the three tasks are not trained for enough time. The performance imporvement is also limited.

Response to W2:

Thank you for the constructive feedback. We would love to address the concerns raised regarding our experiments and clarify the significance of our results.

First, the results from the 2D navigation task are summarized in the table below. KEA-RND achieves more than a 70% improvement compared to RND. While the final performance of KEA-NovelD is comparable to that of NovelD, KEA-NovelD converges significantly faster, reaching a return of 0.6 around 190,000 environment steps, whereas NovelD requires approximately 250,000 steps to achieve a similar return.

As we highlighted in Response to Q1 in our response to the reviewer tbHf, the 2D navigation task is not as simple as it may seem. For instance, SAC with sparse rewards fails to converge even when the number of samples is doubled, highlighting the challenge of this environment when intrinsic reward signals are not effectively utilized.

Second, regarding the choice of RND over NovelD for the analysis of varying UTD ratios, we selected RND due to its more straightforward exploration strategy, which enables clear and intuitive analyses and interpretations. Please refer to Response to Q5 for further details.

For the analysis of varying UTD ratios, we chose to use RND due to its more straightforward exploration strategy, which enables clear and intuitive analyses and interpretations. This simplicity helps isolate the specific impact of KEA without the added complexity introduced by NovelD. By using RND, we can more clearly interpret how KEA performs under different configurations, focusing on the core contributions of our method.

Finally, we acknowledge the concern regarding the training time for the continuous control tasks. We have initiated experiments on the DeepMind Control Suite tasks with 1M environment steps to address this, but further time is required to complete them. We will provide updated results as soon as they are available before the rebuttal ends. Nonetheless, we emphasize that KEA demonstrates significantly faster learning compared to the baselines, particularly under limited demonstration steps.

We hope this clarifies the contributions of our work and addresses your concerns.

Response to Weaknesses:

W3: The presentation/clarity of the paper should also be improved. For example, I didn't fully get Fig. 1's layout in the beginning. Some sentences are also written in a vague way without further explaination (See questions section). The training of the co-behavior policy is not mathematically defined.

Response to W3:

Thank you for your valuable suggestion regarding the clarity and presentation of our manuscript.

First, the interpretation of Fig. 1 is explained in Response to Q1, where we clarify how curiosity-based exploration operates in SAC. Specifically, we discuss how it encourages the agent to explore states with high intrinsic rewards but may fail the agent to recognize truly unvisited states due to a lack of experience in the replay buffer. We will revise Fig. 1 to improve its clarity and alignment with the explanation provided.

Second, as mentioned in Response to Q2, we will reference the original SAC paper [1] to support the statement about SAC’s sensitivity to reward scaling.

Finally, as described in Response to Q3, we implement the co-behavior agent ($A^{B}$) using SAC, which shares a unified replay buffer with $A^{SAC}$ to enhance data efficiency. Both agents are trained concurrently by sampling experiences from the shared buffer.

We will revise the manuscript with further explanation and ensure clarity.

[1] Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018, July). "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." In International conference on machine learning (pp. 1861-1870). PMLR.

The Results in Response to W1:

To explore KEA's generalization beyond SAC, we extended KEA to DQN and evaluated its performance on the 2D Navigation task.

The preliminary results indicate that KEA does not improve the performance of DQN when combined with RND in this task. The results are summarized in the table below:

As discussed in Response to W1, KEA addresses potential problems arising from the interaction between different explorations. However, epsilon-greedy exploration, commonly used in DQN, does not interact significantly with novelty-based exploration. Instead, it either takes random action or selects the optimal action (considering the intrinsic reward).

As part of future work, extending KEA to other off-policy methods, such as DDPG, TD3, or NAF, represents an interesting direction to further explore its generalization capabilities.

The Results in Response to W2:

The results of the extended training for the DeepMind Control Suite tasks to evaluate convergence at 1M steps are summarized in the table below.

• Walker Run Sparse

• Cheetah Run Sparse

• Reacher Hard Sparse

Thank you for your engagement and constructive review of our manuscript! We would love to address each of your comments and questions.

Response to Weaknesses:

W1: The underlying assumption for the switching criterion is not intuitive. In Equation (4) and the following explanation (from around Line 239 to Line 245), it is stated that for low intrinsic reward regions (i.e., low novelty), KEA switches to curiosity-based exploration to encourage the agent to explore novel areas. Conversely, for high novelty regions, KEA switches to a more stochastic sampling policy. However, it would make sense if it was the other way around, as high intrinsic rewards usually indicate that the area is novel and worth exploring further, making it more logical to apply a strong exploration strategy. Conversely, low intrinsic reward regions (low novelty) might be better suited for stochastic sampling. I recommend that this assumption be supported with more quantitative analysis or additional comparative experiments.

Response to W1:

Thank you for your insightful comment. We appreciate the opportunity to clarify the rationale behind our switching criterion and address your concerns.

To evaluate the reviewer's suggestion, we implemented a new switching mechanism that inverses KEA's design. Specifically, in the reviewer's proposed mechanism, the agent switches to $A^{SAC}$ in high intrinsic reward regions and to $A^{B}$ in low intrinsic reward regions. The experimental results for this alternative design are presented in the table below.

Curiosity-Based Exploration Curiosity-based exploration encourages the agent to explore states with high intrinsic rewards, which usually indicate novel states. However, in SAC combined with curiosity-based exploration, these "novel" states refer to states that have already been visited and exhibit high novelty, rather than unvisited states. Unvisited states, while potentially offering high intrinsic rewards, are not recognized by the agent because there are no experiences about them in the replay buffer. As a result, curiosity-based exploration often guides the agent to repeatedly exploit those states with relatively higher novelty among the visited states.

Rationale Behind KEA’s Switching Mechanism Exploration of truly unvisited states relies on the stochastic sampling provided by the policy in SAC. Although the agent dynamically shifts the balance between curiosity-based exploration and stochastic sampling, this delay, along with the risk of repeatedly collecting similar experiences in states with high intrinsic rewards, can lead to inefficiencies. To address this inefficiency, KEA proactively switches to a high-stochasticity agent ($A^B$) in high intrinsic reward regions without waiting for the balance to naturally shift.

Challenges with the Reviewer’s Suggested Mechanism When using the reviewer's proposed mechanism, the inefficiencies mentioned above still persist. Moreover, $A^{B}$ doesn't consider intrinsic reward. It will perform stochastic actions in the low intrinsic reward states before reaching the goal. These may decrease the learning efficiency.

Experimental Comparison We tested the reviewer's suggested switching mechanism (denoted as KEA-RND-inv) in the 2D Navigation task, using RND as the curiosity-based exploration strategy for both mechanisms. The results, shown below, demonstrate that KEA's switching mechanism outperforms the reviewer's mechanism, achieving a higher mean episodic return and reduced standard deviation.

Thanks to the authors' response, as many of my questions have been answered, I’d like to increase the score to 5. However, I still believe there is significant room for improvement for the paper to reach the level of a well-rounded contribution. Therefore, I still consider this paper to be marginally below the acceptance threshold.

Response to Questions:

Q1: How sensitive is the model to the switching threshold parameter?

Response to Q1:

We conducted experiments to analyze the sensitivity of KEA-RND to different switching thresholds, using the 2D Navigation task as the evaluation setting. The results are summarized in the table below.

Our findings indicate that while varying the switching threshold does affect KEA's performance, all tested configurations consistently outperform RND (0.292 \pm 0.197). This suggests that KEA is robust to threshold variations within a reasonable range.

(For reference, the intrinsic rewards are normalized using a running mean and standard deviation, resulting in a standard value of approximately 1.)

Q2: Was this threshold hand-tuned or optimized in some manner?

Response to Q2:

In this paper, the switching threshold was hand-tuned based on preliminary experiments. We hand-tuned the threshold by checking the usage of $A^{SAC}$ is around 85% to 90%.

However, we acknowledge that automated hyperparameter tuning methods, such as grid search or Bayesian optimization, could be employed to systematically optimize the threshold.

Q3: What is the computational impact of using two agents in terms of memory and processing requirements?

Response to Q3:

As outlined in Section 2.3, the two agents utilize a unified replay buffer to share experiences, enhancing data efficiency and minimizing additional memory overhead. During the forward pass, the current intrinsic reward is computed to guide the switching mechanism in selecting which agent will estimate the action. Crucially, the action estimation process itself is equivalent to that of a single SAC agent, ensuring that the computational impact in terms of memory and processing requirements remains minimal.

Q4: Does the model perform well in highly dynamic environments with frequently changing goals?

Response to Q4:

Dynamic environments with frequently changing goals indeed pose additional challenges for exploration. However, the complex interaction between two exploration strategies ---- stochastic sampling and curiosity-based exploration ---- still exists when combining SAC with curiosity-based exploration.

KEA improves performance primarily by addressing this challenge, enabling the agent to better explore unvisited states. In this paper, we focus on fixed-goal environments to provide clear and intuitive analyses and facilitate direct comparisons with baseline methods.

Q5: Can the method applied to other RL algorithms, beyond SAC?

Response to Q5:

Thank you for the interesting question.

In this paper, we focus on the potential problems arising from the interaction between SAC's stochastic exploration and curiosity-based exploration. For KEA to be applicable to other RL algorithms, there must be a similar interaction between the original exploration strategy and curiosity-based exploration.

For example, in epsilon-greedy exploration, commonly used in Deep Q-Networks (DQN), based on a fixed probability, actions are chosen either through exploitation or random sampling. Since this exploration mechanism does not interact with curiosity-based exploration, the balance-shifting behavior observed in SAC may not occur.

Investigating KEA's applicability to other RL algorithms is an interesting direction. We have begun preliminary experiments in this area, but further time is required for setup and model training. We will provide updated results as soon as they are available before the rebuttal ends.

[1] Zhang, T., Xu, H., Wang, X., Wu, Y., Keutzer, K., Gonzalez, J. E., & Tian, Y. (2021). "Noveld: A simple yet effective exploration criterion." Advances in Neural Information Processing Systems, 34, 25217-25230.

[2] Wan, S., Tang, Y., Tian, Y., & Kaneko, T. (2023). "DEIR: efficient and robust exploration through discriminative-model-based episodic intrinsic rewards." arXiv preprint arXiv:2304.10770.

[3] Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., & Dabney, W. (2018, September). "Recurrent experience replay in distributed reinforcement learning." In International conference on learning representations.

Thank you for your engagement and constructive review of our manuscript! We would love to address each of your comments and questions.

Response to Weaknesses:

W1: The paper should discuss how the switching mechanism threshold is tuned and its effect on the performance. This appears to be a key mechanism but somehow ablation studies on this component are limited.

Response to W1:

Thank you for the constructive feedback.

We have conducted experiments to analyze the sensitivity of KEA to different switching thresholds, as detailed in Response to Q1. These experiments demonstrate how the threshold selection impacts performance, providing insights into its role in KEA's effectiveness.

Additionally, in this paper, we hand-tuned the switching threshold based on preliminary experiments. Please refer to Response to Q2 for further details.

W2: The evaluation lacks comparison with papers that address a similar issue, such as [Never give up: Learning directed exploration strategies, Badia et al., 2020]. I would suggest adding stronger baselines to validate the proposed method.

Response to W2:

Thank you for the valuable suggestion.

NovelD [1] is often regarded as performing better than Never Give Up (NGU) for addressing hard exploration tasks [2]. Comparing with NovelD should be meaningful to demonstrate KEA's ability.

On the other hand, NGU uses Recurrent Replay Distributed DQN (R2D2) [3], an RNN-based Deep Q-Network (DQN) algorithm. However, in this paper, we focus on combining KEA with Soft Actor-Critic (SAC). Exploring whether KEA can improve the performance of DQN combined with curiosity-based exploration is an interesting question. Please refer to Response to Q5 for more details. Furthermore, understanding how RNN-based methods impact the interaction between different exploration strategies would be an intriguing line of investigation.

That said, directly applying KEA to NGU is not straightforward, due to the differences in algorithmic structure and exploration strategies. This remains an open question and an exciting avenue for future work to investigate the interplay between RNN-based models and dual exploration strategies.

W3: The paper’s proposed method, KEA, is designed specifically with SAC in mind, leveraging SAC’s ability to handle exploration-exploitation trade-offs in off-policy settings. However, the authors do not discuss how KEA might generalize to other off-policy algorithms.

Response to W3:

Thank you for highlighting this point. KEA is indeed designed with SAC in mind, as SAC's ability to balance exploration and exploitation in off-policy settings provides a suitable foundation for our approach. Investigating KEA's applicability to other off-policy RL algorithms is an interesting and valuable direction.

We have initiated preliminary experiments to explore this possibility, but additional time is required for setup and training. For further details on our thoughts regarding generalization to other RL algorithms, please refer to Response to Q5.

W4: Finally, one of my main concerns comes from the switching mechanism, which utilizes a fixed threshold to decide when to switch between the SAC agent with curiosity-driven exploration and the co-behavior agent. While this approach appears effective in simulations, using a fixed threshold might lack the flexibility needed for different environments or tasks, especially those with varying reward sparsity or novelty patterns. This inflexibility could limit KEA’s adaptability, potentially leading to less effective exploration in more complex or dynamic scenarios

Response to W4:

Thank you for the insightful feedback.

The intrinsic rewards in KEA are normalized using a running mean and standard deviation, which ensures that their range is consistent across different environments and tasks. For instance, in the 2D Navigation task, we set the switching threshold to 1, while in the DeepMind Control Suite tasks, we consistently use a threshold of 0.75 across all tasks. This normalization helps mitigate the impact of varying reward scales and sparsity patterns.

We agree that a dynamic threshold could potentially improve KEA's adaptability to environments with highly variable reward sparsity or novelty patterns. However, defining an effective dynamic switching mechanism, including identifying the appropriate signals to adjust the threshold, is a non-trivial challenge and an interesting direction for future research.

To explore KEA's generalization beyond SAC, we extended KEA to DQN and evaluated its performance on the 2D Navigation task.

The preliminary results indicate that KEA does not improve the performance of DQN when combined with RND in this task. The results are summarized in the table below:

As discussed in Response to Q5, KEA addresses potential problems arising from the interaction between different explorations. However, epsilon-greedy exploration, commonly used in DQN, does not interact significantly with novelty-based exploration. Instead, it either takes random action or selects the optimal action (considering the intrinsic reward).

As part of future work, extending KEA to other off-policy methods, such as DDPG, TD3, or NAF, represents an interesting direction to further explore its generalization capabilities.

Response to Questions:

Q1: In figure 3, the SAC line seems to be flat? I am surprised this task cannot be learned at all by SAC, and that the best method only reaches a ~60% success rate. Perhaps the degree of sparsity is not clear to me from the task description. Perhaps that sparsity could be elaborated on somehow?

Response to Q1:

Thank you for the question.

The 2D Navigation task takes place on a 41x41 grid with sparse rewards. The minimum steps to reach the goal range from 18 (closest start) to 55 (farthest start). Episodes terminate if the agent hits boundaries, obstacles, or reaches the goal. A key challenge is a narrow 4-cell-wide aisle linking the left and right regions, making it difficult for the agent to discover and access the goal. This sparsity in rewards and constrained exploration space explains why SAC struggles to learn this task.

We also tested SAC with 1 million environment steps under varying UTD ratios, but it consistently failed to reach the goal, further highlighting the difficulty of this environment for methods lacking effective exploration mechanisms.

In Figure 3, the y-axis represents mean episodic returns during training. While SAC fails to achieve meaningful progress, NovelD and KEA-NovelD both reach a 100% success rate. Notably, KEA-NovelD converges to 100% success earlier, at approximately 160,000 environment steps, compared to NovelD's 250,000 steps.

Q2: I suggest adjusting page spacing so that the critical algorithms 3 and 4 are on the same page, since they are connected and key to this paper's contribution

Response to Q2:

Thanks for the suggestion! It is indeed making critical algorithms 3 and 4 more readable. We will make sure the two algorithms appear together in the final formatting of the paper.

Q3: Figure 4 feels like supplemental material to me, since it's basically just validating hyperparameter choices.

Response to Q3:

Thank you for the feedback. We apologize for any lack of clarity in our explanation.

KEA primarily addresses exploration inefficiencies arising from the complex interaction between two exploration strategies. While task difficulty and environment design are key factors, exploration efficiency is also influenced by how aggressively the SAC agent and intrinsic reward model are updated.

The UTD ratio affects the evolution of both entropy (which maintains stochastic sampling) and intrinsic rewards, thereby impacting the shifting between the two exploration strategies. When intrinsic rewards impact more than entropy, the agent tends to revisit the same state. Conversely, when entropy has a stronger influence, the agent engages in more random exploration.

Figure 4 highlights KEA's ability to address these challenges by proactively coordinating the two exploration strategies. It demonstrates KEA's robustness under different patterns of shifting between exploration strategies (via varying UTD settings).

We believe the figure provides relevant context for understanding the benefits and trade-offs of our proposed exploration strategy, but we are willing to relegate it to supplementary material if the reviewers disagree.

Q4: The caption on figure 6 should say what the shaded regions represent.

Response to Q4:

Thank you for the feedback! The shaded regions in Figure 6 represent the standard deviation across the evaluation runs. We will update the manuscript to clarify this, along with other suggested revisions.

Q5: Also in figure 6, it looks like the algorithms haven't clearly converged on most of the three tasks. 500k environment steps is not a very large number, perhaps it would be worthwhile to run these experiments for the commonly-used 1 million steps to make sure there's separation at convergence?

Response to Q5:

We have started to run DeepMind Control Suite tasks with 1M environment steps but further time is required. We will provide updated results as soon as they are available before the rebuttal ends. Nonetheless, we emphasize that KEA demonstrates significantly faster learning compared to the baselines, particularly under limited environment steps. This faster convergence can be a key advantage in many practically relevant tasks, such as e.g. learning physical systems in robotics problems.

Thank you for your engagement and constructive review of our manuscript! We would love to address each of your comments and questions.

Response to Weaknesses:

The core issue I have with this paper is that there's very little exploration of the proposed algorithm beyond claims of superior performance. The core concept is simple (not a bad thing), but there's no real consideration given to what makes it good or useful. Seeing as the core contribution of this work is very simple, it should be straightforward to do deeper analysis and benchmarking, but this paper doesn't do much of that, and largely just concludes "it works."

As a result, I can't help but feel this paper is just sort of insubstantial and doesn't contribute that much to the field. It proposes an idea that is novel, but straightforward as an extension of past work, and doesn't do anything with the idea beyond show that it outperforms the baseline. If accepted as is, I'm not sure how much this will actually move the topic of exploration forward.

As such, I'm inclined to recommend rejection, but would like to see an expanded version of this work submitted to a future conference, because I do like the core idea of dynamically switching between explore and exploit policies for off-policy RL.

Response to Weakness:

We appreciate your thoughtful feedback and positive comments on the novelty and potential of proactively coordinating exploration strategies in off-policy RL. We understand your concerns regarding the depth of analysis and benchmarking, and we aim to clarify the contributions and significance of our work.

The core contribution of this work includes introducing a novel method that proactively coordinates exploration strategies (it switches between two exploration strategies from $A^{SAC} and A^{B}$, rather than switches between explore and exploit policies) and analyzing how this coordination addresses key challenges in combining SAC with curiosity-based exploration. Specifically, we provide insights into the interaction between intrinsic rewards and stochastic policies, demonstrating that our approach mitigates these challenges. While the concept is simple, its simplicity is a strength, making the method easily adaptable to a range of existing curiosity-based exploration methods.

In our experiments, beyond merely demonstrating superior performance, we also highlight:

• Adaptability: We apply KEA in RND and NovelD, showcasing its adaptability and ability to improve performance across different exploration paradigms.
• Robustness: Our experiments on varying UTD Ratios emphasize KEA's robustness under different shifting dynamics between exploration strategies (as detailed in Response to Q3).

During the rebuttal, to provide a deeper understanding of when and why the proposed method is effective, we have conducted new experiments to analyze the sensitivity of KEA to different switching thresholds ($\sigma$), as described in Response to Q7 and Response to Q8. These experiments further illustrate the structured and adaptable behavior of KEA.

Furthermore, we are exploring the application of KEA to other RL algorithms to assess its generalizability. While further time is needed to complete these experiments, we will provide updated results as soon as they are available before the rebuttal ends.

The results of the extended training for the DeepMind Control Suite tasks to evaluate convergence at 1M steps are summarized in the table below.

• Walker Run Sparse

• Cheetah Run Sparse

• Reacher Hard Sparse