Careful at Estimation and Bold at Exploration for Deterministic Policy Gradient Algorithm

Thank you for your review.

For question 1,

1. Policy learning lags behind Q-function learning. From an empirical viewpoint, as described in our algorithm and previous algorithms (SAC, TD3), policy learning occurs after Q-function learning.
2. From a theoretical viewpoint, you can refer to the Soft Policy Iteration theorem in SAC, which mandates the initial Q update (soft policy evaluation) followed by policy update (soft policy improvement). In this scheme, Only when the Q value converges can the policy converge to the optimum.

For question 2,

    In the maximum entropy RL framework, the optimal Q value should be multimodal. This assumption was first proposed in the SQL[1] paper, and we also verify this assumption in Appendix E.5 (page 20, Fig. 13(c)), which shows that the Q function is multimodal in practice.

For question 3,

    We sample actions in the action space and then construct a new exploration policy with these actions corresponding to Q values. This operation weakens the relation between action and policy but strengthens the relation between action and Q value.

For qusetion 4,

1. DDQS samples actions with high Q-values according to the Qmax function and updates the target Q-value using the Qmin value. Not only does this guarantee convergence of the two Q-values in theory, but in practical applications, for a specific state, the $Q^{max}$ function encourages the sampling of actions with high Q-values, while the $Q^{min}$ function ensures stable value updates. These two functions balance the learning of the two Q-functions.
2. As for the new formula, we can simplify it to $(\sum_a (Q_1(s,a)+Q_2(s,a)))/Z$, since $Q^{max}$ and $Q^{min}$ are always different, and the partition function Z normalizes the distribution. If both $Q_{1}$ and $Q_{2}$ are large, this operator equals $Q^{max}$. However, if $Q_{1}(s,a)$ is large and $Q_{2}$ is small, the sum leads to an uniform sampling, which makes no sense.

For weakness 2,

    We have discussed the time consumption in Appendix E.4, Page 16.

For weakness 4,

    We have updated the PDF file. You can refer to Table F on Page 21 to check the numerical results. In different environments, the optimal results for these three time steps are all displayed in bold.

[1]Haarnoja T, et al. Reinforcement learning with deep energy-based policies,ICML, 2017.

Hope these comments address your concerns.

Thanks for your review.

For weakness 1,

1. We have made every effort to consider the coherence of paragraphs to ensure the continuity of the reading. For example, at the beginning and end of section 4.2, we attempted to connect sections 4.1 and 4.3.

2. As for the issue of mismatch:

   • for the policy update, there appears to be no issue with the mismatch between the behavior policy and the target policy in off-policy algorithms. We explicitly introduce the conservative policy to establish a connection between the minimum Q value and policy learning. This provides a direct understanding of the role that the minimum Q value plays in the previous paper.
   • for value update, we update the target value with transition sampled from the buffer(collected with policy $\pi_{E}$).

For weakness 2 and 3,

1. We respectfully disagree with some of the viewpoints you've presented. Addressing sparse rewards indeed requires exploration methods based on the state space, but methods for enhancing exploration are not necessarily designed to solve the sparse reward problem.

2. Although our title includes exploration, it does not address sparse reward tasks (explore the state space). Instead, it focuses on achieving efficient continuous control in a complex action space(explore the action space).

3. Citations 1-5 are interesting papers, but they are not very relevant to our research content(problems descirbed in section 3).

for weakness 4,

1. Thanks for introducing the AW-OPT paper, we also read the QT-OPT paper. We understand why you are concerned about the sparse reward issue.

2. The AW-OPT method and our approach do share some similarities. We have updated the paper PDF and introduced this work in the related work section.

for weakness 5,

1. Figure 4 adequately illustrates that our proposed method can address the issues mentioned in Section 3. We present more comprehensive results not only to demonstrate the effectiveness of the approach but also to highlight its limitations.

2. From Fig.3(f) to Fig.3(a), the action space of the environment becomes increasingly complex. To demonstrate the effectiveness of our method, the performance in the challenging task (humanoid) should be more convincing. In Table F(appendix, page21), we present numerical results, showing that even if our results are not on par with the optimal ones, the numerical difference is minimal. Our method was not intentionally tuned.

3. Regarding the results in Fig.5 (a-b), in the flagrun environment, the changing flag position makes it challenging for the Q function to learn. Using the Q function to guide exploration in the early stages can lead to a performance loss, but this can be compensated for later (as indicated by the increasing trends in bacc and oac). The FlagrunHarder environment poses even greater difficulty. Please note the scale of the y-axis, where it's challenging for the episode return to reach 200, indicating that interaction occurs before the first flag change. In this scenario, both bacc and oac prove effective. The high variance in OAC is due to its difficulty in accurately estimating the exploration direction. It also reflects the robustness of our method, as we have mentioned in the paper.

Hope this comment could address your concerns.

Dear Reviewer 3zdT,

Thank you for your reply. We would like to talk more about sparse reward, even though it is not very related to the issue our paper aims to address.

1. We feel it is inappropriate to directly compare our method with sparse reward methods.

   • Generally, state space exploration methods often involve generating intrinsic rewards (no reward is an extreme case of sparse reward). Applying such methods directly in dense reward environments results in reward conflicts. Deciding whether to depend on intrinsic rewards or environmental rewards for learning becomes a challenging task.
   • Consider the noisy-TV problem[1], where intrinsic rewards impede the maximization of environmental rewards, presenting a challenge. And subsequent method[2] merely aim to alleviate reward conflicts.
   • Currently, we lack a clear strategy for balancing intrinsic and environmental rewards when implementing sparse reward methods in dense reward environments. When the intrinsic reward is small, its necessity becomes questionable. On the other hand, if it is substantial, it intervenes the normal policy learning.

2. (BACC+HER) This is indeed an interesting question. There seems to be a way to avoid conflicts very well. We could explore the action space to achieve the generated (dense reward) internal goal with BACC at a low level. Then, we can explore internal goals (state) with HER to achieve external goal at a higher level. This collaboration appears to be an ideal strategy.

[1] Burda Y, et al. Exploration by random network distillation. ICLR, 2019.

[2] Chen E, et al. Redeeming intrinsic rewards via constrained optimization. NIPS, 2022

Hope this reply could address your concerns in weakness 2 and 3.

Thank you for taking the time to review. We would like to provide additional explanations to address your concerns.

For Strengths:

    Thank you for recognizing our work in exploring the continuous action space. In practical applications, we believe this can be helpful in training robots to acquire some fundamental capabilities.

For Weaknesses:

1. Figure 4 illustrates the effectiveness of our approach in addressing the challenges presented in Section 3. And in Figure 3, the performance in the challenging task, particularly with the humanoid, is expected to be more convincing. From Fig. 3(f) to Fig. 3(a), the action space of the environment becomes progressively more complex. We present more comprehensive results not only to showcase the effectiveness of the approach but also to emphasize its limitations.

2. The "estimation error" has been thoroughly addressed in the TD3 paper. They update the target Q value by utilizing the minimum value of the double Q function to handle the impractical argmax operator in the continuous action space. In terms of value update, our approach is also grounded in this concept. We further propose the conservative policy to offer a direct understanding of the role that the minimum Q value plays in the previous paper.

For Questions,

1. The interval is 100, and this interval remains consistent across all plots. In some of the plots, there is a noticeable amount of jaggedness in the lines.

2. "When evaluating the quality...," this primarily pertains to the results in Figure 4.

   • What we want to convey is that it is not very intuitive to identify where the problem lies in the algorithm from the evaluation results. Instead, we should observe the exploration results to determine what happened.
   • In Figure 3, the y-axis shows the evaluated average episode return(deterministic policy). In Figure 4, the y-axis displays the exploration average episode return(exploration policy).

3. The sparse reward is not the core issue that our paper aims to address. The problems we are addressing are outlined in Section 3, unguided exploration and exclusive policy.

   • Although exploration(with more techniques) is the most effective way to tackle sparse rewards , it is not designed solely to address the sparse reward problem, it has wider applications.
   • Our method explores the action space in dense reward environments to achieve more efficient robot control.

4. "compare with RND...",

   • Sparse reward methods are not applicable to dense reward environments. It is challenging to balance environmental rewards and the intrinsic reward generated by the sparse reward exploration method. The most well-known issue is the Noisy-TV problem mentioned in the RND paper.
   • This problem has also been further discussed in the following interesting paper [1].

   [1] Chen E, et al. Redeeming intrinsic rewards via constrained optimization. NIPS, 2022

Hope this comment could address your concerns.