Thanks for the detailed review and valuable feedback on our paper. We appreciate the effort to review our work. Below is our response to the reviewer's comments.

Weakness (convergence rate):

Thanks for pointing out the need for convergence rate analysis. We have addressed the convergence rate of the proposed method in the general response above. Please check the general response.

Q1 (feasibility):

The feasibility function is defined as $\mathcal{F}(x)=0$ if $x\leq0$ else $\infty$, which is commented in the footnote of line 114. This function is used to express the constrained optimization problem as an unconstrained one in equation (6), since two problems "$\max_\pi J_R(\pi)$ s.t. $J_{C_i}(\pi)\leq d_i$" and "$\max_\pi J_R(\pi) - \sum_{i=1}^N\mathcal{F}(J_{C_i}(\pi) - d_i)$" are equivalent. When solving the risk-constrained RL problem, the feasibility function is not used directly; instead, the constraints are used as they are, as shown in equation (7).

According to Theorem 5.3, an optimal policy for the inner problem can be obtained if the policy is updated using the proposed update rule, which is described below in line 169. As mentioned in lines 171-172, the proposed update rule requires satisfying the following conditions on $\lambda_t$ and $\nu_t$, which the reviewer refers to as "dual-variables": 1) $\lambda_{t,i}$ and $\nu_t$ should exist within $[0, \lambda_\mathrm{max}]$, and 2) $\lambda_{t,i}(J_{C_i}(\theta)-d_i)\geq0$ and $\sum_i \lambda_{t,i}=1$ when constraints are not satisfied. Various methods for choosing $\lambda_t$ and $\nu_t$ that satisfy the conditions are presented in Appendix B. By using one of these methods to calculate $\lambda_t$ and $\nu_t$, an optimal policy can be obtained according to Theorem 5.3, ensuring that the constraints are satisfied. A brief insight into the proof of Theorem 5.3 is as follows: the update rule is designed to give more weight to the objective function when the constraints are satisfied and more weight to the constraints when they are not. As a result, through repetitive policy updates, the policy converges within or on the boundary of the constraints.

Q2 (outer optimization):

The reviewer seems to think that calculating $J(\pi; \beta)$ requires solving another inner problem, which is different from Section 5. In other words, the reviewer seems to think that in order to train the sampler, it is necessary to find a policy that maximizes $J(\pi; \beta)$. However, this is not correct. The policy is only updated using the method introduced in Section 5, and only the sampler is trained in Section 6. There is no other inner problem. As seen in the equation below line 240, the sampler's loss function is composed of $J(\pi_{\beta, t}; \beta)$, where $\pi_{\beta, t}$​ is the policy obtained by solving the inner problem from Section 5 for $\beta$ over $t$ iterations. $J(\pi_{\beta, t}; \beta)$ can be calculated using value functions corresponding to $\pi_{\beta, t}$​, and the sampler is trained to output higher probabilities for $\beta$ that yield a higher $J(\pi_{\beta, t}; \beta)$ value.

To be more specific, policies are trained for each of several $\beta$ using the method introduced in Section 5, and these policies are expressed as { $\pi_{\beta,t} | \beta \in B^N$ }. Then, we use these policies to calculate $J(\pi_{\beta,t};\beta)$ and use these values to train the sampler. To implement the set of policies, $\beta$ along with the state are added as inputs to the actor-network $\pi_\theta(a|s ; \beta)$, as detailed in Appendix C.

Additionally, the explanation on lines 237-238 does not mean that we need to find $\max_\pi J(\pi; \beta)$, but rather demonstrates that $J(\pi; \beta)$ can be used as a measure of how well RCRL is solved. If there is any misunderstanding on our answer, please provide further clarification.

Q3 (Robbins-Monro condition):

Since it is possible to adjust the learning rate freely, the Robbins-Monro condition can be satisfied. Furthermore, as mentioned in the above response to Q2, there is no additional inner problem in calculating $J(\pi;\beta)$. Therefore, the method described in Section 6 does not need to satisfy the Robbins-Monro condition.

Thanks for the response. The question on the inequality seems to be arised from the proof of the convergence rate in the general response. In the proof, we said "$J_R(\pi^*) - J_R(\pi_t) \leq 0$ for $t \in \mathcal{N}$", and $t \in \mathcal{N}$ means that the policy at $t$ iteration, $\pi_t$, satisfies all constraints. Therefore, among the constraint-satisfying policies, $\pi^*$ has the maximum value of $J_R$, which results in $J_R(\pi^*) \geq J_R(\pi_t)$.

Thanks for the thorough review and insightful comments on our paper. We appreciate the effort to review our work. The following is our response to the reviewer's comments.

Weakness 1 (inner and outer problems)

As the reviewer commented, handling risk by separating the given problem into the inner and outer problems can be considered as not new; instead, the proposed method for solving each problem can be considered novel. Therefore, we will modify lines 50-51 to emphasize the proposed process of the inner and outer problems rather than the bilevel optimization framework itself.

Weakness 2 (algorithm details)

Algorithm 1 only provides a brief overview of the proposed method, but Algorithm 2 in Appendix C shows details of the method, including information on the policy and the distribution update. To improve clarity, we will change the title of Algorithm 1 to "Overview of SRCPO," and at the end of Chapter 4, we will add a note indicating that a detailed algorithm is provided in Appendix C.

Weakness 3 (overall performance)

Through Section 5, we can find an optimal policy of the inner problem for various $\beta$. It means that we can find a set of optimal policies, $\Pi^*=${ $\pi_\beta^*|\beta \in B^N$ }, where $\pi_\beta^*$ denotes the optimal policy for $\beta$. Additionally, through Section 6, we can obtain an optimal sampler $\xi^*$ that samples only $\beta^* = \arg\max_\beta J_R(\pi_\beta^*)$. Thus, by solving the inner and outer problems, we obtain $\Pi^*$ and $\xi^*$. Sampling $\beta^*$ from $\xi^*$ and finding the corresponding policy in $\Pi^*$ leads to the optimal policy $\pi_{\beta^*}^*$ of the RCRL problem. In conclusion, by solving the inner and outer problems respectively, we can obtain an optimal policy for the original RCRL problem defined in equation (5), thus ensuring overall performance.

Weakness 4 (convergence rate)

We have addressed the convergence rate of the proposed method in the general response above. Please check the general response.

Thank the authors for their response. Since there exists a technical error in the proof, I think that this paper needs a major revision, and thus I decreased my score.

Thanks for the valuable feedback on our paper. We appreciate the effort in reviewing our work. We have carefully considered the reviewer's comments. Below, we address the concerns the reviewer has raised:

Q1 (performance gap):

In the general response above, we analyzed the performance gap caused by the spectrum approximation; please refer to the general response. Briefly, we first bounded the gap by the difference in performance of the original problem at different thresholds. Analyzing the performance difference at different thresholds in a constrained optimization problem is challenging due to discontinuous changes in the feasible set or optimal points. Thus, we made some assumptions to analyze the gap and found that the gap is bounded by $KR_\mathrm{max}/(C_\mathrm{max}M)$. This suggests that the gap affects the scale of the reward and costs rather than the state and action spaces. Therefore, we conjecture that the value of $M$can be set independently of the state and action spaces.

Q2 (intuition behind the sampling process):

Thanks for pointing out the need for clarification. Section 6.2 introduced 1) why we need to learn the distribution of $\beta$ (sampler $\xi$), 2) the training method, and 3) the technique for sampling $\beta$ from $\xi$. Then, in Section 7, we presented the practical implementation method, which seems to be pointed out by the reviewer. As mentioned in lines 230-231, $\beta[j]$ is sampled within the range $[\beta[j-1], C_\mathrm{max}/(1-\gamma)]$. To implement this, distributions with finite intervals, such as the beta or truncated normal distribution, can be used. However, defining the interval of the distribution with $\beta$ complicates the gradient computation graph in PyTorch, increasing the likelihood of computational errors. To address this, we decided to sample the difference of $\beta$, which is motivated from the fact that $\beta[i] \sim [\beta[i-1], C_\mathrm{max}/(1-\gamma)]$ is equivalent to $\beta[i] = \beta[i-1] + \Delta$, where $\Delta \sim [0, C_\mathrm{max}/(1-\gamma) - \beta[i-1]]$. To further remove $\beta$ in the interval, we instead sampled $\Delta$ within $[0, C_\mathrm{max}/(1-\gamma)]$, but it can result in $\beta[i] > C_\mathrm{max}/(1-\gamma)$. Nevertheless, if $\beta$ increases, the value of $J_C(\pi; \beta)$ also increases due to the conjugate function in (2), resulting in reducing $J_R(\pi)$. As a result, the distribution of $\beta$ is trained to output low probabilities for such cases. We will add this explanation to the appendix.

Q3 (loss function):

First, we want to clarify the meaning of line 235, which seems to be pointed out by the reviewer. The meaning is that the process of solving the outer problem is novel rather than claiming that the proposed loss function itself is novel. Specifically, the proposed method enables the inner and outer problems to be solved concurrently rather than sequentially. Thus, we will change the phrase to "novel update process."

Next, the rationale for structuring the loss indicated in lines 236 and 240 is as follows: The goal of the sampler is to produce a high probability for the optimal $\beta$. To achieve this, we need an indicator that can provide feedback on how well the RCRL problem is solved for a given $\beta$. Being inspired by the paper [R1], which converts constrained RL to unconstrained RL using penalty functions, we defined the indicator as shown in line 236.

Additionally, as observed in the proof of Theorem 6.3, any function $J$ that satisfies \max\_\pi J(\pi; \beta)= \max\_{\pi \in \{ \pi|J_{C_i}(\pi;\beta) \leq d_i }} J_R(\pi) can be used. One such $J$ could be $J(\pi;\beta) = J_R(\pi)$ if $J_{C_i}(\pi;\beta)\leq d_i$ for $\forall i$; otherwise, $-R_\mathrm{max}/(1-\gamma)$. Alternatively, instead of providing feedback, a method that uses the policy's value function to compute the optimal distribution in a closed form could be proposed. As mentioned earlier, since we have proposed a new approach to solving the outer problem, it does not seem necessary to propose a more general loss function. Developing more efficient and reliable loss functions could be done in future work.

References

• [R2] Zhang, Linrui, et al. "Penalized proximal policy optimization for safe reinforcement learning." arXiv preprint arXiv:2205.11814 (2022).

Thanks for the clarification. I have read the paper and response again, and I am happy to raise the score accordingly.

Dear Reviewer,

Thanks for the positive review and valuable feedback on our paper. We appreciate the effort in reviewing our work. Below, we respond to each of the points mentioned.

Weakness 1 (line 88 and 168):

$F\_X$ is the cumulative density function (CDF) of the random variable $X$. We will add the definition of $F_X$ to the paper after line 88.

The logarithmic operator in eq. (9) comes from the following derivation of the policy gradient of the sub-risk measure (by differentiating $\pi'$ in eq. (8)): $\nabla\_\theta \mathcal{R}\_\sigma^g(G^{\pi_\theta}) = \nabla\_\theta\mathbb{E}\_{d\_\rho^{\pi\_\theta},\pi\_\theta}[A\_g^\pi(s,a)]/(1-\gamma) = \mathbb{E}\_{d\_\rho^{\pi\_\theta}}[\sum_a \nabla_\theta \pi_\theta(a|s) A_g^\pi(s,a)]/(1-\gamma) = \mathbb{E}\_{d\_\rho^{\pi\_\theta}, \pi_\theta}[\nabla_\theta \log \pi_\theta(a|s) A_g^\pi(s,a)]/(1-\gamma).$

Weakness 2 (conflicting notations):

Thanks for the comments on the conflict notations. We will change the notations as follows:

1. learning rate: from $\alpha$ to $\omega$.
2. Feasibility function: from $\mathcal{F}$ to $\chi$.
3. Iterator of the discretization in Theorem 6.2: from $i$ to $j$.

We will also correct the typo too.

As the reviewer DPN9 pointed out, the proof of the convergence rate provided in the general response above has an error: "$J_R(\pi^*) - J_R(\pi_t) \geq 0$ for $t \in \mathcal{N}$" does not hold. We apologize for the confusion. We modify the proof as follows, starting from the statement "Also, due to $J_R(\pi^*) - J_R(\pi_t) \geq 0$ for $t \in \mathcal{N}$" in the original proof above.

Given that $D + \frac{2T\alpha^2K^2(\alpha + 1)^2}{(1-\gamma)^5} + \frac{2T\alpha^2K}{(1-\gamma)} \leq \frac{\alpha\eta T}{2}$, one of the following must hold: (1) $|\mathcal{N}| \geq \frac{T}{2}$ or (2) $\sum_{t \in \mathcal{N}}(J_R(\pi^*) - J_R(\pi_t)) \leq 0$.
Assume neither condition is satisfied. Then, $\frac{\alpha\eta T}{2} < (T - |\mathcal{N}|)\alpha\eta = \sum_{t \notin \mathcal{N}}\alpha\eta < \sum_{t \in \mathcal{N}}\alpha(J_R(\pi^*) - J_R(\pi_t)) + \sum_{t \notin \mathcal{N}}\alpha\eta \leq D + \frac{2T\alpha^2K^2(\alpha + 1)^2}{(1-\gamma)^5} + \frac{2T\alpha^2K}{(1-\gamma)} \leq \frac{\alpha\eta T}{2}$, which leads to a contradiction. Therefore, one of the two conditions must hold.
Furthermore, if we assume $|\mathcal{N}| = 0$, it follows that $\alpha\eta T \leq \frac{\alpha\eta T}{2}$, which is also a contradiction. Hence, $|\mathcal{N}| \neq 0$.

If $|\mathcal{N}| \geq T/2$, $J_R(\pi^*) - \mathbb{E}\_{t \sim \mathcal{N}}[J_R(\pi_t)] = \frac{\sum_{t \in \mathcal{N}}(J_R(\pi^*) - J_R(\pi_t))}{|\mathcal{N}|}$ $\leq \frac{2(D + \frac{2T\alpha^2K^2(\alpha + 1)^2}{(1-\gamma)^5} + \frac{2T\alpha^2K}{(1-\gamma)})}{\alpha T} \leq \frac{2D + 20K^2}{(1-\gamma)^{2.5}\sqrt{T}}$, and this inequality also holds for $\sum_{t \in \mathcal{N}}(J_R(\pi^*) - J_R(\pi_t)) \leq 0$.
Consequently, $J_R(\pi^*) - \mathbb{E}\_{t \sim \mathcal{N}}[J_R(\pi_t)]\leq\frac{2D + 20K^2}{(1-\gamma)^{2.5}\sqrt{T}}$.
Also, $\mathbb{E}\_{t \sim \mathcal{N}}[J{C_i}(\pi_t)] - d_i \leq \eta = \frac{2D + 20K^2}{(1-\gamma)^{2.5}\sqrt{T}}$.