Off-Policy Primal-Dual Safe Reinforcement Learning

Q1: Is it possible to extend Equation (5) into a third-order equation, and if so, what impact might this have on performance?

The goal of adding the quadratic penalty term to Equation (5) is to convexify the neighborhood area of a locally optimal solution. More specifically, because the second-order derivative of this term is positive definite, the incorporation of quadratic penalty with a large penalty weight can ensure that the augmented Lagrangian has a positive definite second-order derivative in the vicinity of local optima, thereby promoting the convergence of our algorithm (See Appendix B). In comparison, employing the third-order penalty cannot achieve this goal because its second derivative is not guaranteed to be positive definite.

Still, one might consider using a penalty of a higher order, such as fourth, sixth, etc. However, incorporating such a penalty leads to more hyperparameter tuning, and our experiment results show that we can already obtain a satisfactory performance by tuning the hyperparameter $c$.

Q2: What is the connection between the variance in Q-cost value estimation and the process of policy optimization?

In the expression of UCB (i.e., Eq. (3)), the variance of the $Q\_c$ ensemble serves as an indicator of epistemic uncertainty and ensures that the $\\widehat{Q}\_c^{\\text{UCB}}$ can upper bound the cost value of most state-action pairs. Therefore, it induces conservativeness by shrinking the policy search space (i.e., \\mathbb{E}[Q^\\pi\_c]\\leq \\mathbb{E}[\\widehat{Q}^{\\text{UCB}}\_c] implies \\{\\pi |\\mathbb{E}[\\widehat{Q}^{\\text{UCB}}\_c]\\leq d \\}\\subseteq \\{\\pi |\\mathbb{E}[Q^\\pi\_c]\\leq d \\}), as indicated by the conservative boundary in Figure 1. More specifically, by adding the standard deviation in cost estimation, the policy can be optimized to favor actions that not only result in high reward and constraint satisfaction but also yield low uncertainty in cost estimation. This is because to lower the value of $\\widehat{Q}\_c^{\\text{UCB}}$, the policy have to choose actions that not only have low safety costs estimated by each $\\widehat{Q}\_c^i$ but also lead to low variance among the estimations of different $\\widehat{Q}\_c^i$.

Q3: How does the value of "c" affect performance in terms of both cost minimization and reward maximization?

In theory, the value setting of $c$ controls the degree of convexity and also stability in the neighborhood of local optima. Increasing this value can help stabilizing policy learning, but an overly large $c$ imposes a large penalty on policies exceeding the conservative boundary (where $\\mathbb{E}[\\widehat{Q}^{\\text{UCB}}\_c]>d$), which significantly slows down the process of expanding the conservative boundary as shown in Figure 1. According to the ablation studies in Section 4.2, a larger $c$ generally leads to fewer performance oscillations in terms of both cost and reward, but an overly large $c$ can also hinder reward maximization by having the policy stuck in highly conservative solutions where both the reward and the cost are low.

We thank the reviewer for all the valuable comments. Please refer to the point-to-point responses below.

W1: About the theoretical analysis in terms of convergence and stability in policy optimization.

We agree that this work can benefit from a thorough theoretical analysis in terms of convergence and stability. However, we have to clarify that the main contribution of this work is in the extensive experimental campaign, as also argued by reviewer cCE3. To our best knowledge, CAL is the first off-policy primal-dual method that achieves asymptotic performance comparable to state-of-the-art on-policy methods in such a low data regime. The theoretical part in this paper proves the invariance of the local optima under the ALM modification, and more importantly, explains the stabilization effect of ALM from an energy perspective and thus provides an insight into how the two proposed ingredients interplay and how this benefits primal-dual policy optimization, which are then well-verified by our interpretative experiments (e.g., Figure 3). We believe that all these endeavors can be notable contributions to the safe RL community.

W2: About the on-policy experiments.

Thanks for the suggestion. To clarify, the reasons why we trained CAL using fewer samples than that for on-policy baselines is that the performance of CAL can be seen as near convergence in the low data regime presented in Figure 4. However, directly transferring these curves to Figure 6 may make them look incomplete because the x-axes are altered to a much larger scale since on-policy baselines requires much more samples to converge. Also, it is worth noting that the comparative evaluation against on-policy baselines is sufficient to demonstrate CAL's effectiveness.

Still, we agree that it would be better to present CAL's performance in a higher data regime. Thus, we are currently trying to run CAL as many simulation steps as possible in order to obtain a true asymptotic performance, and we will append this result to our future revision.

Q1: Why do we have to reformulate the augmented Lagrangian?

According to our above clarification about the proof that ALM does not alter the position of local optima, the two formulations are equivalent after we add the missing inner expectation in the reformulated objective. To make the objective more intuitive and easier to understand, in the revised version, we have replaced the objective in Section 3.2 with:
$

\begin{cases} \min\limits_{\lambda\geq 0}\max\limits_{\pi}\mathbb{E}\left[\widehat{Q}_r^\pi\right] - \lambda\left(\mathbb{E}\left[\widehat{Q}_c^{\text{UCB}}\right] - d\right)-\frac{c}{2}\left(\mathbb{E}\left[\widehat{Q}_c^{\text{UCB}}\right] - d\right)^2,&\text{if}\ \frac{\lambda}{c}>d-\mathbb{E}[\widehat{Q}_c^{\text{UCB}}] \\ \min\limits_{\lambda\geq 0}\max\limits_{\pi}\mathbb{E}\left[\widehat{Q}_r^\pi\right] + \frac{\lambda^2}{2c} ,&\text{otherwise}. \end{cases}
$

The reason why we did not use this form in our original manuscript is that we wanted to simplify the objective into one single line. However, as the reviewer suggests, this might cause confusion, so we have modified our manuscript to mitigate this issue.

Q2: About the constraint threshold in Hopper.

In theory, according to our description of how ALM helps addressing the instability and over-conservativeness issue, $c=0$ (without ALM) means that the performance can become instable and over-conservative. Therefore, the fact that "when $c=0$, the test-cost seems to be not that bad in the sense that average is below the threshold" in Hopper does not contradict the above statement since the reward in this case oscillates violently and is overly conservative.

Q3: About the page number for [Luenberger et al., 1984].

Thanks for the suggestion. The part relevant to the derivation of augmented Lagrangian in [Luenberger et al., 1984] is Pages 445-453. We have added the page number to the place where the ALM objective first appears (Section 3.2) in our revision.

Q4: Why does taking minimum over $\\pi$ implies underestimation bias?

We misplaced the min operator in the inequality in Section 3.1, and the true inequality we used is actually$\\mathbb{E}\_\\epsilon [{\\min_\pi}({Q}\_c(s,\\pi(s)) +\\epsilon)]\\leq\\min\_\\pi {Q}\_c(s,\\pi(s))$ for any given $s$. We have corrected this in the revision.

In terms of why this inequality is generally true, there is an intuitive experiment in [Thrun & Schwartz, 1993] (Section 2) that helps illustrate this phenomenon, which is also used to serve as the motivation of addressing the overestimation bias in $Q$ function approximation in the TD3 paper (Section 4 in [Fujimoto et al., 2018]). The key point here is that due to the function approximation noise, some of the Q-values might be too small, while others might be too large. The min operator, however, always picks the smallest value, making it particularly sensitive to underestimations.

Q5: Introduce and rephrase the theorem in A.M. Duguid.

Thanks for the advice. We have introduced the corresponding theorem as a lemma in Appendix B, where we also show how this lemma leads to the fact that there exists a $c^*>0$ such that if $c>c^*$, the damping matrix $A$ is positive definite at the constrained minima.

We thank the reviewer for all the valuable comments. Please refer to the point-to-point responses below.

W: Clarification about the proof that ALM does not alter the position of local optima.

Thank you very much for pointing out this issue, and we clarify that this actually results from a typing error.

Specifically, in our original submission, we mistyped the actual objective

$

\min_{\lambda\geq 0}\max_{\pi}\mathbb{E}_{s\sim \rho_\pi,a\sim\pi(\cdot|s)}\left[\widehat{Q}_r^\pi(s,a)\right] - \frac{1}{2c}\left[\left( \max {0, \lambda-c(d-\mathbb{E}_{s\sim \rho_\pi,a\sim\pi(\cdot|s)}\left[\widehat{Q}_c^{\text{UCB}}(s,a)\right]) } \right)^2-\lambda^2\right]

$

as

$

\min_{\lambda\geq 0}\max_{\pi}\mathbb{E}_{s\sim \rho_\pi,a\sim\pi(\cdot|s)}\left\{\widehat{Q}_r^\pi(s,a) - \frac{1}{2c}\left[\left( \max \{0, \lambda-c(d-\widehat{Q}_c^{\text{UCB}}(s,a) \} \right)^2-\lambda^2\right] \right\}
$

by missing an inner expectation.

Apparently, the actual objective does not require an interchange between the expectation and the max or square operator, and thus does not cause the trouble mentioned by the reviewer in the proof that ALM does not alter the position of local optima.

To see why we actually used the objective with an inner expectation, we can first transform this objective into an equivalent form (i.e., Objective (4a)(4b) in our revision, page 4):

\begin{cases} \min\limits_{\lambda\geq 0}\max\limits_{\pi}\mathbb{E}\left[\widehat{Q}_r^\pi\right] - \lambda\left(\mathbb{E}\left[\widehat{Q}_c^{\text{UCB}}\right] - d\right)-\frac{c}{2}\left(\mathbb{E}\left[\widehat{Q}_c^{\text{UCB}}\right] - d\right)^2,&\text{if}\ \frac{\lambda}{c}>d-\mathbb{E}[\widehat{Q}_c^{\text{UCB}}] \\ \min\limits_{\lambda\geq 0}\max\limits_{\pi}\mathbb{E}\left[\widehat{Q}_r^\pi\right] + \frac{\lambda^2}{2c} ,&\text{otherwise}. \end{cases}

When $\frac{\lambda}{c}\leq d-\mathbb{E}[\widehat{Q}\_c^{\text{UCB}}]$, the objective does not involve $\widehat{Q}\_c^{\text{UCB}}$, which is reasonable since we have $d\geq\mathbb{E}[\widehat{Q}\_c^{\text{UCB}}]$. When $\frac{\lambda}{c}>d-\mathbb{E}[\widehat{Q}\_c^{\text{UCB}}]$, differentiating the objective w.r.t. the policy yields

$

\mathbb{E}\left[\nabla_\pi\widehat{Q}_r^\pi\right]-(\lambda-c(d-\mathbb{E}\left[\widehat{Q}_c^{\text{UCB}}\right])\cdot\mathbb{E}\left[\nabla_\pi\widehat{Q}_c^{\text{UCB}} \right],
$

which is equivalent to

$

\mathbb{E}\left[\nabla_\pi\widehat{Q}_r^\pi-(\lambda-c(d-\mathbb{E}\left[\widehat{Q}_c^{\text{UCB}}\right])\cdot\nabla_\pi\widehat{Q}_c^{\text{UCB}} \right].

$

By replacing $\nabla\_\pi\widehat{Q}\_r^\pi$ with the SAC objective, this gradient corresponds exactly to the policy update in our pseudo code (Line 11 of Algorithm 1 in the original version of our paper) and also to the actual implementation (Lines 163-208 in ./CALsubmission/sac/cal.py in CAL_code.zip which can be found in our supplementary materials).

As for the dual update for $\lambda$, according to this objective, we should perform $\\lambda \\leftarrow \\max\\left\\{0, \\lambda - \\alpha\_\\lambda \\mathbb{E}\_{s,a\\sim\\mathcal{B}} \\left[d-\\widehat{Q}\_c^{\\text{UCB}}(s,a) \\right]\\right\\}$ when $\frac{\lambda}{c}> d-\mathbb{E}[\widehat{Q}\_c^{\text{UCB}}]$, and perform $\lambda\leftarrow \max\\{0,\lambda-\frac{\lambda}{c}\\}$ otherwise. However, empirical results show that exclusively performing the former update rule is sufficient to obtain a satisfactory performance, so we adopt this form in our pseudo code (Line 14 of Algorithm 1 in the original version of our paper) and also our implementation (Lines 221-229 in ./CALsubmission/sac/cal.py).

We have modified corresponding parts in our paper (Section 3.2 and Appendix A). Please refer to our revision and let us know if there is still any confusion.

We thank the reviewer for the valuable comments. Please refer to the point-to-point responses below.

W: About the performance with multiple competing constraints.

We agree that this is an interesting research problem. Theoretically, there is no essential difference when switching to the multi-constraint setting since we can simply replace the scalar $Q\_c$ functions and the dual variable with a vector. However, in practice, depending on how these constraints relate to each other, the difficulty in balancing different constraints in policy optimization may vary.

One practically meaningful constraint we can come up with in MuJoCo tasks is the amount of torque of the agent's joints (a large torque may shorten the lifespan of the motor). Yet, we are not sure if this torque constraint would compete with the speed constraint used in our paper, and we are currently investigating this. To be more specific, we are conducting experiments to determine an appropriate value for this constraint in different tasks and for different joints in a single robot. Also, we are trying to incorporate this constraint with the speed constraint to see how the methods perform in such a setting. We will append these results in our future revision as soon as we have finished this experiment.

Q1: About the cost value ensemble.

How the cost value ensemble in (3) is created?

Specifically, we maintain $E$ bootstrapped $Q\_c$ networks, i.e., $\\{\\widehat{Q}^i\_c \\}\_{i=1}^E$, which share the same architecture and only differ by the initial weights. These weights are initialized by sampling from a normal distribution (The corresponding part in our code: torch.nn.init.xavier_uniform_(m.weight, gain=1)). All the networks are trained on the same dataset.

Do you have multiple (i.e., $E$) parallel parameterized models that each estimate the cost in parallel?

Yes. During cost estimation, each $\\widehat{Q}^i\_c$ estimates the cumulative cost in parallel and independently.

Should the expectation in (3) be replaced by an empirical mean (i.e., $\\frac{1}{E}\\sum\_{i=1}^{E}$)?

Yes, the reviewer is right. It would be more accurate to use an empirical mean, and we have modified this definition in our revision.

Q2: About the statement that UCB "encourages the policy to choose actions with low uncertainty in cost estimation."

Yes, the reviewer is right. Actions favored by the conservative policy optimization are those not only yield low uncertainty in cost estimation but also result in high reward and constraint satisfaction. We have modified the corresponding expression in our revision (Section 3.1).

Q3: About the notation for the hyperparameter $c$ in ALM.

Thanks for the suggestion. We will change the notation to avoid confusion in our future revision, but we have to temporarily maintain this notation in the current revision (available in this webpage) since some questions from other reviewers are related to this notation.

We thank the reviewer for all the valuable comments. Please refer to the point-to-point responses below.

W1: About the form of the upper confidence bound.

Thank you very much for pointing out this inaccuracy. It is true that in linear MDPs where the transition kernel and reward function are assumed to be linear w.r.t. the state-action representation, the upper confidence bound has a specific form that does not correspond to Eq. (3) in our paper. We have corrected this part in our revised version (Section 3.1).

W2: About how the bootstrapped value ensemble is constructed and why we chose this form of uncertainty estimation.

Specifically, we maintain $E$ bootstrapped $Q\_c$ networks, i.e., $\\{\widehat{Q}^i\_c \\}\_{i=1}^E$, which share the same architecture and only differ by the initial weights. From the Bayesian perspective, this ensemble forms an estimation of the posterior distribution of $Q\_c$, with the standard deviation quantifying the epistemic uncertainty induced by a lack of sufficient data. Despite its simplicity, this technique is widely adopted and is proved to be extremely effective in many previous works. We have added the above description into our revised version (Section 3.1).

W3: More space in the main paper for the theory behind ALM.

Thanks for the suggestion. We have re-stated the monotonic energy decrease as a proposition and added a proof sketch of it in the main paper to provide an overall insight of the theory (Section 3.2).

Q: "How many Q networks did you use in the ensemble and how did you choose this hyperparameter?"

As described in Appendix D.3, we set the ensemble size to 4 for MuJoCo tasks and 6 for Safety-Gym tasks. In theory, increasing the number of $Q\_c$ networks will improve the reliability of uncertainty quantification, but at the cost of increasing the computational burden. We choose the two hyperparameters because they strike a good balance between computational complexity and performance (we are running an additional ablation study for the ensemble size, and we will update our revision once we have completed experiments). However, note that we did not fine tune this hyperparameter for each task, so it is likely that a better choice of the ensemble size might exist for many tasks.