Safe Meta-Reinforcement Learning via Dual-Method-Based Policy Adaptation: Near-Optimality and Anytime Safety Guarantee

We are grateful and indebted for the reviewer's time and effort invested to evaluate our manuscript, and for all the suggestions and reference recommendations to make our manuscript a better and stronger contribution. We answer the weaknesses and questions as follows.

Weakness 1. Safe RL studies should not assume ergodicity and thus actually consider "reachability" or "returnability" as in (Turchetta, 2016) and (Wachi, 2020). The authors try to propose an algorithm with almost surely. Hence, I feel Assumption 2 is incompatible with the nature of the proposed algorithm.

Answer: Papers (Turchetta, 2016) and (Wachi, 2020) study a safe RL scenario, which aims to develop a policy such that any state $s_t$ on the trajectory is safe, i.e., the cost $c(s_t)$ is small than a constant for any timestep $t$. In these two papers, the algorithms with the almost surely property are proposed, i.e., the probability of visiting an unsafe state is smaller than $\epsilon$.

This manuscript, including the algorithm design and the theoretical results (shown in Section 5 in Theorem 1), studies the constrained MDP where the safety is defined as that the expected accumulated costs $J\_{c\_i,\tau}(\pi)=\mathbb{E}\_\pi [\sum\_{t} \gamma^t c(s\_t)]$ are smaller than a threshold. Therefore, it allows for a safe policy to visit some unsafe states (the states with high costs), as long as the expectation of the accumulated costs along many trajectories is smaller than a threshold. Therefore, we are not trying to propose an algorithm with the almost surely property. As a result, this manuscript does not require some assumptions in papers (Turchetta, 2016) and (Wachi, 2020), such as reachability and the regularity of the constraint function $c(s)$. Moreover, as the proposed method is a model-free method, i.e., the knowledge about the cost $c(s)$ on state $s$ cannot be obtained until the state $s$ is visited, we require the assumption of the state visitation. Therefore, the proposed algorithm is compatible with our assumptions.

Weakness 2. The weaknesses related to the validness of Assumption 2 and Remarks 1 and 2.

Answer: We agree that the ergodicity may be a strict assumption in the safe RL problem and Assumption 2 may require a large $B$. Thanks for your comments and suggestions.

In the revised manuscript, we relax Assumption 2 to be compatible with the safe RL setting. Following your suggestion, the relaxed assumption only assumes the safe policy has sufficient visitation to a subset of safe space. Here is the relaxed assumption.

Relaxed assumption:

There exists a set of states $\mathcal{S}^{v} \subseteq \mathcal{S}$ and a constant $\eta > 0$ such that, for any task $\tau \in \Gamma$ and any safe policy $\pi^{s} \in$ { $\pi \in \Pi: J\_{c_i,\tau}(\pi) \leq d_i + \delta_{max}, \forall i = 1, \cdots, p$ }, $\nu^{\pi^s}_{\tau}(s) \geq \eta$ for all $s \in \mathcal{S}^{v}$.

The new assumption supposes that there exists a set of states $\mathcal{S}^{v}$ such that the safe policy can take sufficient visitation in the set.

Under the new assumption, we rewrite the proofs for Section 5, including Lemma 1, Proposition 4, Corollary 1, and Theorem 1. Overall, with the new Assumption 2, the theoretical results remain unchanged except for some constants. In the proofs for the theoretical results in Section 5, Assumption 2 and the constant $B$ in Assumption 2 (in the old manuscript) are only used when proving Proposition 4, and the proofs of all remaining theoretical results are built on Proposition 4. Therefore, in the revised manuscript, based on the new assumption, we build a new Proposition 4, which uses the constant $\eta$ and $\alpha$ (in the new assumption) to replace the constant $B$. Then, we almost keep other proofs unchanged.

Thanks for the suggestion again. Please refer to Section 5 and Appendix F.4 in the revised manuscript for the details of the theoretical results and proofs.

Question 2. Is it possible to relax Assumption 2 while maintaining the claims in Theorems?

Answer: Yes. As stated in the answer to Weakness 2, we relax Assumption 2 and maintain the theoretical results unchanged except for some constants.

First of all, please note that this is the discussion phase. I do not think it is appropriate for authors to focus on only the reviewers' mistakes while ignoring the paper's weaknesses. In the first rebuttal, it was unclear whether the authors admitted their misunderstandings on ergodicity. I believe it is better to accept mistakes and discuss constructively to improve the paper.

Ergodicity. It was confirmed that the errors have been corrected in the revised version of the paper.

Experimental settings. I am not talking about hyperparameter $\lambda$. I have been discussing how the dataset was collected. The current implementation is based on the old assumptions, and unsafe policies collected a large amount of unsafe trajectories with sufficient coverage. This is not consistent with the current theoretical results.

Regarding the comments by Reviewer 9TCU. The degree of novelty may elicit different opinions from different people, but I understand the perspective of Reviewer 9TCU. The proposed algorithm can be seen as an incremental extension of CRPO.

Conclusion. Again, this paper has suffered from initial critical errors resulting from unreasonable assumptions, which previously led to unreasonable theoretical results and now result in inconsistent empirical analyses. I do not believe this paper can be ready for publication during this discussion phase. Therefore, I will maintain my original rating of 3: Reject considering that this paper should be evaluated in a different review cycle.

Question 1. Line 69 "Both meta-CRPO and meta-CPO provide positive upper bounds of the constraint violation". Do the authors mean that meta-CRPO and meta-CPO cannot satisfy the constraint that the sum of $c_i$ is less than $d_i$?

Answer: Yes. In meta-CRPO and meta-CPO, the constraint violation converges to zero as the number of policy optimization steps becomes sufficiently large or when the KL divergence between the initial policy and the adapted policy is sufficiently small. Consequently, the $\sum_t c_i(t) - d_i$ tends to zero. However, there is no guarantee that it is always smaller than zero.

Question 2. Line 129: There is a sum over $a^{\prime} \in \mathcal{A}$ when defining the softmax policy. Does it mean that the paper only considers discrete action space?

Answer: Thanks for your question. In this manuscript, the action space $\mathcal{A}$ could be either discrete or continuous. The state space $\mathcal{S}$ could be either a discrete space or a bounded continuous space. In the revised manuscript, we have clarified it and modified the definitions, theorems, and proofs that are not compatible with it, including the definition of the softmax policy.

In the experiments, all the environments have continuous state space and action space.

Question 3. Line 137: Should the $J_{c_i, \tau}(\pi)$ be $J_{c_i}(\pi)$ ?

Answer: Yes, thanks for pointing it out.

Question 4. Line 192: it is said that setting $\delta_{c_i}=0$ for all $i$ is too strict, and to alleviate the issue, the paper set $\delta_{c_i}=0$. However, in the hyperparameters provided in Table 2, why $\delta_{c_i}=0$ is set to in all environments?

Answer: Thanks for the question. The statement should be modified to "When the requirement of the constraint satisfaction is not strict, setting $\delta_{c_i}=0$ for all $i$ in problem (1) may overly restrict the policy update step. To enhance the algorithm’s flexibility, we set $\delta_{c_i} \geq 0$ as an allowable constraint violation in problem (1). "

In the experiments, we aim to verify the anytime safe property of the proposed method. Therefore, we set $\delta_{c_i}=0$ such that $J_{c_i,\tau}(\pi) \leq d_{i,\tau}$ always holds for the adapted policies $\pi$.

To compare the experimental results under different values of $\delta_{c_i}$, in the revised manuscript (Appendix D.4 and Figure 6), we add the experiments on two environments, including Half-cheetah and Car-Circle-Hazard.

Question 5. Line 210: "Safety cannot be guaranteed in each step". What is the definition of "step" here?

Answer: The policy optimization algorithm, such as (1), requires the data collection by a single policy, i.e., the initial policy $\pi_\phi$, and produces the adapted policy $\mathcal{A}^{s}(\pi_\phi, \Lambda, \Delta, \tau)$. This is one step of the policy adaptation.

In the manuscript, we consider the anytime property of the policy optimization algorithm, i.e., any policy used to explore the environment should be safe. This anytime property requires that each step of the policy optimization algorithm should be safe, because the output policy from each step of the policy optimization algorithm needs to be safe.

We have included the above definition of "step" in the revised manuscript (Section 3.1, lines 187-189).

We are grateful and indebted for the reviewer's time and effort invested to evaluate our manuscript, and for all the suggestions and reference recommendations to make our manuscript a better and stronger contribution. We answer the weaknesses and questions as follows.

Weakness 1.1. While the proposed algorithm is strong, no intuition is provided when introducing the algorithm structure, making the flow of the paper not smooth.

Answer: Thanks for the comments. In the revised manuscript, we add several explanations after the proposed algorithms are introduced. Please refer to the revised manuscript for the details and the added sentences are highlighted in red color.

For example, after the safe policy optimization algorithm (1), we add the intuition of the algorithm design: "The safe policy adaptation $\mathcal{A}^{s}$ in problem (1) is inspired by the derivation of CPO, where both problem (1) and CPO aim to approximate the original safe RL problem. Specifically, the objective and constraint functions of problem (1) serve as upper bounds of the true objective and constraint functions $J_\tau(\pi)$ and $J_{c_i,\tau}(\pi)$ of the safe RL problem."

Weakness 1.2. The connection between the proposed approach and Meta-CRPO and Meta-CPO is not clear. Although this is briefly discussed in the appendix, I suggest connecting the proposed method with the previous approaches also when introducing the new algorithm. This makes the readers much easier to understand what the key difference in the algorithm is that makes the proposed method perform better.

Answer: Thanks for the suggestions. In the revised manuscript, we add the discussion that connects the proposed method with Meta-CRPO and Meta-CPO, in Section 3.1 and Section 4.1. Please refer to the revised manuscript for the details and the added sentences are highlighted in red color.

Weakness 2. The experimental results can be improved. The influence of the hyperparameters is not discussed or tested, and how to select the hyperparameters is unclear.

Answer: Thanks for the suggestions. In the revised manuscript (Appendix D.4 and Figure 6), we test the experimental results with different hyper-parameter settings, i.e. different allowable constraint violation constant $\delta_{c_i}$, on two environments, including Half-cheetah and Car-Circle-Hazard.

Moreover, we also include guidance about how to choose the hyper-parameters, including $\delta_{c_i}$, $\lambda$, and $\lambda_{c_i}$ in Appendix D.4 of the revised manuscript. The guidance is presented as follows.

Guidance of selecting $\delta_{c_i}$: As indicated in both theoretical results in Section 5.2 and the experimental results in Figure 6, we choose a large $\delta_{c_i}$ when the constraint satisfaction is not required to be strict, and a small $\delta_{c_i} \rightarrow 0$ when the constraint satisfaction is prioritized.

Guidance of selecting $\lambda$ and $\lambda_{c_i}$: We set $\lambda=\lambda_{c_i}$ and tune them such that, the KL divergence of initial policy $\pi$ and the adapted policy $\pi^\prime$ solved from the safe policy adaptation problem (1) is close to $0.03$. If the KL divergence is too large, the objective and constraint functions of problem (1) are not good approximations of the accumulated reward/cost functions, as indicated by Lemma 1. If the KL divergence is too small, the policy adaptation step of problem (1) is too small.

Thank you for the detailed response and clarifications! The presentation and the rigor of the paper have been improved. I have no further questions.

Thanks very much for your time and effort in reviewing our work. Thanks for your suggestions to make our manuscript better. We answer the weaknesses and questions as follows.

Weakness 1. The experiments portion of this paper is relatively short (esp in the main paper). I think including other experiments would further improve the paper. For example, try out different values of (allowable constraint violation) and observe the trade-off between reward and safety.

Answer: Thanks for the suggestions. In the revised manuscript (Appendix D.4 and Figure 6), we conduct the experiments with different allowable constraint violation constant $\delta_{c_i}$ on two environments, including Half-cheetah and Car-Circle-Hazard. The experiment shows the trade-off between reward and safety.

Weakness 2. The allowable constraint violation $\delta_{c_i} seems like a hyper-parameter and I would appreciate further guidance on how to determine the appropriate value for a safety-constrained task. Perhaps performing the experiment suggested in item (1) above could help.

Answer: Thanks for the suggestions. As stated in the answer to Weakness 1, we have added the experiments. The following is the guidance on how to choose the hyper-parameter $\delta_{c_i}$.

As indicated in both theoretical results in Section 5.2 and the experimental results in Figure 6, we choose a large $\delta_{c_i}$ when the constraint satisfaction is not required to be strict, and a small $\delta_{c_i} \rightarrow 0$ when the constraint satisfaction is prioritized.

Weakness 3. The paper does point out the inherent shortcoming of CPO being computationally expensive and proposes a dual method for safe policy adaptation. However, the safe policy adaptation problem outlined in Eq. 4 and 5 seems rather similar to Lagrangian-based online safe RL algorithm, e.g. RCPO, PPO-Lagrangian. The authors might want to illustrate how is this dual method particularly novel.

Answer: Eq. (4) and (5) solve the safe policy adaptation problem in (1) by the dual method. RCPO and PPO-Lagrangian solve the safe RL algorithm by the primal-dual method. Although both the proposed dual method for problem (1) and the primal-dual method in RCPO and PPO-Lagrangian are Lagrangian-based safe policy optimization algorithms, they are different. RCPO and PPO-Lagrangian are not suitable for this safe meta-RL problem and are much worse than the proposed method, even worse than CPO.

Eq. (4) and (5) aim to solve the safe policy adaptation problem in (1). As mentioned in Section 3.1, the safe policy adaptation (1) holds several benefits similar to CPO, including the safety guarantee for a single policy optimization step (using data collected on a single policy) and the monotonic improvement. Moreover, we derive the closed-form solution under certain Lagrangian multipliers for the optimization problem (1). Based on the derived closed-form solution of (1) (shown in (3)), we can use the dual method shown in (4)(5) to solve the safe policy adaptation problem in (1), which significantly reduces the computational complexity during the meta-training.

In contrast, RCPO and PPO-Lagrangian do not hold any of the benefits shown in CPO and the proposed algorithm. First, RCPO and PPO-Lagrangian use the gradient ascent steps on the Lagrangian, which do not have the safety guarantee and the monotonic improvement in each policy optimization step, and therefore cannot guarantee anytime safety in the meta-test stage. Moreover, there is no closed-form solution for the policy optimization step in RCPO and PPO-Lagrangian, and therefore cannot be solved by the dual method, which leads the high computational complexity during the meta-training.

Thanks for the question and the literature recommendation. We have included the above discussion in the revised manuscript (Appendix C).

The theoretical contribution of the proposed method over meta-CRPO. In terms of theoretical contribution, the paper is the first to derive a comprehensive theoretical analysis regarding near optimality and anytime safety guarantees for safe meta-RL. The theoretical contribution of this paper over meta-CRPO and meta-CPO is shown in Table 1. First, we establish the theoretical basis of the algorithm design that guarantees anytime safety, i.e., zero constraint violation for any policy used for exploration. Second, we derive a lower bound of the expected accumulated reward of the adapted policies compared to that of the task-specific optimal policies, which shows the near optimality of the proposed safe meta-RL framework. Finally, we demonstrate a trade-off between the optimality bound and constraint violation when the allowable constraint violation varies, which enables the algorithm to be adjusted to prioritize either safety or optimality. In meta-CPRO, the optimality bound is provided, but anytime safety is not guaranteed. Meta-CPO provides neither an optimality bound nor anytime safety.

Weakness 3. The method lacks novelty; based on my understanding, it does not present new contributions, including in the theoretical aspects. It extends primal-dual settings for meta-safe RL, similar to primal meta-safe RL (meta-CRPO).

Answer: The paper does not extend the primal-dual method in meta-CRPO to meta-safe RL. Instead, we design a new policy adaptation algorithm in problem (1) and solve it by the dual method. Although the primal-dual method and the dual method are both Lagrangian-based safe policy optimization algorithms, they are different, and the primal-dual method in meta-CRPO is much worse than the proposed dual method in the meta-safe RL problem.

The differences between the proposed method and meta-CRPO. This paper designs a new policy adaptation algorithm in problem (1) and solves it by the dual method. The proposed algorithm holds (i) a safety guarantee for a single policy optimization step and (ii) a closed-form solution. The proposed policy adaptation algorithm is the first algorithm that simultaneously offers the two key properties. Based on the derived closed-form solution, we use the dual method to solve problem (1). Meta-CRPO uses the CRPO, a primal-dual-based method for policy adaptation. The method does not hold any of the two properties in our proposed methods. In particular, it does not have a closed-form solution and cannot be solved by the dual method. It uses the primal-dual method to solve safe RL, which cannot guarantee safety for a single policy optimization step. In this paper, the meta-policy training algorithm in (2) aims to maximize the expected accumulated reward of the policies adapted from the meta-policy. We derive a Hessian-free approach to optimize the meta-policy. In meta-CRPO, the meta-policy is learned by minimizing the distance between the meta-policy and the task-specific policy, which does not consider the optimality and the safety of the task-specific policies adapted from the meta-policy. In the following, we elaborate on the advantages of the proposed methods over existing meta-safe RL methods, including meta-CPO and meta-CRPO.

The advantages of the proposed method over meta-CRPO. The proposed algorithms offer three key advantages over existing safe meta-RL methods, including meta-CPO and meta-CRPO. (i) Superior optimality. Our safe meta-policy training algorithm in (2) maximizes the expected accumulated reward of the policies adapted from the meta-policy. In contrast, the meta-training of meta-CRPO learns the meta-policy by minimizing the distance between the meta-policy and the task-specific policy, which does not consider the optimality and the safety of the task-specific policies adapted from the meta-policy. (ii) Anytime safety guarantee during the meta-test. The safe meta-policy training produces a safe initial meta-policy by imposing the safety constraint. The safe policy adaptation imposes a constraint on the upper bound of the total cost, and thus is guaranteed to produce a safe policy for each iteration when the initial policy is safe. By integrating these two modules, anytime safety is achieved. In contrast, meta-CRPO employs CRPO, a primal-dual-based method, for policy adaptation, which does not have any safe guarantee in a single policy adaptation step. Thus, anytime safety cannot be guaranteed in meta-CRPO. (iii) High computational efficiency in both the meta-test and meta-training stages. In the meta-test, the derivation of the close-formed solution of Problem (1) makes it efficient to be solved. In contrast, the meta-CRPO and meta-CPO require to solve a constrained optimization problem, which is more computationally expensive than the solution of problem (1). In the meta-training, the close-formed solution of the policy adaptation (1) is used to derive a Hessian-free meta-gradient and reduces the computation complexity of the proposed algorithm to approach that in the single-level optimization. In contrast, the meta-CPRO requires that the task-specific optimal policies have been learned, which is impractical when the number of training tasks is large. The meta-CPO uses the bi-level optimization to learn the meta-policy, which requires the computation of Hessian and Hessian inverse. We conduct experiments on seven scenarios including navigation tasks with collision avoidance and locomotion tasks to verify these advantages of the proposed algorithms.

We are grateful and indebted for the reviewer's time and effort invested to evaluate our manuscript, and for all the suggestions and reference recommendations to make our manuscript a better and stronger contribution. We answer the weaknesses and questions as follows.

Weakness 1. The related work section lacks thorough investigation, e.g., some multi-task/multi-objective safe RL methods; these can be helpful for meta-safe RL.

Answer: Although all of meta-safe RL, multi-task safe RL, and multi-objective safe RL consider multiple tasks in safe RL environments, however, the most important distinction between meta-safe RL and multi-task/multi-objective safe RL is that the agent in meta-safe RL is required to adapt to a new and unknown environment under few-shot data collection. Therefore, the policy adaptation algorithm is the most important part of meta-safe RL. This manuscript designs a novel policy adaptation algorithm in (1) which holds several benefits for the few-shot policy adaptation that the existing methods do not hold. In contrast, the multi-task/multi-objective safe RL learns the policies for multiple tasks during the training stage, where the policy adaptation is not required. Therefore, the multi-task/multi-objective can borrow existing policy optimization methods and do not need to design a new one.

Thanks for the comments. We have included the above discussion in Appendix A of the revised manuscript.

Weakness 2. The paper is not well-written and appears to rely heavily on language models for content generation, e.g. the abstract.

Answer: We certify that we do not use language models to generate any content of the manuscript. We aim to keep the abstract as concise as possible.