Robust Transfer of Safety-Constrained Reinforcement Learning Agents

We would like thank reviewer LcYf for the insightful feedback, which has greatly helped us improve the paper.

• (Q)uestion1. What is the definition of $\omega$?

$\omega$ is a probability measure on the Borel sets of $S$, referred to as the "weighting function", which we need for a well-defined abstracted task (Li et al., 2006). In practice, this is usually a uniform distribution over the state-space, but indeed it is not uniquely defined and can thus have other shapes.

We will include a discussion in the updated version of the paper.

• Lihong Li, Thomas J. Walsh, and Michael L. Littman. Towards a unified theory of state abstraction for MDPs. In AI&M, 2006.

• (W)eakness2 and Q2. Is it possible to compare the proposed method with Yang et al. (2023)?

Yes, Yang et al. (2023) is the state-of-the-art, which yields non-robust guides. The comparison between non-robust guides and our (robust) guides is shown in Fig 7. Our paper would benefit from making this clear.

• W1. In Related work section, I think there is a missing citations: Zhang, Jesse, et al. "Cautious adaptation for reinforcement learning in safety-critical settings." International Conference on Machine Learning. PMLR, 2020.

We thank the reviewer for this reference and will of course include the citation together with an appropriate discussion.

• W3. Given the source code is not attached, reproducibility is low.

Please refer to our global response General Comment 5.

We hope this clarifies the reviewer's questions. We are happy to follow up if something is not clear.

I consider that the current version presents empirical results much more clearly than the initial version. I changed the score accordingly and now slightly lean on acceptance.

• W7. The safety jump-start metric $J(\pi^\diamond, M^\odot) = \mathbb{E} [\frac{c(\tau) - c(\tau')}{c(\tau')}]$ masks potentially dangerous temporal patterns through trajectory averaging.

Thanks for bringing this up. We would like to point out that the notion of safety in the constrained RL already suffers with this issue. To mitigate such situations we would need to consider the worst-case trajectories, for instance by taking into account the CVaR (Yang et al 2021).

• Yang, Q., Simão, T. D., Tindemans, S. H., and Spaan, M. T. J. (2021). WCSAC: Worst-case soft actor critic for safety-constrained reinforcement learning. AAAI, 10639–10646. https://doi.org/10.1609/aaai.v35i12.17272

• W8. The environments tested share similar underlying physics.

Please refer to our global response General Comment 4 for clarifications about this point.

• W9. The paper's treatment of model parameterization through simple mass and friction variations inadequately captures the complexity of real-world dynamic variations.

While we agree that the target environments may not be sufficient for real-world dynamics variations, we would like to argue that such environments are already sufficient to show that a naive approach would fail to perform a safe transfer. Our work shows that it is possible to robustify a safe policy and safely transfer it to new environments. We believe this will spark the study of more realistic target environments.

• W10. The Lagrangian multiplier approach for policy imitation lacks theoretical convergence guarantees when combined with robustification techniques, potentially leading to unstable learning dynamics.

Only the student is trained with the Lagrangian policy imitation approach, while only the guide is trained with robustification techniques. Despite that, it is true that we do not provide theoretical convergence guarantees, but instead have empirical validation, which aligns with the experiments by (Yang et al., 2023) that display convergence in practical settings.

We hope this clarifies the reviewer's questions. We are happy to follow up if something is not clear.

We sincerely thank reviewer Vwu8 for the thoughtful and constructive feedback. Your insights into both the theoretical aspects and the empirical validation of our work have significantly helped us improve the quality of the paper.

• (W)eakness1. $\sigma$ being continuous and bounded might become problematic for discontinuous and hybrid systems.

That is indeed correct. We will omit the mention of continuity in the revised version, as it is sufficient for $\sigma$ to preserve boundedness for the theorem to hold. Moreover, the theorem remains valid even for discrete systems, provided the state space is a metric space. However, our focus is on continuous systems, as our method is specifically designed with continuous dynamics in mind.

• (Q)uestion1. About the construction of $\sigma$ in practice for high-dimensional state spaces.

Please refer to our global response General Comment 1 for clarifications about this point.

• Q2. How is the bound $\varepsilon$ estimated in practice?

Please refer to our global response General Comment 3 for clarifications about this point.

• Q3 and W6. The discretization of the uncertainty set using only $N = 8$ values may be insufficient. Could you justify this choice?

To determine $N$, we carried out a quantitative and qualitative analysis on the relationship between the estimated worst-case dynamics and $N$, and found that $N=8$ is sufficient. This is reasonable as our uncertainty sets are relatively simple. Settings with more complex uncertainty sets and cost functions would require finer discretization.

• W2. The bounded uncertainty set assumption $\exists \varepsilon \in \mathbb{R}, \forall P, P' \in U^\odot, |P - P'| \leq \varepsilon$ fails to capture fat-tailed uncertainty distributions common in real systems.

The restriction that $U$ is bounded indeed disallows fat-tailed uncertainty sets. However, we would like to highlight that

1. Similar assumptions are relatively common in this kind of setting. For example, Eysenbach and Levine (2022) show that robustness achieved through entropy maximization results in a policy that is safe within a ball of radius $\epsilon > 0$ around the nominal dynamics.
2. Our method can still be used to obtain robust policies in settings where $U$ is unbounded, even if some safety guarantees do not apply.
3. It is challenging to provide safety guarantees without any assumptions on the shape of $U$, as an unbounded $U$ would allow tasks that are infinitely different from each other.

• Eysenbach, B. and Levine, S. (2022). Maximum entropy RL (provably) solves some robust RL problems. In ICLR. OpenReview.net.

• W3. The robustification approach through entropy maximization $r^\diamond(s, a) = r^b(s, a) + \alpha r^H(s, a)$ lacks guarantees about value function Lipschitz constants, potentially compromising stability under state-space perturbations.

We would like to ask the reviewer to clarify what they mean by stability under state-space perturbations and how that relates to our approach.

Our understanding is that entropy maximization indeed lacks guarantees regarding the Lipschitz continuity of the value function. However, this is an issue with the technique of entropy maximization itself, while 1. entropy maximization is a well-established technique in the area of robust RL (Moos et al. 2022) and 2. our method does not solely consist of entropy maximization, as this is just one of the three robustification techniques that we examined.

• Moos, J., Hansel, K., Abdulsamad, H., Stark, S., Clever, D., and Peters, J. (2022). Robust reinforcement learning: A review of foundations and recent advances. Mach. Learn. Knowl. Extr., 4(1):276–315.

• W4. The control switching mechanism $\pi(s) = (\pi^\diamond \circ \sigma)(s)$ if $c > 0$ else $\pi^\odot(s)$ introduces potential chattering issues near constraint boundaries

The control switch approach changes control from the student to the guide at most once per episode. We will make this clear in the paper.

• W5. The p-tail metric $M^{\leq p}(M^\odot, \pi) = \int_B M(M^\odot_P, \pi) dP$ requires additional measure-theoretic assumptions that remain unjustified.

The current definition in the paper is somewhat clunky, and we will update it in the next version. Nevertheless, the $p$-tail does not introduce new assumptions.

Let's fix $p = 0.1$ for simplicity. Then, this definition simply describes the mean of the top 10% highest costs. In theory, it's the integral as defined in the paper, while in practice, it is a simple statistic.

• Q3. Is there any reason why the students are limited to off-policy RL algorithms: SAC and DDPG? For deterministic policy, why is DDPG tested over TD3?

We test off-policy methods as 1. it allows training the student with the samples collected from the guide 2. they are particularly suitable for safety-critical applications due to sample efficiency. Moreover, we specifically use DDPG and SAC as prior work has succeeded with these methods (Tessler et al., 2019; Yang et al., 2023).

• Tessler, C., Efroni, Y., and Mannor, S. (2019). Action robust reinforcement learning and applications in continuous control. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 6215–6224. PMLR.
• Yang, Q., Simao, T. D., Jansen, N., Tindemans, S. H., and Spaan, M. T. J. (2023). Reinforcement learning by guided safe exploration. In ECAI, volume 372 of Frontiers in Artificial Intelligence and Applications, pages 2858–2865. IOS Press.

• Q4. In Appendix F (Safety Jump-Start Heatmaps), non-deterministic policy seems to be less robust than deterministic policy. Could you discuss why? This seems counterintuitive to me, especially for a CMDP task.

We believe that this is due to the "key insight" in section 4.2: Since the nondeterministic guide is more erratic, it cannot recover from the unsafe state as effectively as the deterministic guides.

We hope this clarifies the reviewer's questions. We are happy to follow up if something is not clear.

First, we would like to thank reviewer LbED for the insightful comments and openness for further discussion. Your comments have helped us improve the overall quality and clarity of the paper.

Following the reviewer's (Q)uestion1, we start addressing the weaknesses pointed out by the reviewer and then address the remaining questions.

• (W)eakness1. The paper does not assume the uncertainty set is known. However, the uncertainty set is assumed to be bounded by $\delta$.

Boundedness helps us provide guarantees, as it does not allow dynamics functions within the uncertainty set to be infinitely different from each other. While the uncertainty set is bounded by some $\delta$ by assumption, it is crucial to note that we do not know this value during training. Theorem 1 serves as a "sanity check", to verify that there exists a ball of radius $\delta$ such that if the agent is robustly safe, then it will also be safe after the transfer. It shows that there is a good reason to compute a robust policy, that is, to increase $\delta$. Lastly, similar assumptions are relatively common in this kind of setting. For example, Eysenbach and Levine (2022) show that robustness achieved through entropy maximization results in a policy that is safe within a ball of radius $\epsilon > 0$ around the nominal dynamics.

• Eysenbach, B. and Levine, S. (2022). Maximum entropy RL (provably) solves some robust RL problems. In ICLR. OpenReview.net.

• W2 and W3. The parameter $\delta$ presented in section 4 does not appear later on. Connection between the level of noise/perturbations ($\alpha$) such that sufficient $\delta$ can be reached.

This is a fair criticism; thank you for bringing it up. Please refer to our global response General Comment 3 for clarifications about this point.

• W4. The robust training framework depends on the abstraction function $\sigma$. I am not too sure if this abstraction function can be specified easily in practice.

Please refer to our global response General Comment 1 for clarifications about this point.

• W5. State abstraction function for exploration requires substantial feature engineering effort and exploration bonus is limmited

We agree with the reviewer regarding the feature engineering for exploration. The current reward bonus based on the distance of a subset of the features only induces a limited degree of exploration. The design of more sophisticated model-free exploration to increase the coverage of the state space is still an open question. As this is outside our work's scope, we decided to use the same strategy as Yang et al. (2023).

• W6. Lagrangian multiplier $\lambda$ is used to weigh similarity between two policies. How would this enable the "student's policy to imitate the guide's whenever the cumulative cost is above the safety threshold".

The Lagrangian multiplier $\lambda$ results from the Lagrangian relaxation of the CMDP objective (Altman et al., 2019). In practice, we fit $\lambda$ via gradient descent. If the student's cost is high, $\lambda$ will increase to incentivize the student to accumulate less cost.

We can then use the same weight $\lambda$ in the strength of the policy distillation. In other words, if the student violates the safety constraints, we incentivize them to imitate the guide. For a more detailed explanation, we refer to Yang et al. (2023).

• W7. The safety gymnasium task analyzed in empirical analysis seems to be only Goal task. Having more tasks may strengthen the paper in this aspect.

Please refer to our global response General Comment 4 for clarifications about this point.

• Q2. Is there any challenge in getting the expected costs from multiple batches?

We chose to provide an empirical evaluation of the different guides and vary the type of perturbation, noise level, and environment. Evaluating each of these agents in a set of target environments to test their robustness further increases the computational costs. In summary, training and evaluating so many configurations already had a high computational cost, and running multiple repetitions was simply infeasible.

I thank the authors for the detailed response and revised manuscript. With most of the review comments clarified, I'm willing to increase my score (note: score has already been updated in my review).

With that said, I still think that (1) expanding the experimented task beyond Goal task and (2) explain why TD3 is preferred over DDPG in experiments, especially when TD3 is a newer (and more stable) off-policy deterministic policy algorithm. I wish these can be addressed in the camera-ready version.

We are grateful to reviewer YHLY for their detailed and thoughtful feedback, which has provided us with new perspectives and valuable guidance for improving the paper.

• (W)eakness1 and (Q)uestion1. The assumptions are not as mild as claimed, and there is a hidden assumption that exists a robustly safe policy $\pi$, meaning $\pi$ is safe for all $P$ with $|P - P^\diamond| \leq \delta$.

The reviewer brings up a relevant concern. Please refer to our global response General Comment 2 for clarifications about this point.

• W2 and Q4. What exactly are the safety criteria are you considering? $Q_c$ contraints are much more restrictive than the $C(\pi)$ constraints, what exactly are the safety criteria? Also, the norm and abstract operator are defined sloppily.

Since most algorithms apply the constraints on $Q_c$ in practice, we also developed our analysis based on $Q_c$. This indeed implies that the constraints on the expected cost are also satisfied.

Our theoretical results are independent of the definition of the norm, and one option to measure the distance between two dynamics functions is KL divergence. We will update the paper accordingly. Furthermore, we noticed that the abstract operator is defined late in the section; to fix this problem, we will introduce it earlier.

• W3. The authors should consider other perturbation-based algorithms, such as (Goyal and Grand-Clement, 2023).

Thanks for pointing this out. We will add the discussion to the related work section.

Overall, planning in robust MDP attempts to find an optimal policy under the worst-case transition of the uncertainty set (Wiesemann et al. 2013), for instance, based on perturbations of the transition function (Goyal and Grand-Clement, 2023). Since we do not have access to the model, we rely on perturbing the input and output of the agent (Moos et al., 2022). In this paper, we consider perturbing the output (action and policy) of the agent and leave the study of perturbing the inputs (observation, reward, or cost) for future work.

• Moos, J., Hansel, K., Abdulsamad, H., Stark, S., Clever, D., and Peters, J. (2022). Robust reinforcement learning: A review of foundations and recent advances. Mach. Learn. Knowl. Extr., 4(1), 276–315. https://doi.org/10.3390/make4010013
• Goyal, V., and Grand-Clément, J. (2023). Robust Markov decision processes: Beyond rectangularity. Math. Oper. Res., 48(1), 203–226.
• Wiesemann, W., Kuhn, D., and Rustem, B. (2013). Robust Markov decision processes. Mathematics of Operations Research, 38(1), 153–183.

• W4. The abstraction needs to be safety-irrelevant, but sometimes all the observations are related to safety, and sometimes one does not know which observation is safety-irrelevant.

Please refer to our global response General Comment 1 for clarifications about this point.

• Q2. Is $P$ a matrix or a nonlinear transition operator? is the state and action space finite or continuous?

$P$ is non-linear, and the state and action spaces are continuous.

• Q3. Function $u$ maps a mass and friction pair to the corresponding transition dynamics function. Why is this mapping continuous and bounded? This is usually discontinuous.

This function does not need to be continuous, and we will omit the mention of continuity from the paper. The function preserves boundedness, that is, for any bounded $S \subseteq \mathbb{R}^2$, its image under $u$ is also bounded. This implies that $U^\odot$ is bounded in the empirical evaluation, as the mass and friction multipliers range from 0.5 to 1.5.

• Q5. How is superscript * defined (line 244)?

It is the Kleene closure in this context: $(S \times A)^* = \bigcup_{n=0}^\infty (S \times A)^n$. We will make this clear in the paper.

We hope this clarifies the reviewer's questions. We are happy to follow up if something is not clear.