Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings

We appreciate the reviewer's valuable comments and suggestions, which helped improve the clarity and readability of the paper. We hope we have answered the reviewer's questions clearly and satisfactorily, and are eager for further discussion.

W2: It is somewhat hard to understand Algorithms (1) and (2). There is a lot of white space in Algorithm (1) and (2), why don't use it to add an overview, explanations, and comments for better understanding?

$**Answer:**$ In the updated paper, more detailed explanations and comments have been added to Algorithm 1. However, Algorithm 2 needs a large space for a detailed overview, explanations, and comments. Due to the paper limit, they are included in Appendix I.

[1] Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023). Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1), 1-118.

[2] Janner, M., Fu, J., Zhang, M., & Levine, S. (2019). When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32.

Q1: How does NN input augmentation help embed what physics knowledge into AC?

$**Answer:**$ NN Input Augmentation, i.e., Algorithm 1, does not directly contribute to embedding physics knowledge into AC. Its function is to augment each NN input such that the augmented input vector can embrace all the non-missing and non-redundant monomials of a Taylor series (given the order of series). One motivation behind the augmentation is that according to Taylor's Theorem, the Taylor series can approximate arbitrary nonlinear functions, with controllable accuracy via controlling series order. This augmentation thus empowers our DNN with the controllable accuracy for approximating action-value function and action policy. The second motivation is our proposed safety-embedded reward $r(s(k), a(k))$ is a typical Taylor series; the action-value and action policy are functions of it. If using the Taylor series to approximate the action-value function and the action policy, we can also discover some hidden knowledge.

Q3: How do algorithms (1)(2) achieve NN input aug and NN editing?

$**Answer:**$ For Algorithm 1 for achieving NN input augmentation, we have added more explanations and motivation to the updated paper. To better understand how Algorithm 1 achieves this and also how it can embrace all the non-missing and non-redundant monomials of a Taylor series, the updated paper includes an illustration example shown in Figure 8 in Appendix F (also available at link 2 (anonymous hosting and browsing)).

To show how Algorithm 2 achieves NN editing, the updated paper includes a flowchart of NN editing in a single PhyN layer, which is shown in Figure 9 in Appendix I.1 (also available at link 3 (anonymous hosting and browsing)). Specifically, given the node-representation vector from Algorithm 1, the raw weight matrix obtained via the gradient descent algorithm is edited via link editing to embed assigned physics knowledge, resulting in an edited matrix: $W_t \leftarrow K_t + U_t = K_t + M_t \odot W_t$. We note here the ${K}_t$, generated in Line 8 and Line 14 of Algorithm 2, includes all the parameters in the knowledge set. While the weight-masking matrix ${M}_t$, generated in Line 9 and Line 15, is used to generate uncertainty matrix ${U}_t$ to include all the parameters excluded from the knowledge set. This is achieved by freezing the parameters of raw ${W}_t$ (from gradient descent algorithm) included in the knowledge set to zeros, through $M_t \odot W_t$.

Further, Figure 10 in Appendix I.2 (also available at link 4 (anonymous hosting and browsing)) depicts the flowchart of NN editing in cascade PhyNs. As an example, we consider actor network, whose ground-truth model is

$\pi\left(s\right) = A_\pi \cdot {\mathfrak{m}}(s, r_\pi) + p(s) . ~~~(\text{e}2)$

We note Lines 5-9 of Algorithm 2 mean that $A_\pi = K_1 + M_1 \odot A_\pi$, leveraging which and the letting $r_1 = r_\pi$, $x = s$, and $y = \pi\left({s}\right)$, the ground-truth model (e2) can be rewritten as

$y = (K_1 + M_1 \odot A_\pi) \cdot \mathfrak{m}(s,r_\pi) + {p}(s) = K_1 \cdot \mathfrak{m}(x, r_\pi) + (M_1 \odot A_\pi) \cdot \mathfrak{m}(s,r_1) + {p}(s). ~~~(\text{e}3)$

We obtain from Line 19 of Algorithm 2 that the output of the first PhyN layer is

$y_1= K_1 \cdot \mathfrak{m}(s, r_1) + a_1 \odot \text{act}(U_1 \cdot {\mathfrak{m}}(s,r_1)). ~~~(\text{e}4)$

Recalling that $K_1$ includes all the entries of $A_\pi$ in knowledge set, while the $U_1$ includes remainders, we conclude from (e3) and (e4) that the available physics knowledge pertaining to the ground-truth model has been embedded to the first PhyN layer. As Figure 10 in Appendix I.2 (also available at link 4 (anonymous hosting and browsing)) shows the knowledge embedded in the first layer shall be passed down to the remaining cascade PhyNs and preserved therein, such that the end-to-end critic and actor network can strictly comply with the physics knowledge. This knowledge passing is achieved by the block matrix $K_p$ generated in Line 14 of Algorithm 2, thanks to which, the output of $t$-th PhyN layer satisfies

$[y]_{1:\text{len}(y)}$

= $K_1 \cdot \mathfrak{m}(s, r_1) + [a_t \odot \text{act}(U_t \cdot \mathfrak{m}(\bar{y}, r_t ))]_{1:\text{len}(y)}, ~~\forall t \in \{2,\ldots,p\})$

where $\bar{y} = y_{t-1}$. Meanwhile, the $U_t = M_t \odot W_t$ means the masking matrix $M_t$ generated in Line 15 of Algorithm 2 is to remove the spurious correlations in the cascade PhyNs, which is depicted by the cutting link operation in Figure 10 in Appendix I.2 (also available at link 4 (anonymous hosting and browsing)).

Dear Reviewer BXs3,

We appreciate your quick reply and positive feedback. The current updated paper is not our final version. We are adding more experiments and explanations to the paper. We will let you know when the final version is ready.

Sincerely,

Authors

Q2: The proposed method is only compared to methods without consideration on safety requirements.

$**A2:**$ Insufficient information in experiments leads to the confusion. All the comparison models, excluding the default proportional-derivative controller, considered the safety requirements. Our Phy-DRL and purely data-driven DRL share the same safety-embedded reward, and our Phy-DRL and sole model-based design share the same model-based action policy: $a_{\text{phy}}(k) = F \cdot s(k),$ which is obtained under safety conditions. Specifically, $F$ is computed according to:

Step 1: $\text{argmin}_{Q,R} ~(\text{logdet}(Q^{-1})), \text{ subject to (7) and (11)}$,

Step 2: $F = R \cdot Q^{-1}$,

where Equations (7) and (11) in the paper include the transformed safety conditions. In addition, during training, the DRL and our Phy-DRL share the same termination condition of an episode.

To remove the confusion, we have added the information to the updated paper.

We thank Reviewer bsXP for the positive feedback and insightful comments. Our answers to the three questions will address the concerns and clarify the paper's contributions. We have also updated our paper to clarify the points that the reviewer raised. Let us know if our response clarifies the unclear points.

Q1: It's unclear how critical the residual physics policy is needed.

$**A1:**$ The proposed residual action policy contributes to three distinguished traits of Phy-DRL: 1) automatic construction of safety-embedded reward, 2) fast training, and 3) mathematically-provable safety guarantee. In the updated paper, how the residual policy is leveraged to derive the condition of mathematically-provable safety guarantee is formally presented in Appendix D.1. For this reason, in the discussions below, we explain the critical roles or functions of residual policy in delivering the traits 1) and 2) only.

$------------------------------------------------------------------$

Trait 1): Automatic Construction of Safety-embedded Reward. Given many safety conditions, our proposed Phy-DRL can automatically and simultaneously construct the safety-embedded reward and compute the model-based action policy. To review this, we first recall our proposed safety-embedded sub-reward: $r({s}(k), {a}(k)) = s^\top(k) \cdot {\overline{{A}}^\top \cdot {P} \cdot \overline{{A}} } \cdot {s}(k) - {{s}^\top(k+1)} \cdot {P} \cdot {s}(k+1), \text{with } \overline{{A}} \buildrel \Delta \over = {A} + {{B} } \cdot {{F}}.$ Meanwhile, the model-based action policy in the residual policy diagram is ${a}_{\text{phy}}(k) = {F} \cdot {s}(k).$ We here conclude that model-based action policy in the residual policy diagram, i.e., $F$, is directly used in the safety-embedded reward formula. Besides, we note in the safety-embedded reward, $A$ and $B$ are known and available model knowledge, while $F$ and $P$ are to be designed. In our Phy-DRL framework, they are automatically and simultaneously obtained via Python’s CVXPY toolbox for solving an analytic centering problem. Specifically, the CVXPY toolbox first solves:

${\text{argmin}_{Q,R}} ~ ( {\log \det \left( {{Q}^{ - 1}} \right)}) \text{ subject to (7) and (11),}$ where Equations (7) and (11) of the paper include the transformed safety conditions. With the obtained $Q$ and $R$ at hand, $F$ and $P$ are then obtained as $F = R \cdot Q^{-1}, ~~~~ P = Q^{-1}.$ We here can further conclude that model-based action policy and safety-embedded reward are not separated. Their computations depend on each other under safety conditions formulated in Equations (7) and (11) of the paper.

$------------------------------------------------------------------$

Trait 2): Fast Training Towards Safety Guarantee. To understand how our residual action policy leads to fast training, we review two system dynamics.

$\text{System dynamics with purely data-driven DRL policy:} ~\mathbf{s}(k+1) = A \cdot s(k) + B \cdot a_{\text{drl}}(k) + f(s(k)), ~k \in \mathbb{N}. ~~~(\text{e}1)$

$\text{System dynamics with residual policy:} ~\mathbf{s}(k+1) = A \cdot s(k) + B \cdot (a_{\text{drl}}(k) + a_{\text{phy}}(k)) + f(s(k)), ~k \in \mathbb{N}~~~(\text{e}2)$

where $a_{\text{drl}}(k)$ and $a_{\text{phy}}(k)$ denote data-driven and model-based actions, respectively.

To review the training process from the perspective of safety guarantee, we construct a positive or energy function: ${V}\left(\mathbf{s}(k) \right) = {\mathbf{s}^\top(k)}\cdot {\mathbf{P}} \cdot \mathbf{s}(k)， ~~~{\mathbf{P}} \succ 0.~~~(\text{e}3)$

Along with system dynamics (e1), we have: $V( {{s(k+1)}}) - V( {{s(k)}}) = (B \cdot a_{\text{drl}}(k) + f(s(k)))^{\top} \cdot P \cdot (B \cdot a_{\text{drl}}(k) + f(s(k))) + 2(A \cdot s(k))^{\top} \cdot P \cdot (B \cdot a_{\text{drl}}(k) + f(s(k))) + ~s^{\top} (k) \cdot (A^{\top} \cdot P \cdot A) \cdot s(k). ~~~(\text{e}4)$

We note in our proposed residual policy diagram, the physics-model-based action policy is $a_{\text{phy}}(k) = F \cdot s(k)$, where the $F$ is computed automatically (see Remark 5.5 in the paper), such that it renders the following inequality holds:

$\overline{{A}}^\top \cdot {P} \cdot \overline{{A}} - {P} \prec 0, ~~\text{with }~\overline{{A}} \buildrel \Delta \over = {A} + {{B} } \cdot {{F}}. ~~~(\text{e}5)$

With this designed property (e5) at hand, along with system dynamics (e2), we have $V( {{s(k+1)}}) - V( {{s(k)}}) = (B \cdot a_{\text{drl}}(k) + f(s(k)))^{\top} \cdot P \cdot (B \cdot a_{\text{drl}}(k) + f(s(k))) + 2( \overline{A} \cdot s(k))^{\top} \cdot P \cdot (B \cdot a_{\text{drl}}(k) + f(s(k))) + s^{\top} (k) \cdot ( \overline{{A}}^{\top} \cdot P \cdot \overline{{A}}) \cdot s(k). ~~~(\text{e}6)$

With the consideration of designed model-based property (e5), through comparing (e4) and (e6), we can discover that $V( {{{s}(k+1)}}) - V( {{{s}(k)}})$ under the residual policy, i.e. the (e6), has a constant negative term: ${{{s}}^\top}(k) \cdot ( {{{\overline A}}^\top \cdot {{P}} \cdot {{{{\overline A}}}}} - {P}) \cdot {{s}}(k) < 0$. While ${{{s}}^\top}(k) \cdot ( {{{A}}^\top \cdot {{P}} \cdot {{{{A}}}}} - {P}) \cdot {{s}}(k) < 0$ is not guaranteed in the $V( {{\mathbf{s}(k+1)}}) - V( {{\mathbf{s}(k)}})$ under purely data-driven DRL policy (i.e., the (e4)). In addition, $V( {{\mathbf{s}(k+1)}}) - V( {{\mathbf{s}(k)}}) \le 0$ holds for $\forall k \in \mathbb{N}$ is desired for safety guarantee. Because it, in conjunction with the safety envelope (defined as state space that satisfies $s^\top \cdot P \cdot s < 1$), means the system states never leave the safety envelope (starting inside the safety envelope). Hereto, we can conclude that the residual policy can lead to fast training because it introduces a global attractor inside the safety envelope (through the model-based design property (e5)), such that $V( {{{s}(k+1)}}) - V( {{{s}(k)}}) \le 0$ is much more quickly to occur. This phenomenon can be depicted by the figure available at the link (anonymous hosting and browsing) (and also Figure 6 in Appendix D.2 of the paper). Both figures illustrate that the model-based property of residual policy (e5) can be understood as putting a global attractor inside the safety envelope, which generates an attracting force toward the safety envelope. Obviously, in any training stage, because of the attracting force that always exists, system states under the control of residual policy are much more likely to stay inside the safety envelope.

Motivated by the insightful question from the reviewer, we have added a subsection entitled Appendix D.2 Extension: Explanation of Fast Training to the paper, right after the proof of mathematically-provable safety guarantee.

Q3: The method relies on known state definitions. If the state can not be obtained from the environment or definition, how will this framework be affected?

$**A3:**$ In our proposed Phy-DRL framework, the data-driven DRL action policy and model-based action policy share the same real-time system state data for action computations. For our testbeds of autonomous systems, including autonomous cars, quadruped robots and drones, the available system states that the sensors directly output typically include CoM x-position, CoM y-position, CoM z-height, roll, pitch, yaw, CoM x-velocity, CoM y-velocity, CoM z-velocity, velocities of roll, yaw and pitch, car's longitudinal velocity, and car's wheel (angular) velocities. All these sensor data cover the needed system state of the general dynamics models of autonomous cars [1], quadruped robots [2], and drones [3].

For our application systems, we currently do not face the problem of some unavailable state from the environment or definition. If this problem happens, intuitively, the physical model for our model-based design in Phy-DRL may not be available or have a large model mismatch that leads to an infeasible model-based action policy and safety-embedded reward. To address this problem, a practical approach is to use currently available real-time state data from the environment to obtain or learn a corresponding model represented by $(A, B)$, via system identification (e.g., non-asymptotic identification using a single and finite-time trajectory, proposed in [4]). The newly identified $(A, B)$ are then used for constructing the safety-embedded reward and designing the model-based action policy for Phy-DRL.

$\left[ 1 \right]$ Rajamani, R. (2011). Vehicle dynamics and control. Springer Science & Business Media.

$\left[ 2 \right]$ Di Carlo, J., Wensing, P. M., Katz, B., Bledt, G., & Kim, S. (2018, October). Dynamic locomotion in the MIT cheetah 3 through convex model-predictive control. In 2018 IEEE/RSJ international conference on intelligent robots and systems (pp. 1-9).

$\left[ 3 \right]$ Six, D., Briot, S., Chriette, A., & Martinet, P. (2017). The kinematics, dynamics and control of a flying parallel robot with three quadrotors. IEEE Robotics and Automation Letters, 3(1), 559-566.

$\left[ 4 \right]$ Oymak, S., & Ozay, N. (2019, July). Non-asymptotic identification of LTI systems from a single trajectory. In 2019 American control conference (pp. 5655-5661).

Dear Reviewer bsXP,

We thank you for the very critical suggestion. We will add the wall-clock time to demonstrate the fast training towards safety guarantee further. We will let you know when the experimental results are added to the paper.

Sincerely,

Authors of paper.

For fast training property mentioned in the discussion on Q1, it would be great to provide experimental results about improvements on either sample efficiency or wall-clock time.

$**Answer:**$ Appendix K.7 in the updated paper includes experimental results with information of sample efficiency and wall-clock time. For convenience, the experimental results are also available at link (anonymous hosting and browsing). We saved the learned agent at 50K, 75K, 100K, and 200K steps for safety evaluation.

As depicted in the plots at link (anonymous hosting and browsing), our proposed phy-DRL framework in the model-free setting (Figure. a) demonstrates the capability to render the safety envelope invariant after just 50K training steps (note: we adopt step-wise optimization, where 50K training steps also mean 50K interaction steps in model-free setting). Phy-DRL trained in the model-based setting (Figure. b) can not render the safety envelope invariant after 50k training steps. This is attributed to the inherent inaccuracy of the model used in the model-based setting, leading to suboptimal agent performance in the true test MDP.

In contrast, the purely data-driven DRL requires more training steps to learn in both model-free (Figure. c) and model-based settings (Figure. d). The evaluation results at 50K and 75K steps indicate unstable training progress without convergence. However, after training for 100K/200K steps, performance improves, though not as effectively as with Phy-DRL.

To answer the question of sampling efficiency (interaction steps), Phy-DRL achieves convergence within 50K interaction steps (Figure. a), while traditional DRL requires considerably more steps (Figure. c). However, comparing model-free to model-based settings, the latter requires zero interaction with the true MDP, as the model is predetermined rather than learned. In real-world applications, additional interaction steps are typically needed to gather useful data for model learning/updating. This process may need to be repeated multiple times until the agent converges. Moreover, in some scenarios, data collection isn't straightforward, making it challenging to design an effective controller initially for data collection purposes. How many interaction steps are needed to improve the model is also dependent on the algorithm of updating the model and the policy, which makes the comparison of interaction steps challenging.

From a wall-clock time perspective (training steps), the plots illustrate that Phy-DRL achieves convergence after 50K training steps, making it the most time-efficient approach.

Comparisons with [2]. The work in [2] is seminal since it discovered that if DRL’s reward is a control Lyapunov function (CLF), a well-trained DRL can exhibit a mathematically-provable stability guarantee. Intuitively, if a CLF reward is constructed with consideration of safety conditions, a well-trained DRL with such a CLF reward can exhibit a mathematically-provable safety guarantee. For this reason, our paper presents a systematic comparison with the work in [2]. In the updated paper, all the pure DRL models have a CLF reward.

We also perform analysis and experimental comparisons on our safety-embedded reward and CLF reward, which are presented in Appendix L.7 (analysis report is also available at link 2 (anonymous hosting and browsing)). In summary, the difference is our reward decouples known invariant and unknown for learning (i.e., the data-driven component only learns the unknown), while the CLF reward mixes known invariant and unknown (i.e., the data-driven part component learns both unknown and known invariant). Decoupling known invariant and unknown for learning can be a critical reason our safety-embedded reward leads to fast training, compared with CLF reward.

W2: The effect of each of the invariant embeddings is not shown empirically. An included ablation analysis would be useful to determine this.

$**Answer:**$ The updated paper has the ablation analysis. Due to the page limit, some parts are included in the Appendices. To improve the clarity of the paper, we have added information indicating ablation analysis and connecting links to the corresponding content in the Appendices. The critical clarifications are presented below.

Effect of concurrent residual action policy and safety-embedded reward. The experiments on a cart-pole system in Section 7.1 are only to demonstrate the effect of concurrent residual action policy and safety-embedded reward. In other words, the only difference between the DRL models and our Phy-DRL model is that our Phy-DRL has the concurrent residual action policy and safety-embedded reward, while the DRL models have CLF rewards. All their actor and critic networks are implemented as the same multi-layer perceptron, with four fully connected layers. The output dimensions of all critic and actor networks are 256, 128, 64, and 1, respectively. All the activation functions of the first three neural layers are ReLU, while the output of the last layer is the Tanh function for the actor-network and Linear for the critic network. Such information is included in Appendix K.5 in the updated paper.

Effect of safety-embedded reward. The effect of sole safety-embedded reward is demonstrated through analysis and comparing training. However, due to the page limit, the experiment and analysis on the effect of sole safety-embedded reward are presented in Appendix L.7. In the experiment, we consider two Phy-DRL models. Their only difference is the reward; one is our safety-embedded reward, while the other is the CLF reward. All models' other structures and configurations are the same.

Effect of NN Editing. The effect of NN Editing is viewed from training. The experiment is in Appendix L.6 of the updated paper. For the considered models, their only difference exists in critic and actor networks. Specifically, one model's critic and actor networks are built on our proposed networks with NN editing. The other model's critic and actor networks are built on the same networks, but they do not have NN editing. All models' other structures and configurations are the same.

We thank Reviewer GsDG for the positive feedback and insightful comments. Our answers to the questions will address the concerns. Let us know if our response clarifies the unclear points.

Q1: Fig. 4 does not have comparisons with any modern methods for MBRL or physics based RL. Is there a reason why this was omitted?

$**Answer:**$ We would like first to recall that model-based RL involves using a model (whether learned or known) to predict future states for optimizing policies. Within the model-based RL framework, the model is used primarily to generate states, allowing the agent to query the Markov Decision Process at any desired state-action pair to train the action policy [r2].

We previously did not consider the comparisons with model-based DRL, because of the limitation of model-based RL, as discussed in [r3]: the performance is constrained by modeling errors or model mismatch. Our application systems, such as autonomous vehicles and quadruped robots, have system-environment interaction dynamics. These systems have environmental factors (e.g., road friction force and latency) that are often hard to model or measure, especially in dynamic and unforeseen operating environments. Therefore, relying on a model for only state prediction (in model-based RL) may lead to sub-optimal performance and safety violations potentially. Because of the concerns, in our proposed Phy-DRL, we introduce an alternative approach for using a model: Phy-DRL uses the model only for the automatic constructions of safety-embedded reward and residual action policy. In other words, our Phy-DRL does not adopt the model for state prediction.

In addition, we note the residual action policy in Phy-DRL can well address the challenge induced by a model mismatch between our used model and the real nonlinear dynamics model. Specifically, In our residual policy, the model-based action policy can guide the exploration of DRL agents during training. Meanwhile, the agent training is to learn a data-driven action policy that can effectively deal with uncertainties and compensate for the model mismatch of the model-based action policy.

Because our Phy-DRL does not adopt a model for state prediction. To have a fair comparison, our baseline models of DRL also do have the inside model for state prediction, i.e., they are model-free. For this reason, our previous experiment section did not include comparisons with model-based DRL.

To better answer the question, the updated paper also includes the comparisons with model-based DRL, which are presented in Appendix K.7 (also available at link 1 (anonymous hosting and browsing)).

[r2] Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023). Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1), 1-118.

[r3] Janner, M., Fu, J., Zhang, M., & Levine, S. (2019). When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32.

Dear Reviewer GsDG,

The discussion period is coming to an end, and we hope that we have addressed all of your comments. Could you take a look at our responses and updated paper, and let us know if you have any follow-up questions? We will be happy to answer them.

Kind regards,

the Authors

Dear Reviewer GsDG,

We appreciate your positive feedback, in-depth questions, and important suggestions.

S: One final note, the paper as-is is quite dense at the moment, it may be useful to relegate some text to the appendix (maybe some of the algorithm pseudocodes) then spacing text appropriately.

A： We agree with the reviewer the current version is quite dense. This problem can be easily addressed if our paper is finally accepted for publication. Because, in the final version, one paragraph of demonstration links can be updated with just one link. But to be anonymous in the discussion stage, we cannot use the simple link to replace the long paragraph.

Thanks for addressing my questions. The paper has many good ideas but I am inclined to keep my evaluation for two major reasons:

• Model Mismatch with Linear Model not fully qualified: The experiments on the quadruped robot with the linear model ignoring friction and setting robot angles as zero is promising for the methods real-world applicability. I would have liked a more formal statement on when the model mismatch is too great for the method to work well (in expectation).
• Provable safety is mostly empirical and on a limited model of the environment: To the best of my knowledge, the calculations of a provably-safe trajectory are within the Linear Model of the environment (hence the LMIs). While real-world trajectories that satisfy the safety sub-reward property are `safe' in practice and stay within the safety envelope, this does not mention anything about general deployed policies in the real world (in expectation or otherwise). Without a precise measure of model mismatch (missing from the paper), I fear this guarantee might be of limited utility.

Dear Reviewer,

We appreciate you for the very critical questions. The discussion deadline is approaching, but we would like to share two clarifications for your questions.

Q2: Provable safety is mostly empirical and on a limited model of the environment: To the best of my knowledge, the calculations of a provably-safe trajectory are within the Linear Model of the environment (hence the LMIs). While real-world trajectories that satisfy the safety sub-reward property are `safe' in practice and stay within the safety envelope, this does not mention anything about general deployed policies in the real world (in expectation or otherwise). Without a precise measure of model mismatch (missing from the paper), I fear this guarantee might be of limited utility.

Answer: The linear model in our Phy-DRL is only leveraged for the automatic constructions of safety-embedded reward and residual action policy. Our derived conditions of safety guarantee -- $r(\mathbf{s}(k), \mathbf{a}(k)) \ge \alpha - 1$ is generic (a sufficient condition for safety guarantee), and is not on a limited model of the environment. Because the $r(\cdot)$ is our defined safety-embedded reward for a real plant (not a linear model), and the $\mathbf{s}(k)$ and $\mathbf{a}(k)$ are real-time sensor outputs (system states and actions) of a real plant. Besides, the derived condition has the following two usages.

1) Training Stopping. During training stage, we will output and observe real-time $r(\mathbf{s}(k), \mathbf{a}(k))$. If $r(\mathbf{s}(k), \mathbf{a}(k)) \ge \alpha - 1$ always hold, according to our Theorem 5.3, system safety is guaranteed, training can stop.

2) Safety Evaluation of Trained Policy. The condition $r(\mathbf{s}, \mathbf{a}) \ge \alpha - 1$ can be also used for the evaluation of a trained policy deployed to a real plant. Sampling any state $\mathbf{s}$ inside a safety envelope, if $r(\mathbf{s}, \mathbf{a}) \ge \alpha - 1$ always holds, the trained action policy has a safety guarantee according to our safety definition.

Q1: Please clarify what is meant by invariant embeddings here.

$**Answer:**$ In this paper, an 'invariant' refers to a prior policy, prior knowledge, or a designed property, that is independent of agent training. `invariant embedding' refers to embedding such an invariant into DRL. For example, integrating a data-driven action policy with a model-based action policy yields our residual action policy. The model-based action policy is $a_{\text{phy}}(k) = F \cdot s(k)$, where $F$ completely depends on model knowledge $(A, B)$ (obtained offline), i.e., it is independent of agent training. Therefore, the residual action policy belongs to an invariant-embedding design. In addition, our proposed safety-embedded reward has an always-holding property: $\mathbf{P} - \overline{\mathbf{A}}^\top \cdot \mathbf{P} \cdot \overline{\mathbf{A}} \succ 0$ (independent of agent training). So, our proposed safety-embedded reward also belongs to invariant-embedding design. We have added the clarification to the updated paper.

Q3: Could you provide more insight into the "residual action policy"? How does it ensure invariance with respect to training?

$**Answer:**$ In Phy-DRL, the terminal action $a(k)$ has the residual form: $a(k) = a_{\text{drl}}(k) + a_{\text{phy}}(k)$, where the $a_{\text{drl}}(k)$ denotes a date-driven action from DRL, while the $a_{\text{phy}}(k)$ is a model-based action computed according to $a_{\text{phy}}(k) = F \cdot s(k)$, where the computation of $F$ is based on the model knowledge $(A, B)$.

The keyword ``residual" stems from Phy-DRL's working mechanism: the model-based action policy can guide the exploration of DRL agents during training. Meanwhile, the agent training is to learn or search for a data-driven policy to effectively deal with uncertainties and compensate for the model mismatch existing in the model-based action policy. The model mismatch exists because the model-based action policy is based on model knowledge $(A, B)$ (representing a linear model), while the dynamics model of real systems is nonlinear.

The keyword ``invariant" in residual policy stems from model-based action policy $a_{\text{phy}}(k) = F \cdot s(k)$. The $F$ completely depends on model knowledge $(A, B)$ (obtained offline), independent of agent training. In other words, model-based action policy is a prior design, not influenced by training.

We thank Reviewer 4Lie for the positive feedback and insightful comments. The detailed responses are provided below.

W2: The lack of real-world results is also concerning, given the efforts by the community to test robots in the real world.

$**Answer:**$ For the experiments of a quadruped robot, we have the corresponding real (R&D) A1 Explorer quadruped robot from Unitree. In our project, we train the robot in a simulator and then deploy the $\underline{trained}$ policy to the real robot. We cannot directly perform training on real robots. Otherwise, the robot can be easily damaged (e.g., the robot falls and the motor burns). Therefore, all the experimental results related to training (such as our claimed fast training towards safety guarantee) are obtained in the simulator. The simulator we are using is built on Pybullet, and released on GitHub. The simulator package includes an interface for direct sim-to-real transfer and has a strong record of deploying a trained policy to the same version of the A1 Explorer quadruped robot with us.

The experiments about the trained policies (for validating our derived safety guarantee) also need to be performed in the simulator. The critical reason is the sim-to-real gap. If a violation of safety occurs in a real robot, it can be caused by the sim-to-real gap, rather than our proposed Phy-DRL framework.

Beyond Scope of This Paper: addressing the sim-to-real gap via safe continued learning. To thoroughly demonstrate Phy-DRL in the real robot, our first step is to address the sim-to-real gap. To achieve this, we propose to develop a safe continued learning framework, which is built on neural Simplex [1]. In this framework, we will deploy the well-trained package (rather than a trained policy) from the simulator to the real robot and continue the training there. The neural Simplex will work as a fault-tolerant software architecture. If a safety violation occurs during the continued training, the Simplex triggers the switching to the high-assurance mode (a protection strategy) for temporarily suspending training. The Simplex will trigger switching to continued training when safety violations or concerns disappear. The continued training will stop if our derived condition of safety guarantee is satisfied in the real robot. The development of a safe continued learning framework for addressing the sim-to-real gap is beyond the scope of the paper.

[1] Phan, D. T., Grosu, R., Jansen, N., Paoletti, N., Smolka, S. A., & Stoller, S. D. (2020). Neural simplex architecture. In NASA Formal Methods: 12th International Symposium, NFM 2020, Moffett Field, CA, USA, May 11–15, 2020, Proceedings 12 (pp. 97-114).

Dear Reviewer,

Thanks a lot for your precious time in helping us improve our paper. We would like to thank you for your recognition.

Best regards,

Authors

I personally feel this is a good paper. The readability was a problem in the initial manuscript (this problem was discussed a lot by different reviewers as I checked) but the writing is greatly improved in the updated version. The approach is novel, sound, and very interesting, at least to me.