MICE: Memory-driven Intrinsic Cost Estimation for Mitigating Constraint Violations

We sincerely appreciate the reviewer's insightful comments. The following are the detailed responses to your comments.

Weakness 1

How were these hyperparameters set in the paper? How can they be appropriately determined by a practitioner hoping to use this method for a problem where, for e.g. the acceptable level of constraint violation is subjective?

Response: We appreciate the reviewer's valuable comment and provide a detailed explanation of the hyperparameters. The intrinsic factor $h$ is a critical hyperparameter in MICE, set to 0.6 in this paper. By adjusting $h$, we can control the acceptable level of constraint violation to meet different task requirements, as shown in Figure 7(a)(b). For tasks prioritizing safety, increasing $h$ enhances risk aversion and improves constraint satisfaction. Conversely, for tasks emphasizing performance, decreasing $h$ allows for better policy performance.

Memory capacity, set to 64 in this paper, has a comparatively smaller impact on MICE's performance, as shown in Figures 7(c) and 7(d). We provide a detailed analysis of these hyperparameters in Appendix C.2.1 and C.2.2.

Weakness 2

What if many trajectories have costs just below the threshold and are therefore not stored?

Response: Thank you for your valuable comment. For risk-sensitive tasks, when many trajectories have costs just below the threshold, more conservative criteria can be adopted for judging unsafe trajectories. For instance, a trajectory can be stored in memory if its cumulative cost exceeds a specific quantile of the constraint threshold, thereby improving constraint satisfaction.

We have conducted experiments where trajectories were stored if their cumulative cost exceeded 90% of the constraint threshold. The results, detailed in the Appendix C.2.6 and Figure 17 (highlighted in red for clarity), demonstrate that this approach leads to a more conservative policy with reduced constraint violations.

Weakness 3

Why the standard deviation of results was provided?

Response: We appreciate the reviewer's careful review and valuable comments. To ensure fairness in the experimental results, we adopt the same evaluation method using the standard deviation as employed in the baseline works.

Weakness 4

The plots are not always the clearest, as axes are sometimes labelled and sometimes not. The plot legends do not indicate what value is changing.

Response: Thank you for your valuable comment. We have modified the revision according to your suggestions to improve clarity.

Question 1

Why a generator network is required as it would appear that an explicit memory and trajectory-distance metric are enough for the rest of the method to work. Why not simply store violating trajectories and then provide an intrinsic cost based on new trajectories' distance to the stored trajectories? What benefit does the generator provide?

Response: We appreciate the reviewer's insightful questions and have provided a detailed explanation about this in the revision for clarity. The Generator is designed to minimize memory accesses, thereby reducing computational complexity. During the agent’s learning process, if the cumulative cost of the current trajectory $\tau$ is below the constraint threshold, the generator directly produces the intrinsic cost. Additionally, the generator network’s representation capability enhances the generalization of intrinsic cost signals.

We hope our responses address your comments. Please let us know if you have follow-up questions. We would be happy to discuss.

We sincerely appreciate the reviewer’s insightful comments. Below are our detailed responses to your feedback.

Weakness 1, 2 and Question 2

Despite experiments across several environments, the evaluation lacks assessments in a broader range of task types and robot types. Could the authors demonstrate the scalability of MICE across more tasks and in more complex settings, particularly with high-dimensional inputs (e.g., trajectories based on images)?

Response: Thank you for your valuable suggestion. We have extended our evaluation to encompass more tasks and complex settings, including SafetyPointButton1-v0 and SafetyHopperVelocity-v4, with results presented in Appendix C.2.5 and Figure 16 (highlighted in red for clarity). The SafetyPointButton1-v0 task has more complex settings, with an observation dimensionality of 76 (range $(-\infty, \infty)$) and an action dimensionality of 2 (range $(-\infty, \infty)$). SafetyHopperVelocity-v4 introduces a different robot type. These experiments results demonstrate that MICE effectively balances performance and constraint satisfaction compared to multiple baselines, showcasing its scalability across a broader range of task types and robot types.

For high-dimensional state spaces, the random projection layer in the MICE framework compresses trajectories into a latent space, significantly reducing computational complexity. Additionally, the intrinsic generator in MICE further minimizes computational complexity by reducing memory accesses, directly generating the intrinsic cost when the cumulative extrinsic cost of the current trajectory is below the constraint threshold. We only compute the distance between trajectories when the cumulative extrinsic cost of the current trajectory exceeds threshold, thus optimizing the computational costs. Thank you for your valuable comments. We are considering scaling MICE to vision-based tasks in future work.

Weakness 3

Although this method shows advantages over baselines in balancing constraint satisfaction and performance, it still does not address safety issues during the learning process, merely increasing conservativeness.

Response: MICE can achieve zero constraint violations during the learning process by increasing the intrinsic factor, as demonstrated in Figures 7(a) and 7(b). Additionally, the extrinsic-intrinsic cost value function in MICE is guaranteed to converge to the optimal value.

Question 1

Figure 5 indicates that this method can still ensure that actual costs align with thresholds rather than being excessively low. How they maximize costs (implying limits on hazardous behavior) while largely avoiding constraint violations?

Response: In constrained reinforcement learning (CRL) tasks, the optimization process is complex because maximizing rewards may lead to increased costs, preventing the actual costs from being excessively low. The optimal policy in CRL tasks typically lies near the constraint thresholds, balancing performance and constraint satisfaction. The objective of MICE is to maximize cumulative rewards while ensuring cumulative costs remain below the constraint threshold. During maximizing cumulative rewards, if cumulative costs exceed the threshold, indicating constraint violations, MICE reduces the costs to bring them back under the threshold, which ensures the actual costs align with the thresholds while maintaining good reward performance.

We hope our responses address your comments. Please let us know if you have follow-up questions. We would be happy to discuss.

We sincerely appreciate the reviewer's insightful comments. The following are the detailed response for your comments.

Weakness 1

A baseline like TD3 addresses underestimation bias by using the maximum of the output from two separately-learned action-value networks for cost to combat this underestimation bias in cost.

Response: Thank you for your valuable comment. We have conducted experiments to compare the cost value estimation bias between TD3 cost value function and MICE with the PIDLag optimization method in SafetyPointGoal1-v0 and SafetyCarGoal1-v0, the results are shown in Appendix C.2.4 and Figure 15 (highlighted in red for clarity).

The results show that TD3 mitigates underestimation bias in cost value estimation, but it cannot fully eliminate it. This limitation arises from the inherent slow adaptation of neural networks, which results in a residual correlation between the value networks, thus preventing TD3 from completely eliminating the underestimation bias. In contrast, MICE can completely eliminate this bias by adjusting the intrinsic factor, leading to improved constraint satisfaction.

Additionally, compared to the TD3 cost value function, the flashbulb memory structure in MICE helps address the catastrophic forgetting issue in neural networks [1], where agents may forget previously encountered states and revisit them under new policies. The MICE mechanism generates intrinsic cost signals that guide the agent away from previously explored dangerous trajectories, effectively preventing repeated encounters with the same hazards.

Weakness 2

Elaboration on Eq3

Response: Thank you for your valuable comments. We have made modifications to this part in the revision to improve clarity and readability (highlighted in red).

a. What does the discount factor here signify? This "intrinsic discount factor" is different from the reward and cost discount factor and its exponent is $k$: iteration number of value estimation. Why is there compounding discount as the iteration increases?

Response: The intrinsic discount factor $\gamma_I$ works with the iteration number $k$ of the value function update, which differs from the discount factor $\gamma$ used for rewards, which operates with the rollout time step $t$. The intrinsic discount factor $\gamma_I$ is designed to ensure that our cost value converges to the optimal value. As shown in traditional value function [2], increasing $k$ leads to a gradual convergence of the value estimate to the optimal value, with the underestimation bias decreasing. The term $\gamma^k$ can ensure that our proposed value function converges to the optimal value, as detailed in Theorem 3 with the convergence analysis.

b. The L2-norm distance depends on time index $t$ such that it only compares the state-action similarity (between current trajectory and historical trajectory) at time $t$. This is quite counterintuitive because RL policy function or value function are usually not time-index dependent.

Response: In equation (3), $t$ represents the current time step of agent's rollout in environment. $\tau(t)= \{ s_0, a_0, \cdots, s_{t-1}, a_{t-1} \}$ denotes the current trajectory from the initial time step to $t$. $\tau_i^m(t)=\tau_i^m[0:t-1]$ denotes the segment of the $i$-th unsafe trajectory in memory $M$ from the initial time step to $t$. At each time step $t$, the flashbulb memory mechanism generates the intrinsic cost $c^I(s_t, \pi(s_t))$ as defined in equation (3), which corresponds to the extrinsic cost $c^E(s_t, \pi(s_t))$ provided by the environment. The cost is then used to update the value function $V_C(s_t)$ and policy $\pi(s_t)$ at the current time step $t$.

c. The weight $\omega$ seems to be another hyper-parameter. How does one decide the right weight to use?

Response: When calculating the trajectory distance, $\omega$ is used to adjust the weights of different types of states to generate the intrinsic cost in MICE. We tested across multiple environments, demonstrating that $\omega = 0.5$ is a robust choice for achieving balanced performance and constraint satisfaction, as shown in Appendix C.4.

References

[1] Zachary C Lipton, Kamyar Azizzadenesheli, Abhishek Kumar, Lihong Li, Jianfeng Gao, and Li Deng. Combating reinforcement learning’s sisyphean curse with intrinsic fear. arXiv preprint arXiv:1611.01211, 2016.

[2] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actorcritic methods. In International conference on machine learning, pp. 1587–1596. PMLR, 2018.

I thank the reviewer for the explanation. That explains how the authors arrive at the conclusion that overestimation bias of MICE doesn't seem "as bad" as the underestimation bias of TD3.

One thing to highlight is that Fig15 only plots the estimation error throughout training, it doesn't plot the effect it has on safe RL like in Fig3. In fact, the accumulated cost of PID-Lagrangian (purple curve) in CarGoal and PointGoal isn't too bad in Fig3, even though Fig15 shows that PIDLag has significant underestimation. Thus, I'm not entirely sure if the slight underestimation bias of TD3 adversely impacts its ability in adhering to constraint as claimed. Perhaps a similar figure like Fig3 could shed light on this point.

We sincerely appreciate the reviewer's insightful comments. The following are the detailed response for your comments.

Weakness 1

The writing of this paper requires significant improvement and polish. Many unclear notations and statements make it hard to follow. For example, in equations (3) and (4), the symbols $\tau_i^m(t)$ and $\tau(t)$ are never defined. The term $c^I_t$ is stated to be a function of $\gamma_I^k$, where $k$ represents the iteration number; thus, $c_t^I$ should also depend on $k$. The symbol $W$ is described as a set, yet in equation (3), $W$ appears to be used as a weight. Furthermore, it is unclear what $n$ represents in this context. Additionally, $c^E$ is not defined in equation (4).

Response: Thank you for your valuable comments. We have made modifications to this part in the revision to improve clarity and readability (highlighted in red). $\tau(t)=\{s_0, a_0, \cdots, s_{t-1}, a_{t-1}\}$ denotes the current trajectory from the initial time step to $t$. $\tau_i^m(t)=\tau_i^m[0:t-1]$ denotes the segment of the $i$-th unsafe trajectory in memory $M$ from the initial time step to $t$. $W$ denotes the weight vector, and $n$ denotes the number of unsafe trajectories stored in memory. $c^E$ denotes the extrinsic cost, which is provided by the environment. We also provided the Notations Table in the first page of the Appendix.

Weakness 2

In the definition of $J_C^{EI}$, which depends on both extrinsic and intrinsic costs, $c^E$ seems to rely on flashbulb memory. This makes it a random variable dependent on the policy applied at each step. Fundamentally, the expectations in $J_C^{EI}$ and $J_C(\pi)$ differ, and thus, one cannot prove lemmas such as Lemma 1 by simply subtracting one from the other.

Response: We appreciate the reviewer's careful observations. We would like to provide a detailed explanation of Lemma 1 and its proof. $c^E$ denotes the extrinsic cost, which is task-related and generated by the environment, independent of the flashbulb memory. $c^I$ denotes the intrinsic cost generated by the flashbulb memory. Both $c^E(s_t,\pi(s_t))$ and $c^I(s_t,\pi(s_t))$ depend on the policy $\pi$.

The term $E_{\tau|\pi'} \left[ \sum_{t=0}^\infty \gamma^t A_{C}^{EI}(s_t,a_t|\pi) \right]$ in Lemma 1 indicates that using the advantage function $A^{EI}_C$ over $\pi$ to evaluate the trajectory generated by $\tau \sim \pi'$, similar to the performance difference lemma in TRPO.

The expectations in $J_C^{EI}(\pi')$ and $J_C(\pi)$ can be expanded as:

$

J_C^{EI}(\pi'):= E_{\tau \sim \pi'} [\sum_{t=0}^{\infty} \gamma^t (c^E(s_t, \pi'(s_t)) + c^I(s_t, \pi'(s_t)))]

$

$

J_C(\pi):= E_{\tau \sim \pi} [\sum_{t=0}^{\infty} \gamma^t c^E(s_t,\pi(s_t))] = \mathbb{E}_{s_0\sim\rho}[V_C^\pi(s_0)]

$

The trajectories generated by $\pi'$ remain unchanged during the proof. The expectation of $J_C(\pi)$ can be denoted as $\mathbb{E}_{s_0\sim\rho}[V_C^\pi(s_0)]$, which independent of the trajectories $\tau \sim \pi'$. The initial state $s_0$ in $V_C^\pi(s_0)$ depends solely on the initial state distribution $\rho$, allowing the expectation over $s_0 \sim \rho$ to be expressed as an expectation over $\tau|\pi'$. Therefore, the expectations in $J_C^{EI}$ and $J_C(\pi)$ can be subtracted.

Due to word limitations, we provide a detailed explanation of the proof in the Appendix B.2.

Weakness 3

The Bellman equation presented in (6) does not appear to be correct. As mentioned in the introduction, safe RL is often solved using a primal-dual approach, which implies that the value function update should be in the SARSA form.

Response: We appreciate the reviewer's careful review and valuable comment. We have clarified this point in revision. Equation (6) applies to the Q-learning case, corresponding to the analysis in Section 4.1. In our work, both MICE-CPO and MICE-PPOLag are PPO-based methods, with their cost value updates employing the Generalized Advantage Estimation (GAE) method [1], as shown in Appendix C.1:

$\hat{A}^{EI}_t = \sum _ {l=0}^\infty (\gamma \lambda)^l \delta _ {t+l}$

where $\delta_t = c^E_t + c^I_t + \gamma V(s_{t+1}) - V(s_t)$. The details of GAE can be found in [1].

Weakness 4

Based on the experimental results, it seems the paper’s claimed contribution towards addressing training violations is not clearly substantiated.

Response: Thank you for your valuable comment. By increasing the intrinsic factor, MICE can achieve zero constraint violations during the learning process, as demonstrated in Figures 7(a) and 7(b).

By adjusting $h$, we can control the acceptable level of constraint violation to meet different task requirements. For tasks prioritizing safety, increasing $h$ enhances risk aversion and improves constraint satisfaction. Conversely, for tasks emphasizing performance, decreasing $h$ allows for better policy performance. The detailed analysis is provided in Appendix C.2.1.

Questions 1

Over what is the expectation in the equation (5) taken?

Response: Thank you for your valuable comment. We have clarified this point in revision. The expectation in equation (5) is taken over trajectories $\tau$ where the cumulative extrinsic cost exceeds the constraint threshold during the agent's learning process.

Questions 2

Are there any justifications for calculating the difference in equation (3) in terms of individual steps?

Response: Thank you for your insightful question. If the current trajectory shares more similar state-action pairs with an unsafe trajectory in flashbulb memory, it indicates a higher risk that the current trajectory similarly violates the constraint as the unsafe trajectory. The purpose of Equation (3) is to direct the current trajectory away from previously encountered unsafe trajectories and prioritize segments with significant costs.

Questions 3

LP-based approaches are missing in the related work.

Response: Thanks to your reminder, we have added the relevant method in the related work of the revision.

Questions 4

Typo in Figure 6? The word "constant" appears on top of each figure.

Response: Thank you for your careful comment, and we have made it more clear in the revision.

We hope our responses address your comments. Please let us know if you have follow-up questions. We would be happy to discuss.

References

[1] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.