Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration

We sincerely appreciate the reviewer's positive and insightful comments. The followings are detailed responses to the points raised by Reviewer kehG.

When using a random network to project into latent space, my understanding is that distance in random latent space is mostly a "these two states are or are not similar" thing, rather than having use as a scalar quantity. As MICE uses a k-NN distance in this space, it makes me wonder if the numerical value matters or if it's just a matter of being moderately distant from previously sampled datapoints. Could the intrinsic rewards be quantized, and would that affect performance? I don't think this is critical to answer but the use of a random latent space seems at odds with the idea of specific distances mattering.

Response: We appreciate the reviewer’s insightful comments regarding the use of random projection in state comparisons.

In our experiments, random projection is implemented using a Gaussian random matrix of shape $(n, m)$, which projects states from the original dimension $n$ to a lower dimension $m$. According to the Johnson-Lindenstrauss lemma [1], random projection approximately preserves Euclidean distances in the original space, which is proven to be valid in similar KNN-based methods in prior works [2][3].

Additionally, we fully agree with the reviewer that relative similarity between states is more important than absolute scalar distance. In our implementation, both extrinsic and intrinsic costs are normalized to ensure training stability. This normalization alters absolute distance values but preserves similarities between states, which is key to maintaining meaningful guidance for intrinsic cost computation and policy learning.

To further assess the impact of random projection, we conducted an additional ablation study on it. The results in Figure 1 (https://anonymous.4open.science/r/7532-6C07/experiments3.pdf) demonstrate that using random projection does not degrade policy performance or increase constraint violations, further supporting the idea that state similarity is likely more critical than absolute numerical values.

We appreciate the reviewer’s constructive questions and hope these additional experiments and clarifications address the concern.

The paper title at the top of each page is not correct.

Response: We sincerely appreciate the reviewer’s careful review. We will correct this typo in the revised manuscript.

[1] Johnson, W. B., Lindenstrauss, J., et al. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26(189-206):1, 1984.

[2] Hu, H., Ye, J., Zhu, G., Ren, Z., and Zhang, C. Generalizable episodic memory for deep reinforcement learning. In International Conference on Machine Learning, pp. 4380–4390. PMLR, 2021.

[3] Zhu, G., Lin, Z., Yang, G., and Zhang, C. Episodic reinforcement learning with associative memory. In International Conference on Learning Representations.

We sincerely appreciate the reviewer's positive and insightful comments. The following are the detailed responses to the points raised by Reviewer rbBW.

The paper might benefit from investigating how this embedding function interact with the effect of KNN choice.

Response: We appreciate the reviewer’s valuable comment regarding investigating the embedding function and KNN choice.

In our paper, we utilize a random projection method as the embedding function to compress states before applying KNN. Specifically, states are projected from their original dimension $n$ to a lower embedding dimension $m$ using a Gaussian random matrix of shape $(n, m)$. This significantly reduces the computational complexity of KNN from $O(N n)$ to $O(N m)$, where $N$ is the number of states in memory. Prior studies [1][2] have validated the effectiveness of random projection in similar KNN-based methods.

We further conducted ablation experiments of random projection (Figure 1, https://anonymous.4open.science/r/7532-6C07/experiments2.pdf), which confirm that using random projection in MICE does not reduce policy performance or increase constraint violations, while substantially reducing training time.

Additionally, we examined the sensitivity of $N_k$ selection in KNN across multiple environments (Figure2, https://anonymous.4open.science/r/7532-6C07/experiments2.pdf). Results show that a larger $N_k$ enhances policy safety by considering more unsafe states, but it increases computational overhead. Conversely, a smaller $N_k$ may not fully utilize memory information, potentially leading to higher constraint. To balance safety, performance, and efficiency, we set $N_k = 10$ in MICE for all tasks.

We appreciate the reviewer’s constructive comments and hope that these analyses provide further clarity.

The baseline comparison with CPO (in Appendix C.3.1) should be moved to the main paper. I'd think that expanding to more tasks might verify that MICE is reliably safer than CPO.

Response: We appreciate the reviewer’s valuable comments. Following your advice, we will move the comparison between MICE and CPO from Appendix to the main text for improved clarity.

To further verify the safety advantages of MICE, we conducted additional experiments in a broader set of environments. The results in Figure 3 (https://anonymous.4open.science/r/7532-6C07/experiments2.pdf) demonstrate that MICE consistently achieves superior constraint satisfaction across all tasks, while CPO exhibits significant constraint violations.

Additionally, to provide a more intuitive quantitative comparison, we present the difference in constraint violation rates between CPO and MICE during training. The results in Table 1 (https://anonymous.4open.science/r/7532-6C07/experiments2.pdf) show that CPO consistently exceeds MICE in violation rates in all environments, with an average violation rate 34.4% higher than MICE. This further confirms the safety advantages of MICE.

The authors might want to consider expanding to other tasks to check how the discount rate of 0.99 behaves and comparing with PPO-Lagrangian and PID-Lagrangian.

Response: We appreciate the reviewer’s valuable comment regarding the behaviour of discount factor 0.99 in more tasks and its comparison with PPO-Lagrangian and PID-Lagrangian. To address this, we evaluated MICE with a discount factor of 0.99 on additional tasks and compared its performance against PPO-Lagrangian and PID-Lagrangian. As shown in Figure 4 (https://anonymous.4open.science/r/7532-6C07/experiments2.pdf), MICE consistently achieves strong constraint satisfaction while maintaining superior policy performance. In contrast, PPO-Lagrangian and PID-Lagrangian exhibit significant oscillations in constraint during training.

In our experiments, we used the consistent discount factors across all methods.

[1] Hu, H., Ye, J., Zhu, G., Ren, Z., and Zhang, C. Generalizable episodic memory for deep reinforcement learning. In International Conference on Machine Learning, pp. 4380–4390. PMLR, 2021.

[2] Zhu, G., Lin, Z., Yang, G., and Zhang, C. Episodic reinforcement learning with associative memory. In International Conference on Learning Representations.

We sincerely appreciate the reviewer's positive and insightful comments. The following are the detailed responses to the points raised by Reviewer 9DRS.

The assumptions (e.g., finite MDP, extensive sampling) might restrict practical applicability in more complex or continuous environments.

Response: We thank the reviewer’s valuable comment. To address this concern, we conducted additional experiments comparing MICE with CPO in SafetyCarButton1-v0 and SafetyPointButton1-v0 environments. These tasks are more complex, requiring agents to navigate to a target button and correctly press it while avoiding Gremlins and Hazards. Results presented in Figure 1 (https://anonymous.4open.science/r/7532-6C07/experiments1.pdf) show that MICE achieves superior constraint satisfaction while maintaining policy performance comparable to CPO.

Furthermore, we acknowledge potential limitations where direct environmental sampling is infeasible. To address this, offline data can be leveraged to construct memory, thus enhancing MICE’s applicability in these special cases.

I suggest the authors provide a hyperparameter sensitivity experiment on $N_k$ in KNN.

Response: We appreciate the reviewer’s valuable suggestion. To address this, we conducted hyperparameter sensitivity experiments for $N_k$ across multiple environments and analyzed the impact, as shown in Figure 2 in (https://anonymous.4open.science/r/7532-6C07/experiments1.pdf). The results show that increasing $N_k$ enhances the safety of the policy, as it considers more unsafe states in memory, but also raises computational overhead. Conversely, a smaller $N_k$ may lead to insufficient leveraging of unsafe state information, potentially leading to higher constraint. In this paper, we selected $N_k = 10$ uniformly across all environments to balance safety, performance, and computational efficiency.

Please provide the detailed implementation specifics of the random projection layer.

Response: We appreciate the reviewer's valuable comment regarding the details of random projection.

Our random projection layer is implemented using a Gaussian random matrix with shape $(n, m)$, projecting states from the original dimension $n$ to a lower embedding dimension $m$. According to the Johnson-Lindenstrauss lemma [1], this approach approximately preserves relative Euclidean distances in the original space, which is proven valid by prior KNN-based methods [2][3].

In our work, this dimensionality reduction effectively decreases the computational complexity of KNN from $O(N n)$ to $O(N m)$, where $N$ is the number of states in memory. Specifically, in SafetyPointGoal1, the state dimension is reduced from 60 to 8.

Furthermore, we conducted ablation experiment on random projection. As demonstrated in Figure 3 (https://anonymous.4open.science/r/7532-6C07/experiments1.pdf), utilizing random projection in MICE does not degrade policy performance or increase constraint violations, while effectively reducing training time.

The flashbulb memory only stores states with extrinsic cost greater than zero. However, the states with outlier extrinsic costs may not store in the memory and therefore might not be adequately corrected by the intrinsic cost mechanism. How does the paper handle these cases?

Response: We appreciate the reviewer’s insightful question. In our tasks, the extrinsic cost is derived by the environment based on the agent's and obstacles' coordinates, providing unbiased real data without outliers. Underestimation primarily occurs due to constraint minimization during optimization, which disrupts the zero-mean property of noise, especially in regions with higher extrinsic costs. Therefore, current memory mechanism adequately captures these underestimated states in our tasks.

However, we fully agree with the reviewer that in other special tasks, where extrinsic costs may be biased, outlier states could arise. To handle such cases, alternative criteria for adding new states into memory could be applied. One possible strategy is using the expected subsequent costs as the criterion for adding states into memory: a state would be stored if its expected future cost exceeds a threshold. This approach effectively mitigates the impact of individual outliers by considering multiple future states collectively.

[1] Johnson, W. B., Lindenstrauss, J., et al. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26(189-206):1, 1984.

[2] Hu, H., Ye, J., Zhu, G., Ren, Z., and Zhang, C. Generalizable episodic memory for deep reinforcement learning. In International Conference on Machine Learning, pp. 4380–4390. PMLR, 2021.

[3] Zhu, G., Lin, Z., Yang, G., and Zhang, C. Episodic reinforcement learning with associative memory. In International Conference on Learning Representations.