Safety through feedback in Constrained RL

Thank you for your thoughtful review and constructive feedback. Your insights and suggestions are appreciated, and we look forward to addressing your questions below:

1. In binary classification problems, the ground truth is often represented as a Dirac measure (indicator function), and maximum likelihood estimation (MLE) leads to the widely used Binary Cross-Entropy (BCE) loss. Motivated by this, we represent the cost function as a probability measure in our work and subsequently use MLE.

2. We understand the concern and have added plots showing the inferred cost function and the estimated costs in Figure 1 in the attached document. It can be seen that the cost function closely approximates the true cost in all the environments. In Hopper, there is a bit of divergence, but with enough steps, we are able to get close.

3. Yes, this suggestion makes for interesting future work. The challenge lies in converting the MDP to text, instead eliciting feedback from multi-modal foundation models that are shown videos of the trajectory is another promising direction.

4. We understand this is a subjective assessment of the reviewer and we respect that. However, we do wish to point out that there are multiple key differentiating factors between the proposed work and existing literature that the reviewer seems to have missed:

   • [25,29,10,7,5] are restricted to state-level feedback from the evaluator. Which can be a burden on the evaluator. The proposed method does not have the same restriction. Our results on the Circle and Mujoco based environments show that it is possible to collect feedback at the trajectory level and still obtain good performance.
   • All existing methods except [7] do not consider the problem of selecting a subset of all the trajectories taken for feedback. Eliciting feedback for every trajectory taken can again lead to a significant burden on the evaluator. They do not scale to complex environments with large state-action spaces. To the best of our knowledge, our paper is the only one to experiment on complex environments like the Safety Gymnasium benchmark, which is used to evaluate the performance of Constrained RL algorithms with known cost functions.
   • Scalable methods have been proposed in Constraint Inference from demonstrations [6,30,20]. These methods assume the existence of constraint-abiding expert demonstrations which may not be readily accessible in all scenarios. Instead of generating expert trajectories, we only need evaluation of automatically generated trajectories by human/machine.
   • Novelty-based sampling has been introduced in the context of exploration problems [8,26]. To the best of our knowledge, we are the first to apply it in feedback-based RL settings. Additionally, we extend the concept of novelty to trajectories, inspired by its similarity to edit distance between trajectories [24]. The effectiveness of novelty based sampling as a form of sampling states that have a high prediction error is shown in Figure 5 in the attached document, and the overall gain in performance from using this sampling method is present in Figures 1 and 7.

   We would refer the reviewer to Section D in the Appendix for further details on the prior work.

   Our analysis of Table 1 reveals that the proposed RLSF algorithm demonstrates notable improvements over the baseline approaches across all tested environments. The baseline methods faced challenges in balancing cost estimation and reward optimization. In some cases (Point Circle, Biased Pendulum, Blocked Swimmer, Half Cheetah, Point Goal, Point Push), the baseline approaches tended to underestimate costs, resulting in higher cost violations. In others (Car Circle, Car Push), costs were overestimated, leading to suboptimal reward outcomes. The RLSF algorithm, however, showcases consistent superior performance compared to these baseline methods. Additionally, it achieves results comparable to PPOLag in several environments which underscores the potential of RLSF as a promising approach for addressing the complexities of reinforcement learning tasks without knowing the underlying cost constraints and just having demonstrations.

Thank you for your thorough and positive review of our paper. We look forward to addressing your questions and suggestions below:

1. By sparsity we mean the density of violations within the segment is low. This makes distinguishing safe states from unsafe states challenging, as it becomes difficult to identify which states caused the segment to be marked unsafe. Consider Figure 8 in the Appendix, in this case less than 10% of the states in that trajectory are actually unsafe. Thus, the rest of the states get an incorrect pseudo-label of unsafe. This would reduce significantly if the cost violations were more dense, say (~50%) of the states within the trajectory were unsafe, or if feedback was collected over shorter segments (which inherently reduces sparsity). To summarise, dense cost violations and shorter feedback segments simplify the cost inference problem, while sparse violations and longer segments make it more challenging.

2. Thanks to the reviewer for pointing this out and we do need to modify it as follows:

   “We posit that with a sufficient number of samples, we will be able to distinguish the safe states from unsafe states. This is because we observe that:

   (i) safe states will appear in both safe and unsafe segments; and (ii) unsafe states would appear only in unsafe segments and not in safe segments;”

   Hence with a sufficient number of samples, it would become feasible to distinguish safe from the unsafe states. We would direct the reviewer to our strong performance on multiple safety environments (Circle, Goal and Mujoco based) that indicates this hypothesis to hold quite well in environments when the feedback is collected for the entire trajectory.

3. We can broadly categorise the reasons for model prediction errors into four main branches: Limited Data, Noise in the Labels, Model Misspecification and Optimization Error [Burda, et al.]. The latter two sources must be addressed before the learning process. Novelty based sampling addresses the first source of error. Label Noise can be reduced by decreasing the segment length. Other methods such as eliciting extra state-level feedback on states with noisy labels are also possible, making for interesting future work.

4. Please note that for doing cost transfer, there is an explicit cost function that has to be learnt that can be transferred. We would like to highlight that SIM and SDM do not learn explicit cost functions. Additionally, previous work [25,29,10,7,5] has not been scaled to handle the environments with continuous state and action spaces considered in these experiments. Therefore, our cost transfer experiments (Table 2) were limited to the aforementioned algorithms.

5. We appreciate the feedback on Table 1, we will bold the numbers in the final draft.

6. The main motivation for proposing the surrogate loss was due to the numerical instability that arises from multiplying the probabilities over a long horizon. To further illustrate this point, we plotted the norm of the gradient after the first 10 epochs of training the classifier using Eq 3 directly, as shown in Figure 1 (in the attached document). The gradient either collapses to 0 or explodes very quickly in the first few iterations making optimization challenging. The surrogate objective side-steps the need to multiply probabilities over long horizons and thus results in stable gradients as shown in Figure, leading to easier optimization. We acknowledge that this motivation was brief in the paper (due to space limitations), but will take this feedback into consideration and make the motivation more clear.

Citations: Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2019). Exploration by random network distillation. In International Conference on Learning Representations (ICLR).

Thank you for your detailed review and constructive feedback. We appreciate your insights and will address your questions and concerns below:

1. Please note that $p_{gt}^{safe}(s,a)$ is the ground truth probability. Since ground truth is exactly known for a state on whether it is safe or unsafe, $p_{gt}^{safe}(s,a)$ is either 0 or 1 depending on whether $c_{gt} = 0 \text{ or } 1$ respectively. This is the reason for using indicator function for ground truth. $p^{safe}(s,a)$ is our estimate of the probability and this will typically have values that are not 0 and 1 only.

2. The surrogate loss introduces a binary classification problem where each state is assigned a pseudo-label $y^{\text{safe}}$ based on the segment it occurred in. Safe states (w.r.t ground truth cost) can be assigned both unsafe and safe pseudo-labels depending on whether they occur in a segment that contains an unsafe state (or not). We refer to this phenomenon as “noisy labels” in the surrogate classification task. Yes, the evaluator is tasked to classify a segment unsafe if it contains one or more unsafe states. The rest of the states (if any), although inherently safe, are assigned the pseudo-label of unsafe.

3. Thanks to the reviewer for pointing this out and we do need to modify it as follows: “We posit that with a sufficient number of samples, we will be able to distinguish the safe states from unsafe states. This is because we observe that: (i) safe states will appear in both safe and unsafe segments; and (ii) unsafe states would appear only in unsafe segments and not in safe segments;” Hence with a sufficient number of samples, it would become feasible to distinguish safe from the unsafe states. We would direct the reviewer to our strong performance on multiple safety environments (Circle, Goal and Mujoco based) that indicates this hypothesis to hold quite well in environments when the feedback is collected for the entire trajectory.

4. There is a single evaluator (human or a program that is pre-trained/pre-coded) that classifies a segment. Each trajectory that requires evaluation (chosen based on novelty) is then broken down into segments, and each segment is passed to the evaluator for feedback. Feedback is not explicitly collected for individual states unless the segment length is 1.

5. We theoretically quantify the difference caused by transitioning from Equation 3 to Equation 4 in Proposition 3, in terms of cost overestimation. If this difference is considered excessive (results in overly conservative policies), the segment length can be reduced (by the user). However, this adjustment comes at the cost of increased evaluation effort, thus introducing a tradeoff.

   Directly optimising Equation 3 with trajectory-level feedback results in numerical instability (Figure 2, attached document). This highlights the necessity of the proposed surrogate objective.

   We further investigate the effects of transitioning from Equation 3 to Equation 4 in Table 1 of the attached document. MLE corresponds to taking feedback at the state level, equivalent to solving Equation 3. The Surrogate, on the other hand, represents solving Equation 4 with feedback collected at the trajectory level.

   We find that in the Circle environment, optimising the surrogate loss with trajectory-level feedback achieved performance comparable to optimising Equation 3. This suggests that the difference between Equation 3 and Equation 4 is minimal in the Circle environment. Additionally, the strong performance of RLSF compared to PPOLag in Table 1 of the main paper indicates that this minimal difference likely extends to the other Mujoco-based environments as well, demonstrating the promising applicability of our approach across a wide range of problems.

   However, Table 1 in the attached document shows a divergence between the two objectives in the Goal environment. The poor performance of RLSF in this case can be attributed to the overestimation bias discussed in the paper and the complexity of the cost function in this task, as described in lines 267-275. Therefore, in this scenario, state-level feedback is necessary for good performance.

Thank you for your detailed and constructive review of our paper. We appreciate your positive assessment of our work and the valuable feedback you provided. We hope to address the weaknesses and questions you raised below:

1. We use a regular Multi-Layer Perceptron (MLP) to estimate the probability of safety. The model takes the state-action pair as input and outputs the probability $p^{safe}_{\theta}(s,a)$. We request the reviewer to refer to Section C.1 in the Appendix for further details on the network architecture of the Classifier.

   Yes, we did consider directly estimating $d_{g}$ and $d_{b}$, some possible methods include:

   • SimHash can be used to discretize the state-space and store $d_{g}$ and $d_{b}$ via counts. We believe hashing states via raw feature state features is good enough for novelty estimation but the classification task would require more rich features. This is because this type of model is essentially a linear model, limiting its capacity to capture state densities at the accuracy required for cost function estimation.
   • Exemplar methods, as proposed by Fu et al., estimate state densities through a proxy binary classification task. This task distinguishes between a single state $s^*$ and the states in the feedback buffer $B$. Let $D$ represent the dataset containing only $s^*$. Since $s^*$ may also exist in $B$, $p(D|s^*)$ is not necessarily 1, requiring the classifier to account for the density of $s^*$ in $B$. This approach enables the estimation of $p(s^*|B)$, i.e, the state density. We believe solving two proxy binary classification problems (one for each buffer) is inefficient compared to the single problem specified by Eq 4.
   • Generative Models such as GANs [Goodfellow, et al.], VAEs [Kingma, et al.] and Normalising Flows [Rezende, et al.], etc. that implicitly store the densities in order to generate samples. The density can be extracted from Flow based models but GANs and VAEs require additional modifications to extract the density. These models are more complex and hence introduce additional constraints such as increased data requirements and computational demands.

   Given the above reasons, we believe directly estimating the probability $p^{safe}_{*}$ is a more suitable approach.

2. Thanks for pointing this out. We have added an ablation on the size of the feedback buffer in the attached document (Figure 3). As expected, we notice that the performance of RLSF increases with the size of the feedback buffer as feedback given in the past is not discarded.

3. We agree that a discussion on this point would be insightful and will add it in the revised version. Corollary 1 is true for any threshold $c_{\text{max}}$, since the estimated cost function has an overestimation bias. Consequently, satisfying the $c_{\text{max}}$ threshold can lead to overly conservative policies. To counter this, ideally one could increase the threshold $c_{\text{max}}$ to $c_{\text{max}} + \text{bias}$. Calculating the bias apriori without the knowledge of the ground truth cost function is infeasible. Hence, one could take a heuristic approach and increase the threshold to $c_{max} + \delta$, with $\delta \in \mathbb{R}$. We conducted additional experiments in the SafetyCarCircle environment to test this method (Figure 4 in the attached document), as we observed higher overestimation bias in this setting (Table 1 in the main paper). We observed that adding such a bonus does indeed boost the performance of the proposed method.

4. Novelty-based sampling is a form of uncertainty sampling as it targets states with high epistemic uncertainty—uncertainty arising from a lack of feedback. Epistemic uncertainty is known to adversely affect prediction accuracy [Burda, et al.]. Since we cannot directly measure epistemic uncertainty, we instead analyse the correlation between novelty of trajectories and the model's prediction accuracy on states within those trajectories. This analysis, presented in Figure 5 of the attached document, serves to evaluate the effectiveness of our proposed novelty-based sampling in identifying states with high epistemic uncertainty. The results demonstrate that our novelty-based sampling method does indeed identify states with high predictive errors, which is indicative of high epistemic uncertainty. By eliciting feedback for these states, we can improve the model's accuracy on these states. This further explains the superior performance of our proposed sampling method, as evidenced in Figures 1 and 7 of the main paper.

5. Please note that Corollary 1 is assuming ground truth, $c_{gt}$ information is correct (and not misspecified), the feedback is “sufficient” and is based on the optimal classifier $p^{safe}_{*}$

   We acknowledge that real-world scenarios may deviate from these assumptions:

   • $p^{safe}_{\theta}$ \text{ may not always converge to } $p^{safe}_{*}$ due to factors such as optimization errors, model capacity, etc.

   • Feedback may be missing for some states.

   • Evaluation (meta) noise may occur, particularly with human evaluators. Possible methods to account for this (meta) noise are:

     a) $p^{safe}_{gt}(s,a)=\mathbb{I}[c_{gt}(s,a)=0]$ (noise in safety preferences)

     b) In $y^{safe}$ (noise in labelling process)

   We agree that theoretical results encompassing all these factors would be valuable. However, developing such a comprehensive analysis is non-trivial and beyond the scope of this work. Hence, we have also provided experimental results for cases where these assumptions cannot be satisfied.

Citations:
Burda, Y., et al.. Exploration by random network distillation. ICLR.
Fu, J., et al. EX2: Exploration with exemplar models for deep reinforcement learning. NeurIPS
Goodfellow, I., et al.. Generative adversarial nets. NeurIPS
Kingma, D. P., et al. . Auto-Encoding Variational Bayes. ICLR.
Rezende, D. J., et al. Variational Inference with Normalising Flows. ICML