Thank you for your insightful review and encouraging feedback on our work. We are pleased that our empirical and theoretical contributions were well-received and found promising. We really appreciate your recognition of the paper's strengths. In the following paragraphs, we address your comments and questions in detail.

Baselines and their hyperparameter tuning

To clarify, the CPQ results in Table 1 are based on the original results reported in the DSRL whitepaper, some of which we verified by running CPQ ourselves, yielding comparable results (as shown in Figure 2b). Recent works, such as FISOR, also report similar CPQ performance, further supporting the results provided here. Regarding the reviewer’s concern about standard RL baselines, FISOR is a robust recent baseline that serves this purpose effectively in our comparisons. We agree that detailing the hyperparameters for all baselines in the main text would enhance clarity; however, given space constraints, we propose adding these details to the supplementary materials or appendix and referencing the corresponding works/repos for further implementation specifics.

Safety-performance trade-off in LSPC-S vs LSPC-O

We appreciate the reviewer's insightful comment regarding the comparison between LSPC-S and LSPC-O, particularly the disproportionate increase in cost relative to reward in Table 1. In the hard metadrive experiments, we conducted hyperparameter tuning for LSPC-O to assess potential performance improvements, including increasing the size of the restricted latent safety space and raising the dimensionality of the latent space from 32, as we suspected that this might not be expressive enough for the task's difficulty. However, we found that these adjustments resulted in only increased costs, while reward performance tended to saturate, likely due to the challenging nature of the tasks. Consequently, efforts to enhance rewards often led to similar disproportionately larger increases in cost, as noted by the reviewer. Although our methods significantly outperform other baselines in this set of tasks, we acknowledge that the reviewer’s observation may also reflect some limitations inherent in our proposed framework.

Oscillation of cost return while training LSPC-O

This is an astute observation. While there may be alternative sets of optimal hyperparameters, we opted to use a consistent set across all tasks. We believe that the oscillations are primarily attributed to the model's attempts to find a balance between safety and reward optimization, as you noted.

Firstly, during early training stages, the CVAE decoder, being a neural network, is still undertrained and has limited exposure to samples from the prior distribution. While capable of modeling complex, multimodal distributions of in-distribution actions, the decoder may initially map some samples to out-of-distribution or unsafe actions. This occurs as the latent safety encoder ($\delta$) seeks to maximize rewards by exploiting the inaccuracies in these mappings. Over time, as the decoder encounters a more comprehensive set of samples from the prior, it stabilizes, forcing the encoder to select actions from safer, in-distribution regions.

Secondly, LSPC-S is designed to be conservatively safe. When the latent safety encoder in LSPC-O regresses over this latent space, our findings show that it tends to select actions that are on the higher end of the cost Q-value spectrum. This means that among the actions modeled by the LSPC-S decoder, LSPC-O often chooses actions that correspond to higher cost Q-values. This behavior is intuitive, as LSPC-S represents a conservative policy derived from the offline dataset, and any other policy (including LSPC-O) seeking reward maximization may not achieve the same level of safety.

We think that these factors mainly contribute to the oscillating and higher costs observed for LSPC-O during training.

Page (1/2)

Experimental Concerns

Gap between simulation and real-world application

We acknowledge that simulations may not fully capture real-world uncertainties and could impact the performance of LSPC and any other reinforcement learning method in practical applications. While our main contribution is a general framework that should be applicable to real datasets as well, we have opted to use established simulated environments for comparison with existing baselines to ensure consistency. We recognize that bridging the sim-to-real gap is indeed a challenge and an active area of research as well.

To extend the learned policy from a simulation environment to a real-world deployment, we can calibrate the simulation environment with prior domain knowledge from a real test-bed or environment such that the learned policy can be more adaptive. Other techniques from sim-to-real literature can also be utilized to ensure the latent space to fully capture real-world safety constraints, which can indeed be an interesting direction. Nevertheless, as noted in the conclusion section as a future work, we are committed to exploring the application of LSPC in real-world settings to validate its effectiveness under various uncertainties and dynamic conditions and testing its potential capabilities in sim-to-real applications.

Dataset limitations and bias

Indeed, the reliance on pre-collected data is a fundamental characteristic of offline RL methods, including ours. We acknowledge that this approach may lead to suboptimal performance in situations not represented in the dataset or when the training data exhibits bias towards specific patterns. However, it is important to note that this challenge is not unique to LSPC but is inherent to all offline RL methods. In fact, regularizing the learned policy to remain close to the behavior policy is a common and necessary practice in offline RL to address the issue of distribution shift. While this constraint may limit the policy's ability to generalize to entirely novel situations, it is a crucial safeguard against the potential unsafe behavior that can arise from extrapolating beyond the known data distribution. We appreciate this feedback and recognize the importance of addressing OOD generalization in offline RL.

Interpretation of safety-performance trade-offs

We agree that certain tasks may allow minor safety violations to achieve higher rewards. In fact, our benchmark tasks, as detailed in Appendix A.4, allow different levels of constraint violation cost budget for each trajectory. While our method does not directly operate on this budget, we use a restriction hyperparameter $\epsilon$ that controls how restrictive the latent space is for the optimal policy search. This hyperparameter effectively manages the safety-performance trade-off.

As illustrated in Figure 4, varying levels of $\epsilon$ can change both safety adherence and reward performance. Increasing $\epsilon$ allows the latent safety encoder policy in LSPC-O to explore a larger region within the latent space, potentially finding actions that are better at reward optimization. However, this improved performance comes with a trade-off of higher cost returns. However, this improved performance comes with the trade-off of higher cost returns. As the reviewer noted, policies can be considered 'optimal' for different levels of allowed cost budgets, but when the search space becomes excessively large, this balance may break down, leading to unsafe and unrecoverable behaviors.

As an additional example/ablation, here [ https://imgur.com/a/i4jXnsc ] are the results of training our methods with different levels of restriction on the halfcheetah-velocity environment. In this specific example, the cost return of LSPC-O increases beyond the standard thresholds considered in this work when $\epsilon$ is 0.5 or more. However, LSPC-S which we can always obtain as a backup policy during the LSPC-O training maintains the safety adherence, as observed in the second figure in [ https://imgur.com/a/i4jXnsc ].

To summarize our answer, we provide a flexible framework that can be tuned to different safety requirements while still optimizing for rewards within those constraints.

Page (2/3)

Thank you for your thoughtful review and feedback. We are glad to know that you appreciated the balance between safety and performance, as well as the theoretical analysis in our work. In the following paragraphs, we will try to answer your concerns in detail.

Theoretical Concerns

Capturing real world safety constraints in the latent space

The CVAE in LSPC is designed to capture safety constraints by learning a distribution of safe actions conditioned on observed states, thereby effectively modeling safety within the dataset's scope. While any data-driven model may face limitations in unforeseen situations, future work could address this by incorporating uncertainty estimation or ensemble methods to identify out-of-distribution actions. Within our experiments, the approach demonstrates substantial empirical effectiveness in benchmark simulation environments. Although real-world tests were not conducted in our study, we evaluated robustness by transferring policies across MetaDrive tasks (Appendix A.5) and found that the policies maintained safety constraints, suggesting reliable constraint adherence even under varying task scenarios.

Reliability of constrained optimization and generalization beyond the dataset

In offline RL, selecting in-distribution actions is critical to maintaining reliability, as the agent is limited to the behaviors observed in the dataset and cannot explore. LSPC addresses this by optimizing within a CVAE-trained latent space that models safe, in-distribution actions, helping to ensure that policy actions adhere to the dataset’s safety constraints. This design reduces the risk of unsafe or unpredictable behaviors for out-of-distribution states. Notably, this selection of in-distribution actions is especially crucial in safe offline RL, where unseen actions may pose significant risks of constraint violations that cannot be assessed due to the lack of interaction.

Additionally, the policy is learned through training a parameterized deep learning model, which has been shown to have good generalizability despite its possible algorithmic instability, nonrobustness, and sharp minima (Kawaguchi et al. 2017). Therefore, even for state-action not encountered during training, the optimization in the latent space will remain valid if data distribution drift is not extreme, as CVAE should, to some extent, be able to handle unseen actions during training. In an extreme scenario where most state-action pairs are out-of-distribution, an online fine-tuning method can be adopted quickly to ensure our method to be performant. Alternatively, a robust variant of our proposed algorithm can also be developed to mitigate this issue.

KL Divergence Minimization Between Learned and Behavior Policies and Its Implications

We thank the reviewer for raising such a good point for our theoretical analysis. Indeed, the KL divergence may not always accurately represent the complexities of real-world situations, but it has widely been used in many offline (safe) RL methods, as it is able to prevent drastic deviations from the behavior policy. To enhance the applicability in real-world scenarios, it is crucial to carefully design a behavior policy that captures the desired behaviors and safety constraints a priori. Regardless of the gap between theory and practice, the minimization of KL divergence is still an effective approach to approximately capture the possible real-world scenarios. Also, KL divergence as a regularizer prevents the learned policy from making sudden large jumps away from the behavior policy, leading to more stable learning and smoother policy updates. It also encourages the exploration-exploitation balance by limiting how much the learned policy can diverge from the behavior policy and prioritizing actions that are likely to be successful based on the observed data. It is also a decent metric to handle complex environments where the optimal policy might not be easily identifiable, since the agent learns a policy that is still reasonably close to the observed behaviors.

Page (1/3)

Thank you for the thorough review of our work. We are grateful that you found our approach to balancing safety and reward maximization with a generative model in offline safe reinforcement learning to be an innovative contribution. We also appreciate your recognition of the theoretical foundation and strong empirical results in our work. In the following paragraphs, we will try to answer your review comments and rebuttal questions.

Effectiveness and application in multitask learning

We agree that multi-task reinforcement learning represents an exciting direction for extending the LSPC framework. Such an extension could examine whether the inferred latent safety boundary might serve as a shared safety constraint representation across tasks. However, as of right now, applying the framework across tasks with differing observation and action space dimensions could introduce additional challenges and may require some innovation in the current architecture. In Appendix A.5, we conduct an exploratory analysis of policy transfer across the MetaDrive tasks, observing that policies retain safety when transferred, which we find promising. Your suggestion to investigate variations in safety constraints and reward structures as a broader multi-task application aligns well with our findings and could form an intriguing future line of research. Nevertheless, we want to emphasize that our main contribution and novelty is in safe offline reinforcement learning; this focus on single-task scenarios allows us to thoroughly address the core challenges of safe offline RL, which remain relevant and crucial in learning safe policies from static datasets.

Handling variations in safety constraints and reward structures across tasks

Across different tasks, we can extend the single encoder-decoder model to multiple encoder-decoder models that are able to learn safety constraints tailored to each individual task. Alternatively, we can design hierarchical constraints, which define high-level safety principles that are then transferred into specific constraints for each task. Though the current problem formulation is able to take different safety constraints into account, we have primarily focused on a single task. The extension could be fairly non-trivial in terms of algorithm design. Additionally, due to the diverse natures of reward structures, reward engineering also plays a central role in efficient learning. Some weighting techniques may be required for each reward component. Another appealing method is to establish the multi-objective optimization problem for different tasks and to search for the optimal Pareto front, which presents a set of solutions that are non-dominated to each other but are superior to the rest of solutions in the search space. We believe that addressing the variations in safety constraints and rewards structures across different tasks is of indispensable importance and independent interest, and can be a critical future research direction.

Specific advantages of LSPC across different tasks and performance under different safety constraints

Our approach with LSPC is designed to learn safety constraints from the dataset and represent them in a latent space, allowing for generalized handling of constraints regardless of their specific nature. While we have provided detailed descriptions of tasks and safety constraints in Appendix A.4, we acknowledge that including more details on different safety constraints in the main text would enhance the paper's self-containment and accessibility. Additionally, Figure 3 attempts to visualize the constraints represented by the inferred latent safety boundary in the action space, offering some insight into how LSPC interprets and enforces these constraints.

Page (1/2)

3. Analyzing the dependence of our method on the asymmetric loss function of IQL, particularly for cost management and safety adherence.

We analyzed the impact of varying the coefficient in the asymmetric loss function used for expectile regression in Implicit Q-Learning (IQL) on our LSPC framework. In our original experiments, we used a coefficient of 0.7, which discourages underestimation of cost Q-values. We then compared this to using a coefficient of 0.5, which results in a symmetric Mean Squared Error (MSE) loss, equivalent to SARSA-style policy evaluation. Our findings [ https://imgur.com/a/OzLNuIs ] reveal a trade-off between reward optimization and safety adherence. For LSPC-S, using the symmetric loss ($\xi$=0.5) led to improved reward-wise performance compared to the asymmetric loss ($\xi$=0.7). However, for LSPC-O [ https://imgur.com/a/6yzmTJ1 ], where an additional encoder is trained for reward optimization, using the symmetric loss resulted in higher cost returns, indicating potential safety compromises. These results again highlight the delicate balance between performance and safety in our framework, demonstrating that while less restrictive policies may yield better rewards, they can also lead to increased safety risks when further optimized for performance.

We will include this analysis in the revised version of the paper. If you have any suggestions regarding specific ablation studies that you believe would help clarify the paper's contribution, we would greatly appreciate your input.

Why CVAE among many other generative models

We acknowledge that more recent generative models have shown promising results in various fields, including reinforcement learning, and may indeed improve performance as the reviewer speculates. However, we emphasize that the choice of generative model is orthogonal to our methodology—rather, the purpose of using such a model is for capturing the safety constraints and reducing distribution shift, which the reviewer also noted in the summary. That said, using certain models like diffusion would require adaptations, as diffusion models typically rely solely on decoders and do not inherently include an encoder component. Thus, using these models together with reward-optimizing encoders might involve further methodological innovation and could be an interesting research direction beyond the current work.

Our decision to use CVAE is largely driven by the relatively low-dimensional nature of the tasks at hand, where CVAE proves expressive enough (as demonstrated by the strong empirical results) and allows for a more interpretable and tractable representation of the inferred latent safety boundary (Figures 1b and 3). We will some technical clarifications into the revised draft.

Question regarding performance bound on Lemma 2

We really appreciate this thoughtful comment/question from the reviewer. In Cen’s paper, they have bounded $D_{TV}(d^{\pi}(s)||d^{\*}(s))$ by $D_{TV}(d^{\mathcal{D}}(s)||d^{\*}(s))$ in Lemma 2 based on Assumption 4. This assumption indicates that the stationary state distribution of learned policy is closer to the optimal state distribution than dataset distribution. Therefore, such a replacement makes the bound somewhat looser. Also, in their Theorem 2, they did not provide further analysis on the first term. Instead, in our analysis, we did not use the replacement, but leveraging Lemma 1 to find the relationship between $D_{TV}(d^{\pi}(s)||d^{\*}(s))$ and $D_{TV}(\pi(\cdot|s)||\pi^{\*}(\cdot|s))$ such that we can further proceed the analysis with Lemma 3 in our paper to eventually get a bound with respect to the sample size in Theorem 3 or 4. However, as pointed out by the reviewer, the offline safe RL is more challenging than just offline RL. The tighter bound may be overclaimed in the paper. We agree with the reviewer since the relationship between $D_{TV}(d^{\pi}(s)||d^{\*}(s))$ and $D_{TV}(\pi(\cdot|s)||\pi^{\*}(\cdot|s))$ comes with a cost of $\mathcal{O}(\frac{1}{(1-\gamma)^2})$, instead of $\mathcal{O}(\frac{1}{1-\gamma})$. We will properly state this again in the revised draft and add some necessary clarifications.

Page (3/4)

Ablation Study

Thank you for your constructive comment. We agree that ablation studies are crucial for understanding the impact of individual components of the model on performance. We provide a detailed classification of our hyperparameters in Appendix A.3.1. While ablation studies on IQL and AWR hyperparameters are possible, we would like to note that 1) this line of study has already been explored in depth within the offline RL literature, and such studies generally fall outside the primary scope of our contributions (one could also consider alternative offline RL baselines, such as EQL and XQL, as ablation factors, which would constitute an additional extensive line of investigation); 2) we aimed to keep hyperparameter tuning minimal, opting for standard values that are commonly used across prior offline RL works. Nevertheless, we acknowledge that this particular set of values may not universally apply to every task or dataset.

In our approach, the principal hyperparameter is the restriction parameter $\epsilon$, which we discuss in detail in Appendix A.3.1 with accompanying examples for LSPC-S, LSPC-O and CVAE policy. Additionally, we could perform further ablations on the dimension of the latent safety space. Our chosen dimensionality of 32 seems effective across diverse tasks, likely due to the shared order of dimensionality, which ensures that the latent space is adequately expressive for capturing safety constraints. To address the reviewer's interest in ablation studies, we propose the following:

1. Investigating the effect of reversing the roles of CVAE and safety encoder, where CVAE is trained for reward maximization and the encoder for cost minimization within the inferred latent space.

To investigate the effect of reversing the roles of the CVAE and safety encoder, we train a model with the architecture depicted here [ https://imgur.com/a/Nh68jCw ], which we refer to as Converse LSPC-O. In this configuration, the CVAE is trained for reward maximization, while the encoder focuses on cost minimization within the inferred latent space. We trained Converse LSPC-O for 300k timesteps in the HalfCheetah-velocity environment, varying the restriction level ($\epsilon$) in the latent space. The training curves can be found here [ https://imgur.com/a/LpGLIRA ], and our key findings include:

a. With low $\epsilon$ (high restriction), the encoder ($\delta$) struggles to learn a cost-reducing policy effectively.

b. As $\epsilon$ increases, we observe a modest decrease in cost, which comes at the expense of lower reward returns.

c. While we can further loosen the restriction to get safer policies, very loose restrictions (high $\epsilon$) may yield out-of-distribution (OOD) actions that falsely appear to minimize cost. This is likely due to the decreasing density of samples seen by the decoder during training as we move away from the mean in the latent space.

d. Importantly, the original LSPC framework generally avoids these issues by prioritizing safety; the samples within the inferred LSPC boundaries are both in-distribution and safe. Additionally, the original framework always maintains a conservative policy (LSPC-S) learned during the training of LSPC-O as a backup.

2. Examining how variations in the inverse temperature hyperparameter affect the performance of LSPC-S and whether this influence extends to LSPC-O.

In this ablation experiment, the inverse temperature hyperparameter $\lambda$ dictates the trade-off between behavior regularization and value function optimization. For LSPC-S, a lower $\lambda$ indicates a higher degree of behavior cloning, while a larger $\lambda$ places sharper weight on samples with better cost advantages. To evaluate the impact of this parameter on both reward performance and safety adherence, we trained LSPC-S with $\lambda$ values of 1, 2, and 4 in the CarRun and BallCircle environments. In these training logs [ https://imgur.com/a/Hl1FehM ], LSPC-S with all three $\lambda$ values maintained safety across both tasks; however, higher $\lambda$ values typically resulted in restricted reward-wise performance. This effect was particularly pronounced in the BallCircle [ https://imgur.com/a/0yvkUmj ] environment, where LSPC-S with $\lambda$=1 significantly outperformed those with $\lambda$=2 and $\lambda$=4. The key question then arises regarding what these varying reward-return levels and similar safety adherence imply for LSPC-O. The results in the figure [ https://imgur.com/a/fRNjjan ] indicate that the reward performance of LSPC-O is enhanced when CVAE (or LSPC-S) is trained with a lower $\lambda$, although this improvement is accompanied by an increase in cost returns. While LSPC-S can achieve high reward returns and balance cost returns with lower $\lambda$ values, the encoder trained on top of it to further maximize reward may lead to unsafe policies.

Page (2/4)

Thank you for your thoughtful review and helpful suggestions. We are glad you found our approach effective across benchmarks and our theoretical analysis a meaningful contribution. Below, we address your concerns and questions in detail.

Concerns about strict safety constraints

Thank you for the comment. We agree that satisfying the condition $V_c^\pi(s) \leq \kappa$ is essential for a policy $\pi$ to fully adhere to safety constraints. However, it is worth noting that there may exist a subset of the state space (let us denote it by $\mathcal{S’}_f$) for which no feasible policy exists. Additionally, this infeasible set may intersect with, or be disjoint from, the state distribution in the offline dataset. In this context, one of the most conservative approaches available in offline safe RL is to learn a policy that minimizes $V_c^\pi$, which LSPC-S is specifically designed to achieve (regardless of the specific value of $\kappa$). For any $\kappa$ and any learned $V_c(s)$ using the chosen offline RL method, LSPC-S is expected to produce unsafe actions only when conditioned on states within the infeasible set $\mathcal{S’}_f(\kappa)$. By sampling actions from high-probability regions in the CVAE latent space, LSPC-S provides a conservative policy that reduces the risk of out-of-distribution actions and generates safe actions in expectation.

While LSPC-O similarly samples from this restricted latent space and is theoretically expected to maintain equivalent safety, it may introduce some trade-offs in safety by optimizing for reward-maximizing latent variables rather than random latent samples. Also, we acknowledge that, unlike FISOR [Zheng et al., (2024)], in this work, we primarily consider a soft constraint for safety, instead of a hard constraint, which can be an extension beyond our current work and of independent interest. We appreciate your feedback very much.

Exact sample complexity of the methods

In this work, we provide an approximate sample complexity for the offline dataset that enables us to achieve a safe and optimal solution. The analysis is built upon the stationary state distribution, as the policy learning is based on a given static dataset, and does not depend on the online interactions with an environment, which often requires us to probe particularly the convergence rate in terms of a dynamic metric such as regret [Zhong et al. (2024)] utilized in the stochastic or online optimization area. Though the direct sample complexity analysis can be significantly beneficial to provide a solid theoretical foundation for the proposed algorithm, a thorough investigation on this aspect is currently out of the scope of our study, and still remains extremely challenging and requires a substantial amount of non-trivial efforts. Notably, a few existing works have reported interesting results in only either safe or offline RL settings [Nguyen-Tang et al. (2021); Li et al. (2024); HasanzadeZonuzy et al. (2021); Shi & Chi (2024)], and we have cited them as valuable references for qualitative comparison. However, the approximate sample complexity analysis conducted in our work still offers us meaningful insights on the relationship among the sample size, optimality, and safety, as shown in Theorem 3 and Theorem 4. Additionally, the complexity complies with the existing literature listed above, facilitating the theoretical understanding of offline safe RL.

Page (1/4)