Rethinking Behavior Regularization in Offline Safe RL: A Region-Based Approach

Response to Weakness 3:

We believe the reviewer may have misunderstood the concept of behavior regularization. The purpose of behavior regularization is to minimize the distance between the learned policy and the behavior policy. In general, this distance should be minimized for all states, not just a single state. If an offline dataset is available, the loss can be expressed as:

where $\mathcal{D}$ is the offline dataset of state-action pairs, and $D(\cdot \Vert \cdot)$ represents a divergence measure (e.g., KL divergence).

Since we consider a general optimization problem, the expectation should be taken over a uniform distribution of all states. Minimizing the distance for only a single state, as suggested, is incorrect because when we say two policies are close, we mean they are close across all states.

The theorem can be easily generalized to the case when the initial state distribution is not fixed. Thus, our theoretical result is correct.

Response to Q1:

The term "least region" here is intended to mean "largest." Specifically, we refer to the largest safe region, and all BARS does in this part is to learn and identify that largest safe region. We have clarified this in the revised paper.

Response to Q2:

Same as Response to Weakness 2. Please see General Response. Actually, fine-tuning parameters is never a preferred solution to the problem. To further address the reviewer's concern, we did some additional simulations with tuning the parameters of FISOR. In the original FISOR, weights are split into "safe weight" and "unsafe weight." Following the reviewer's suggestion, we experimented by setting a higher ratio for unsafe weight (1:3 for "safe weight" to "unsafe weight") and increasing the cost temperature parameter to see if these adjustments would enhance FISOR’s performance. However, our experiments on CarButton1, CarButton2, and CarGoal2 (shown in Tables CarButton1 Results, CarButton2 Results, and CarGoal2 Results) indicated that FISOR was still unable to learn a safe policy when data quality was poor, even when we adjusted the parameters.

Furthermore, the FISOR authors themselves state in their appendix (Section E), “However, we find that FISOR is not very sensitive to hyperparameter changes,” suggesting that parameter tuning would not significantly affect the results. Therefore, we conclude that while FISOR performs well on the standard DSRL benchmark [1], it struggles to learn a safe policy in the presence of low-quality data—a limitation observed across all baseline algorithms, not only FISOR. BARS is the first approach to effectively address this issue without requiring extensive hyperparameter tuning.

Table: CarButton1 Results

Table: CarButton2 Results

Table: CarGoal2 Results

We thank the reviewer for the constructive comments and feedback on our paper. We provide the detailed responses separately as follows:

Response to Weakness 1.1:

Our work focuses on the existing issue of using behavior regularization in offline safe RL. FISOR and all the other baselines cannot ensure a safe policy when the behavior policy is suboptimal. Please see our detailed response in General Response.

Response to Weakness 1.2:

We believe the reviewer may have misunderstood the TRPO used in Eq.(12). In our algorithm, we introduce the parameter $K_{\text{trust}}$, which indicates the point after which we optimize the policy using $\text{Eq.(11)}$ and $\text{Eq.(12).}$ This ensures that the current policy does not diverge significantly from the previous one by considering the trust region. This occurs only when the policy is safe enough. Besides, we would like to emphasize that only using Eq.(11) can still ensure learning a safe policy (as shown in Table 4).

In our simulation, we simply set $K_{\text{trust}}$ to half of the total training episodes without any tuning. Additionally, we would like to clarify that in $\text{Eq.(12)}$, we do not use behavior regularization from the offline dataset; instead, we employ the previously trained policy (target policy) for regularization. The parameter $k$specifies how often we update the target policy, analogous to the target network updates used in DQN. We have clarified these points in our revision. This approach does not violate our formulation.

Response to Bellman equation:

In the paper, we identified a typo in the Bellman equation (we missed the discount factor). The correct form is:

The term $(1-\alpha)$ is included to model the scenario where, with probability $1 - \alpha$, an episode ends after the current time step (e.g., after taking a very unsafe action). It is easy to show that $c(s,a) + \gamma \check{V}_c(s') = \check{Q}_c(s,a)$ as $\alpha \rightarrow 1$ (we use $\alpha = 0.99$ in our simulation). Importantly, FISOR defines the feasible region using the value function for the hard constraint based on Hamilton-Jacobi (HJ) reachability from control theory (hence the inclusion of the maximization term). However, this formulation applies only to deterministic systems, which are rarely encountered in most RL environments, and cannot be generalized to problems with more flexible cost limits. As we indicated in our paper, the Bellman equation used in FISOR to train the value function does not hold in stochastic systems.

Response to Weakness 1.4:

Here, $b_1$ and $b_2$ act as loss ratio controls, while $\mu_1$ and $\mu_2$ serve as temperature parameters to regulate behavior regularization. Introducing $b_1$ and $b_2$ allows us to easily adjust the focus on the safe or unsafe loss components.

Response to Weakness 1.5:

Here, the value of the threshold $\delta$ is the same as the cost limit $l$ in our simulation, which means $\delta = l > 1, l = 10, 20, 40$. Therefore, the denominator issue does not arise (when $Q^{\pi}_c(s,a) > \delta$, it cannot be small or close to zero), and the existing setup is adequate.

Response to Weakness 1.6:

Our work focuses on the existing issue of using behavior regularization in offline safe RL. FISOR and all the other baselines cannot ensure a safe policy when the behavior policy is suboptimal. Please see our detailed response in General Response.

Response to Weakness 2:

We would like to emphasize that in our experiments, we did not perform a hyperparameter search in our simulations. Instead, we used a single set of loss control parameters, $b_1 = 1$ and $b_2 = 3$, for all tasks in Table 2 and most tasks in Table 1. The only exceptions were "AntVel" and "HalfCheetah" in Table 1, where we set $b_1 = 3$ and $b_2 = 1$ to achieve higher rewards. Please see details in General Response. Actually, fine-tuning parameters is not a viable solution to the problem; the issue must be addressed at the logical level.

Response to Practical application issues:

Time-consuming: We would first like to acknowledge FISOR for providing an approach that reduces training costs. Their novel energy-guided sampling method simplifies the training process by eliminating the need for a complex time-dependent classifier. Additionally, our implementation leverages the JAX framework, which offers a significant speed advantage compared to other frameworks like PyTorch and TensorFlow. In practice, training for 1 million steps takes approximately 50 minutes on a single RTX 4090 GPU, enabling high efficiency and scalability (Table Training Steps/Minutes).

Table: Training Steps/Minutes

No safety guarantees: Regarding the "train-collect-train" framework, we highlight the improved quality of the new data, demonstrating that our policy adheres to safety requirements, as the mean episode cost is consistently lower than the cost limit 10.

Table: Episode and Total Costs Comparison

Response to Q1:

Since the regions are plotted based on the trained $\check{V}_c$, and (b), (c), and (d) use three different networks, it is expected that the plotted regions may show minor differences.

Response to Q2:

The value "8" is simply a static configuration in the file. When we uploaded the file, we did not provide a complete running script. In practical training, we set $\delta = l = \text{cost limit}$.

Response to Q3:

FISOR is designed to be cost limit-agnostic due to its reliance on a hard constraint framework. However, as we indicated in our paper, this design limits the generalization of their approach to other scenarios where a specific cost limit is considered, which is a more practical setting. Our aim is to demonstrate that increasing the cost limit should potentially lead to better performance for an algorithm. We do not understand why the reviewer considers this to be a weakness of our paper, especially since FISOR is one of the most closely related works and it is reasonable to add more discussion.

We thank the reviewer for the constructive comments and feedback on our paper. We provide the detailed responses separately as follows:

Response to OOD issue:

OOD actions can be managed by applying behavior regularization in the safe region. In the unsafe region, we directly minimize the cost function, which is trained using IQL. We thank the reviewer for raising this question, as it highlights the potential for further improvement by incorporating a penalty term into the critic network (similar to CPQ) to enhance control over OOD actions.

Response to Bellman equation:

In the paper, we identified a typo in the Bellman equation (we missed the discount factor). The correct form is:

The term $(1-\alpha)$ is included to model the scenario where, with probability $1 - \alpha$, an episode ends after the current time step (e.g., after taking a very unsafe action). It is easy to show that $c(s,a) + \gamma \check{V}_c(s') = \check{Q}_c(s,a)$ as $\alpha \rightarrow 1$ (we use $\alpha = 0.99$ in our simulation). Importantly, FISOR defines the feasible region using the value function for the hard constraint based on Hamilton-Jacobi (HJ) reachability from control theory (hence the inclusion of the maximization term). However, this formulation applies only to deterministic systems, which are rarely encountered in most RL environments, and cannot be generalized to problems with more flexible cost limits. As we indicated in our paper, the Bellman equation used in FISOR to train the value function does not hold in stochastic systems.

Response to Weakness 3:

The transformation from Eq. (5) to Eq. (9) involves:

1. Eq. (9) introduces $A_r^\pi(s, a)$ and $\check{A}_c(s, a)$ to explicitly measure the impact of actions on the optimization objective.
2. The constraint $\int_a \pi(a|s) \, da = 1$ ensures that the policy $\pi(a|s)$ is a valid probability distribution over actions for each state $s$.
3. Instead of using $V_c^\pi(s) \leq l$ as in Eq. (5), Eq. (9) refines this by constraining individual actions $\\{a | \check{Q}_c(s, a) \leq l \\}$, ensuring that every action chosen adheres to the cost limit.

We thank the reviewer for the constructive comments and feedback on our paper. We provide the detailed responses separately as follows:

Response to lower costs and lower rewards:

We would like to emphasize that in safe RL, ensuring a safe policy (one with a cost lower than the threshold) is always the top priority. This is also a common observation in the offline safe RL benchmark (Liu et al., 2023a). We believe one reason for this is that the cost functions designed for different environments vary significantly, and balancing the reward and cost functions often requires environment-dependent parameters and careful fine-tuning. In our approach, since BARS uses a consistent set of hyperparameters across all environments in Table 2, we do not perform environment-specific fine-tuning. However, we can ensure a safe policy across all simulations. We believe that developing an adaptive hyperparameter selection method for different environments is an interesting direction for future work.

Response to Q1:

Thanks for your suggestion. We implemented the BC-SAFE using the same dataset of CarButton1 we used in Table 2 to complete a comparison (Table BC-SAFE Comparison). The result shows that BC-SAFE also fails to learn a safe policy.

Table: BC-SAFE Comparison

Response to Q2:

In offline RL, the dataset is collected by some behavior policy. Thus, the optimization problem is to maximize the policy (for both RL and safe-RL) using only the offline dataset. The expectation is therefore taken over the behavior policy, which can be written as $(s, a, s') \sim \mathcal{D}$. This is the reason why the distribution shift issue exists in offline RL—because the distribution of the dataset is different from the distribution under the policy we aim to optimize. We refer the reviewer to a more detailed explanation provided in this paper [R1].

[R1] Levine, Sergey, et al. "Offline reinforcement learning: Tutorial, review, and perspectives on open problems." arXiv preprint arXiv:2005.01643 (2020).

Response to Q3:

The value of the threshold $\delta$ is the same as the cost limit $l$ in our simulation. We define it as $\delta$ in a more general form to control the robustness of the algorithm, as the $Q$-functions may not be accurate and this is likely to hold true in practice.

Response to OOD issue:

The OOD actions can be controlled by adding behavior regularization in the safe region. In the unsafe region, we directly minimize the cost function, which is trained using IQL. We thank the reviewer for raising this question, as it opens the possibility of further improvements by incorporating a penalty term into the critic network (as CPQ) to better control OOD actions.

Response to Q5:

In the following table, we report the average cost per episode (cost limit $l = 10$) based on $600,000$ data samples with 600 trajectories for the original offline dataset, the first collection (after training for $200K$ steps), and the second collection (after training for $400K$ steps). We can clearly observe that the mean episode cost drops dramatically after each new collection, which implies that the policy is safe enough for exploration.

Table: Average Episode Cost

Response to Q2:

Thank you for your suggestion. Given the limited time of rebuttal, we conducted two additional baselines, BCQ-Lag and BEAR-Lag, on CarGoal1 and CarGoal2 to evaluate their performance under different data distributions, as shown in Tables BEAR-Lag Results and BCQ-Lag Results. The results demonstrate that both baselines fail to learn a safe policy, further highlighting the strong performance of BARS. We will add the results for other environments in the final revision.

Table: BEAR-Lag Results

Table: BCQ-Lag Results

Response to Q3:

In our experiments, we use the parameters provided in Appendix Table 7 and did not perform extensive tuning on these parameters. Please see the details in General Response.

Response to Q4:

We refer to the result under FISOR as $(0.268, 0.72)$ , which is slightly worse than what they reported in their paper $(0.30, 0.35)$.

Response to Q5:

Thanks for your question. We give a more clear summary of our contributions in the revised paper. The key components are:

1. $\text{Eq.(5)}$, where we remove the behavior regularization for the unsafe region.
2. The Region-Based Risk-Reward Optimization technique in $\text{Eq.(8)}$.

To further validate our approach, we conducted an experiment (Table Component Analysis) in the CarButton2 environment (data ratio 1:1 in Table 2) to demonstrate that our key components are essential for learning a safe policy under different data distributions.

In this experiment, we divide the following four situations and report the results in Table Component Analysis:

1. (i) ${D}(\pi||\pi_{\beta})\le\epsilon$ only in the safe region.
2. (ii) ${D}(\pi||\pi_{\beta})\le\epsilon$ both in safe and unsafe regions.
3. (a) $\frac{Q_r^\pi(s,a)}{Q_c^\pi(s,a)} \cdot 1_{\{Q_c^\pi(s,a) > \delta \}} + Q_r^\pi(s,a) \cdot 1_{\{Q_c^\pi(s,a) \leq \delta \}}$.
4. (b) ${Q_r^\pi(s,a)} \cdot 1_{\{Q_c^\pi(s,a) > \delta \}} + Q_r^\pi(s,a) \cdot 1_{\{Q_c^\pi(s,a) \leq \delta \}}$.

This experiment shows that using (ii) cannot assure a safe policy, and component (i) is the crucial factor to guarantee a safe policy. The results highlight how each component of BARS contributes to learning a safer policy.

Table: Component Analysis

Our method can be applied to any region-based approach that distinguishes between safe and unsafe regions. Since BCQ-Lag and CDT are not region-based approaches, they cannot leverage this technique to enhance their performance.

Response to Q6:

Thanks for the suggestion, we have made it more clear in the updated revision.

We thank the reviewer for the constructive comments and feedback on our paper. We provide the detailed responses separately as follows:

We have carefully polished our paper and fixed the typos.

Response to Weakness 2:

Thanks for your suggestion. We implemented the BC-SAFE using the same dataset of CarButton1 we used in Table 2 to complete a comparison (Table BC-SAFE Comparison). The result shows that BC-SAFE also fails to learn a safe policy. Additionally, we plan to include CDT in the final revision. However, due to the limited time, we could not complete all the simulations. Furthermore, we believe CDT is also unlikely to achieve a safe policy when the behavior policy is poor. As reported in FISOR's paper and (Liu et al., 2023a), CDT cannot guarantee a safe policy, while FISOR demonstrates the best results among current methods.

Table: BC-SAFE Comparison

Response to Weakness 3:

The motivation for new data collection stems from the fact that the performance of offline RL algorithms is highly dependent on the quality of the dataset. A suboptimal behavior policy can limit the ability to learn the optimal policy. In offline Safe-RL, collecting new data is risky unless a reasonable and reliable safe policy is available (this is not an issue in unconstrained RL). Therefore, we propose a framework to demonstrate a potential way of gradually and safely learning a safe policy. We do not compare our approach with existing offline-to-online algorithms because our focus is not on solving the problem of pre-trained offline policies with online fine-tuning and a limited number of interactions. This represents a different setting.

In the following table, we report the average cost per episode (cost limit $l = 10$) based on $600,000$ data samples with 600 trajectories for the original offline dataset, the first collection (after training for $200K$ steps), and the second collection (after training for $400K$ steps). We can clearly observe that the mean episode cost drops dramatically after each new collection, which implies that the policy is safe enough for exploration.

Table: Average Episode Cost

Response to different as FISOR:

Our work focuses on the existing issue of using behavior regularization in offline safe RL. FISOR and all the other baselines cannot ensure a safe policy when the behavior policy is suboptimal. Please see our detailed response in General Response.

Response to Q1:

The weighted loss function is specifically designed for diffusion settings. CPQ and COptiDICE are not diffusion-based approaches, so they cannot be directly applied with a weighted loss. Additionally, BARS incorporates several key components to ensure that the modified optimization problem can be efficiently solved. Simply adding the weighted loss to their algorithms may not work effectively.

We thank the reviewer for the constructive comments and feedback on our paper. We provide the detailed responses separately as follows:

Response to Bellman equation:

In the paper, we identified a typo in the Bellman equation (we missed the discount factor). The correct form is:

The term $(1-\alpha)$ is included to model the scenario where, with probability $1 - \alpha$, an episode ends after the current time step (e.g., after taking a very unsafe action). It is easy to show that $c(s,a) + \gamma \check{V}_c(s') = \check{Q}_c(s,a)$ as $\alpha \rightarrow 1$ (we use $\alpha = 0.99$ in our simulation). Importantly, FISOR defines the feasible region using the value function for the hard constraint based on Hamilton-Jacobi (HJ) reachability from control theory (hence the inclusion of the maximization term). However, this formulation applies only to deterministic systems, which are rarely encountered in most RL environments, and cannot be generalized to problems with more flexible cost limits. As we indicated in our paper, the Bellman equation used in FISOR to train the value function does not hold in stochastic systems.

Response to definition of $A^{\pi}_r$:

We would like to emphasize that the definition of $A_r^\pi$ is not ad hoc; it directly follows from our theoretical formulation with practical reasoning. Due to the difficulty in obtaining accurate estimates of $Q^\pi_r$ and $Q^\pi_c$ (which is likely to be true in practice due to the use of neural networks), we implemented a practical approach to penalize the reward $Q$-function in $\text{Eq.(8)}$. Specifically, we penalize the reward $Q_r^\pi$-function based on the cost $Q_c^\pi$-value under the current policy whenever $Q_c^\pi(s, a) > \delta$, where $\delta$ is the cost limit (other parameters can also be used to control the robustness of the algorithm). This indicates that the current policy is not safe. When the policy is safe, we directly optimize $Q_r^\pi$. Given that the estimation may not be accurate for both $Q$-functions, we penalize the reward $Q$-function according to the magnitude of the cost $Q$-function, assuming that the trend of the $Q$-values is well learned, even if the estimation is not precise.

Response to bootstrapping error accumulation:

As described in our algorithm, it is divided into two distinct stages. First, we learn the largest safe region, which is determined by the most conservative policy. Specifically, this ensures that for any state within this region, there exists no unsafe policy. Using IQL to learn the value function and achieve $\check{V}_c(s) = \min_a \check{Q}_c(s, a)$ under the most conservative policy is both straightforward and efficient. This approach can be implemented using offline data only, avoiding the need to train the policy network. As a result, it provides an accurate estimation, is time-efficient, and eliminates bootstrapping error issues.

We believe the reviewer may have misunderstood the TRPO used in $\text{Eq.(12)}$. In our algorithm, we introduce the parameter $K_{\text{trust}}$, which indicates the point after which we optimize the policy using $\text{Eq.(11)}$ and $\text{Eq.(12)}$. This ensures that the current policy does not diverge significantly from the previous one by considering the trust region. This occurs only when the policy is safe enough. Besides, we would like to emphasize whether only using $\text{Eq.(11)}$ can still ensure learning a safe policy (as shown in Table 4).

In our simulation, we simply set $K_{\text{trust}}$ to half of the total training episodes without any tuning. Additionally, we would like to clarify that in $\text{Eq.(12)}$, we do not use behavior regularization from the offline dataset; instead, we employ the previously trained policy (target policy) for regularization. The parameter $k$ specifies how often we update the target policy, analogous to the target network updates used in DQN. We have clarified these points in our revision. This approach does not violate our formulation.

Thanks for the responses. However, the concerns are still unresolved.

1. Bellman Equation. The authors argure that they adopt the Discounted Reach-Avoid Bellman Equation, which is $\mathbb{B}(Q^*(s,a))= (1-\gamma) h(s) + \gamma (\max (h(s), V^*(s)))$. Note that this original equation has a max term, so the $(1-\gamma) h(s) + \gamma (\max (h(s), V^*(s)))$ will not degenerate to $h(s)+\gamma V^*(s)$. However, the authors just naively adopt this equation, where the $-\alpha c$ and $\alpha c$ in Eq. (6) will cancel out, making Eq. (6) just $c+\gamma \times \alpha V$, which seems that the carefully designed $\alpha$ term just adjust the discounting factor, rather than being the termination probability argued by the authors. I noticed the reviewer RHPd also pointed out this concern.

2. Bootstrapping Error. I disagree with the author that just adopting expectile regression to learn the value function while not regularizing the policy can avoid OOD issues. The value function in IQL is learned through in-sample learning, which can avoid bootstrapping, but still has some overestimated OOD regions. The value is stable since the value learning and policy learning is decoupled, and the value function will never querry these OOD regions. However, these OOD regions EXIST. So, during policy optimization, IQL still requires behavior regularization (AWR, which equivalents to a implicit policy regularization w.r.t a KL-divergence constriant and the objective is $\min_\theta - \mathbb{E}_{(s,a)\sim\mathcal{D}}(\log \pi(a|s)exp(A)$. Note all data are sampled from the dataset $\mathcal{D}$) to prevent exploiting on these OOD regions. However, according to Eq. (10), the authors try to directly optimize the cost value function without any regularization, and the objective is $\min\mathbb{E}\_{s\sim\mathcal{D}, a\sim\pi} exp(A)$. Note that here, the action $a$ is sampled from the policy $\pi$, and this objective has NO regularization. So, I can tell that the policy optimization obejective is not IQL as the author said: we use IQL to directly optimize the policy.I implemented a lot of offline RL methods and their variants, but found no one can obtain stable results without any behavior regularization when directly maximizing the Q value obtained by IQL.

3. Weakness 7. Yes, FISOR is one cost-limit agnostic method. But, how can the authors get totally different results (the costs are varied from 0.06 to 0.36) for different cost limits in their first submission. After I pointed this out, the authors directly removed this result, without comparing to any baselines. There are some safe offline RL methods that can handle varied cost limits, so why did not the authors compare against these baselines? Also, I noticed that the FISOR results reported by the authors are not consistent with the original FISOR paper, which is also noticed by the reviewer Yqdv. I found one recent ICLR submission (Constraint-Conditioned Actor-Critic for Offline Safe Reinforcement Learning) that also reproduced FISOR and reported far more better results, even compared to the original ones. So, this paper potentially not well reproduced the baselines and the experimental results are not convincing.

So, regretfully, I cannot raise my score and recommend rejection for this paper.

We thank the reviewer for the constructive comments and feedback on our paper. We provide the detailed responses separately as follows:

Response to different as FISOR

Our work focuses on the existing issue of using behavior regularization in offline safe RL. FISOR and all the other baselines cannot ensure a safe policy when the behavior policy is suboptimal. Please see our detailed response in General Response.

Response to "pointing out which component or improvement of BARS"

We claim that the new optimization framework we proposed in $\text{Eq.(5)}$ is the key component in mitigating regularization issues. To further validate our approach, we conducted an experiment (Table Component Analysis) in the CarButton2 environment (data ratio 1:1 in Table 2) to demonstrate that our key components are essential for learning a safe policy under different data distributions.

In this experiment, we divide the following four situations and report the results in Table Component Analysis:

1. (i) ${D}(\pi||\pi_{\beta})\le\epsilon$ only in the safe region.
2. (ii) ${D}(\pi||\pi_{\beta})\le\epsilon$ both in safe and unsafe regions.
3. (a) $\frac{Q_r^\pi(s,a)}{Q_c^\pi(s,a)} \cdot 1_{\{Q_c^\pi(s,a) > \delta \}} + Q_r^\pi(s,a) \cdot 1_{\{Q_c^\pi(s,a) \leq \delta \}}$.
4. (b) ${Q_r^\pi(s,a)} \cdot 1_{\{Q_c^\pi(s,a) > \delta \}} + Q_r^\pi(s,a) \cdot 1_{\{Q_c^\pi(s,a) \leq \delta \}}$.

This experiment shows that using (ii) cannot assure a safe policy, and component (i) is the crucial factor to guarantee a safe policy. The results highlight how each component of BARS contributes to learning a safer policy.

Table: Component Analysis

Response to "summary of contributions" and limitations

Thanks for the suggestion, we revised the contribution in the revised draft. Here, we will provide a brief summary of our contributions.

1. We propose a new optimization framework that rethinks the need for behavior regularization in offline safe reinforcement learning. First, we determine safe and unsafe regions using the correct formulation of the Bellman equation, then remove behavior regularization in unsafe regions. This framework helps the agent learn safe actions across various data distributions, even when some datasets are of low quality.
2. We introduce a cautious updating method called "Region-Based Risk-Reward Optimization," which enables the agent to avoid unsafe actions, even when starting in a safe region.

Response to $A_r^\pi(s, a)$

We penalize the reward $Q_r^\pi$-function based on the cost $Q_c^\pi$-value whenever $Q_c^\pi(s, a) > \delta$ (where $\delta$ is the cost limit). This indicates that the policy is unsafe. When the policy is safe, we directly optimize $Q_r^\pi$.

Given that $Q$-function estimation may not always be accurate, we penalize $Q_r^\pi$ based on the cost $Q_c^\pi$, assuming the $Q$-value trends are well-learned even if the estimation isn't precise. We do not include a neural network for $V_r^\pi$, as we consider it redundant, though it would work if trained. This choice avoids redundancy as both advantage and $Q$-networks have been extensively discussed in RL literature.

Response to distributional shifts

If no distributional shifts exist, behavior regularization isn't required when learning the policy. This is only true when the behavior policy closely resembles the optimal policy. In $\text{Eq.(11)}$, the regularization term is not required, as discussed in our optimization problem in $\text{Eq.(5)}$.

Response to $\delta$, $k$, and $K_{trust}$

• The threshold $\delta$ is the same as the cost limit $l$ in our simulation but is defined in a general form to control the algorithm's robustness since $Q$-functions may be inaccurate in practice.
• $K_{\text{trust}}$ ensures the policy does not diverge significantly from the previous one. It is applied only when the policy is safe enough. In our simulation, $K_{\text{trust}}$ is set as half the total training episodes without tuning.
• $k$ specifies the update frequency for the target policy, similar to target network updates in DQN.

Finally, we emphasize that not using $\text{Eq.(12)}$ can still ensure learning a safe policy (as shown in Table 4).

We thank the reviewer for the constructive comments and positive feedback on our paper. We provide the detailed responses separately as follows:

Difference between BARS and FISOR

Our work focuses on the existing issue of using behavior regularization in offline safe RL. FISOR and all other baselines cannot ensure a safe policy when the behavior policy is suboptimal. Please see our detailed response in General Response.

Response to lower costs and lower rewards

We would like to emphasize that in safe RL, ensuring a safe policy (one with a cost lower than the threshold) is always the top priority. This is a common observation in the offline safe RL benchmark (Liu et al., 2023a).

We believe one reason for this is that cost functions vary significantly across environments, and balancing reward and cost functions often requires environment-dependent parameters and careful fine-tuning. In our approach, BARS uses a consistent set of hyperparameters across all environments in Table 2 without environment-specific fine-tuning. However, we can ensure a safe policy across all simulations. Developing an adaptive hyperparameter selection method for different environments is an interesting direction for future work.

Response to "summary of contributions" and limitations

Thank you for the suggestion. We have revised the contribution section in the introduction and added a limitations section in the revision.

Summary of Contributions:

• We propose a new optimization framework that addresses the existing issues of using behavior regularization in offline safe reinforcement learning.
• We introduce a practical algorithm, BARS, which builds on our theoretical findings and enables the agent to optimize the policy separately in different regions, determined by the feasible region.
• We determine safe and unsafe regions using the correct formulation of the Bellman equation, then remove behavior regularization in unsafe regions. This framework helps the agent learn safe actions across various data distributions, even when some datasets are of low quality.

Response to Hyperparameters

We emphasize that in our experiments, we did not perform a hyperparameter search. Instead, we used a single set of loss control parameters, $b_1 = 1$ and $b_2 = 3$, for all tasks in Table 2 and most tasks in Table 1. The exceptions were "AntVel" and "HalfCheetah" in Table 1, where we set $b_1 = 3$ and $b_2 = 1$ to achieve higher rewards. As demonstrated, no other baselines can ensure safety due to the regularization issue we highlighted. Please see our detailed response in General Response.

Response to using IQL

As indicated in our paper, solving the optimization problem in $\text{Eq.(4)}$ is challenging because:

1. The $Q$-networks and policy are parameterized by neural networks, which introduces instability issues.
2. Accurate estimation is difficult, requiring a long training period even for evaluating a single policy.

To address these, we solve an alternative optimization problem under the largest safe region, determined by the most conservative policy. This ensures that for any state within this region, no unsafe policy exists. Using IQL to learn the value function and achieve $\check{V}_c(s) = \min_a \check{Q}_c(s, a)$ under the most conservative policy is both straightforward and efficient. This method uses offline data only, avoiding policy network training, providing accurate estimation, being time-efficient, and eliminating instability.

Response to $u$ in IQL

In our loss function, we prioritize $u > 0$ as it learns the value function under the most conservative policy. This is a minimization problem rather than the maximization problem used in the IQL paper. Therefore, the asymmetric loss function operates in the opposite direction. We have clarified this point in the revised paper.

Thanks for the authors' response. I am convinced by the authors and have no more concerns. Although other reviewers have some concerns, I believe this work can provide some insights into the offline safe RL field. After reading the authors' responses to other reviewers, I hold a positive attitude that they can address their problems to some extent. The author should declare their core contribution and clarify their motivation for designing the experiment to convince other reviewers. I tend to raise my score but I will first wait for other reviewers' responses.