Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning

Thank you for your comments on our paper. Please find our point-by-point response to your questions below.

• Response to contributions: We respectfully ask the reviewer to evaluate our theoretical contributions. We would like to mention that our approach has significant differences from ATAC. It is also unfair to say the "primary differences are the authors' focus on the safe offline RL setting and the inclusion of a cost value in their theory," since all existing studies in safe RL consider the cost function in the formulation of RL, it is the default setting. This consideration makes the problem entirely different and more difficult than the unconstrained case; i.e., the optimal policy will no longer be a greedy policy, how to ensure safe learning and obtain a safe policy, and balancing the reward and cost is extremely important. We are the first to present a Safe Robust policy improvement over any reference policy with an optimal statistical rate. Following the approaches in ATAC can only guarantee a sub-optimal rate, and it is highly non-trivial to extend their results to the constrained setting.

• Response to compare with baselines: We thank the reviewer for pointing out the papers CDT and FISOR. The reason we did not incorporate these two algorithms as additional baselines is that their setups do not align well with ours. CDT introduces additional information, target reward, and target cost, during evaluation. These pieces of information are not required in the evaluation process of WSAC and the other baselines we have chosen. Additionally, FISOR considers a different setting from ours as they consider a different type of constraint which is defined for any possible state. For the addition simulations with different cost limits, we report the average performance of our results and other baselines in the following Table (Table 1) under cost limits [10,20,40] for BallCircle and CarCircle and [20, 40, 80] for PointButton and PointPush following the standard setup ([R1]). We can observe that WSAC maintains safety across all environments, and WSAC's performance is comparable to or even better than the best baseline in each environment. We will add the details in the final revision.

Table 1: The normalized reward and cost of WSAC and other baselines for different cost limits. Each value is averaged over 3 distinct cost limits, 20 evaluation episodes, and 3 random seeds.

[R1] uxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu,114 Tingnan Zhang, Jie Tan, et al. "Datasets and benchmarks for offline safe reinforcement learning". arXiv preprint115 arXiv:2306.09303, 2023.116

Averaging the number of safety constraint violations

There seems to be some confusion. In our paper, we used a single cost limit (thus, there was no point in taking an average) similar to what the baselines (CDT, COptiDICE, CPQ) did in their original papers. The average is taken over different random seeds, but we used a single cost limit. We believe that the results over different random seeds in Table 2 can correctly reflect the true performance of our approach. From Table 2, it is clear for the single cost limit, our approach is the only one that can satisfy the constraints across different benchmarks.

During the rebuttal phase, per the reviewer's request, we ran the algorithm with different cost limits following the exact format used in the Offline Safe RL Benchmark (OSRL) paper, where they report the average across performance with different cost limits. It is clear that our algorithm performs well, as no other algorithms are consistently safe, in terms of average performance. To further address the reviewer’s concern, we report individual results with different cost limits under our algorithm in the following table, where our algorithm achieves very low costs and high safety rates and is nearly safe for all environments. Note that due to time constraints in the rebuttal phase, we did not perform any parameter tuning, and we believe we can further improve the performance (both in terms of reward and safety) if we do so. Note that, we can't compare with other baselines since the OSRL paper doesn't have the results for different cost limits (they only have the average results). We also believe that although one of our main contributions is theoretical, our approach with theoretical support is quite general and has the potential to be incorporated into other practical Safe-RL algorithms.

Table 1: The normalized reward and cost of WSAC for different cost limits. Each value is averaged over 20 evaluation episodes, and 3 random seeds.

limited subset of environments

We focus on environments where most (if not all) baselines are unsafe (e.g., no baselines are safe in PointButton, and only BCQL is safe in PointPush). Yet, we show that our proposed approach can achieve safety while maintaining good reward which points towards its efficacy. We believe these environments are both challenging and representative, effectively justifying an algorithm's ability to guarantee safety.

We would like to emphasize that our main contribution in this paper is to address the Safe Robust Policy Iteration (SRPI) and policy coverage limitations in theoretical offline safe RL which are quite important in the theoretical safe-RL community (see Table 1). For example, none of the existing approaches (including the baselines) have a safe robust policy improvement guarantee using only a single policy coverage assumption. In particular, our approach provides a way to achieve a safe policy using only offline data with bare minimum richness (single policy coverability). Existing approaches that are based on primal-dual concept require more richness in data (all policy concentrability which is not possible to achieve in practice). Please see Table 1 and discussion in Introduction. Furthermore, the baselines do not provide any theoretical guarantees, since they aim to design practical algorithms. To demonstrate the empirical efficiency of our approach, we included four challenging environments in our paper. We agree that running the algorithm on all 38 environments in the baseline Safe RL paper would surely have values, and we will try to evaluate our approach on more baselines in the final version. However, we believe that the environments we selected are sufficient to demonstrate the core ideas of our paper and validate the theoretical insights. Even the state-of-the-art algorithms (without theoretical guarantees) only include a limited number of representative environments in their papers; for instance, CDT has 5, CPQ has 3, and COptiDICE has 4.

We sincerely hope that the reviewer will reevaluate the rating based on the novel contributions of our paper.

Thanks for increasing your score and engaging with us.

Technical Differences with ATAC ATAC is the first paper in the literature to investigate the property of RPI. While we certainly draw insights from their results, our work has the following significant differences.

• First, we focus on the constrained Markov decision process (CMDP) setup rather than an unconstrained setup. The CMDP setup is fundamentally different from the unconstrained one. For example, in CMDP, the optimal policy can be stochastic, unlike in the unconstrained MDP. In the constrained setup, it is essential to bound both the sub-optimality of the reward and the constraint violation, whereas, in the unconstrained setup, only the sub-optimality of the reward needs to be bounded. Naturally, the analytical results and algorithms differ significantly from those in the unconstrained ATAC setup.

• Furthermore, our approach to training the critics is different from that of ATAC. ATAC uses a squared Bellman error, while we utilize the average Bellman error to train the critic. Consequently, we achieve a $1/\sqrt{N}$ sample complexity error, while ATAC achieves a $1/N^{1/3}$ sample complexity error. The key difference is that we use an importance-weighted Bellman error to obtain an unbiased estimator of the critic for both the reward and cost, tuning the weight parameter to achieve a better rate, unlike ATAC.

• Moreover, while primal-dual-based methods exist for solving the offline CMDP, achieving robust safe policy improvement and relaxing the assumption of all-policy concentrability remained open challenges (Table 1 in our paper). We resolved this open question. Our approach guarantees robust policy improvement, so if a safe reference policy (e.g., a safe behavioral policy) is provided, our algorithm will yield a policy that remains safe without sacrificing reward. Such a guarantee was previously missing from the literature. As pointed out in the introduction, all-policy concentrability is difficult to satisfy in practice, especially in a safe setting where the dataset may not cover state-action pairs from an unsafe policy. Instead, we only require single-policy concentrability, making our theoretical results highly impactful for the safe RL community.

• We consider a policy improvement over any reference policy not only the behavior policy.

• It is worth noting that to provide such a guarantee, we developed a rectified penalty-based approach rather than a primal-dual-based one. As a result, our analysis differs from existing primal-dual approaches. In fact, the existing primal-dual-based approaches only guarantee all-policy concentrability, so our analytical insights and proposed approach open new avenues for finding a safe policy from an offline dataset.

compare with the SOTA baselines

it is crucial to compare with the state-of-the-art (SOTA) baselines to demonstrate the strength of our proposed approach. Therefore, we compare our practical version with existing approaches on selected benchmarks. As requested by the reviewer, we have included results from CDT in our rebuttal. We apologize for not making this clearer earlier. Notably, ours is the only approach that achieves safety, underscoring the efficacy of our method. Additionally, CDT uses a transformer architecture, which naturally results in a longer computational time compared to our approach.

Finally, we would like to mention that in safe RL, there are two types of constraints: soft constraints (in the long-term average sense) and hard constraints (step-wise). It is difficult to say which one is more important because, in the long-term average case, taking some risk is necessary; otherwise, the problem would be no different from an unconstrained problem. Moreover, the existing solutions in theoretical safe RL for addressing these two types of constraints are significantly different. We agree that the cost limits are chosen by humans, and we appreciate the reviewer pointing out that a fairer comparison should consider different sets of cost limits. We observe that there is a trend: the reward increases when the cost limit is higher. To understand the differences between two settings, we can also observe that as reported in the FISOR paper, some environments (e.g., SwimmerVel, CarButton1, CarGoal2) exhibit very low or even negative rewards because they aim to learn very safe policies. We will definitely add more discussion in the final revision.

We are also happy to learn more about the reviewer's opinion on selecting the cost limits. What we typically do is to make sure the problem itself is feasible and the optimal solution is stochastic in synthetic CMDPs and follow what people use (like OSRL and other baselines) in complicated environments.

We hope that our response addresses the concerns of the reviewer and is open to further discussion. Thank you again for raising the score!

We greatly appreciate the reviewers’ positive evaluation of the novelty of this paper. The current formulation can be applied to the case with $0-1$ cost, indicating whether a constraint is violated or not at each step. Nevertheless, if we only get feedback over the entire trajectory whether it is safe or not rather per step feedback, it is an indeed interesting future research direction on how to address such a scenario. In fact, even in the online setting such a scenario is yet to be resolved. We add a discussion.

We strongly agree with the reviewer on the gap between theories and practical algorithms in RL and safe RL. Our approach is motivated to take a step towards bridging the gap as we consider offline setup and we only need single policy coverability for the theoretical results. Regarding the comments on external tools like LLMs, one possible direction could be to consider the cost preference (like RLHF) instead of cost function observations and determine whether it is possible to design an algorithm with provable guarantees of safety.

Thank you for your positive feedback on our paper! We appreciate your support and comments. Please find our point-by-point response to your questions below.

• Response to clarification of "Adversarial": The adversarial training in this paper is designed based on the concept of relative pessimism [11,4,53], which aims to optimize the worst-case relative performance over uncertainty. In particular, we adversarially train the critic to find weighted Bellman-consistent scenarios where the actor is inferior to the reference/behavior policy (Eq. (2) and Eq. (4)). Note that we consider an offline setup, hence, we cannot access any new data, rather whatever data is available, we have to find policy based on that. Without adversarial training, if some state-action pairs are not covered in the dataset, one can set (incorrectly) a very high $Q$ value without affecting the training loss. The adversarially trained critic addresses this issue by setting the lowest possible $Q$-value avoiding setting a high value for our-of-distribution values. We will clarify and discuss the differences between our approach and the paper mentioned by the reviewer in the final version.

• Response to finite selection of $W$: The selection of $W$ is not arbitrary: it needs to be chosen by maximizing the importance-weighted average Bellman error (Eq. (2) and Eq. (4)). This is crucial in the formulation because maximization over $w$ in the importance-weighted average Bellman regularizer ensures that the Bellman error is small when averaged over measure $\mu \cdot w$ for any $w \in W$. This can control the suboptimality of the learned policy and guarantee the optimal statistical rate of $1/\sqrt{N}.$

• Response to lack of discussion of assumption gap in the practical version: For the assumptions made in our paper, as far as we know, we require minimal assumptions in offline RL (as shown in Table (1)), especially in safe offline RL. The single-policy coverage required by our approach is far milder than the all-policy coverage assumption as it is impractical to assume that the dataset would contain state-action pairs from unsafe policies. Thus, we achieve our result for a more practical and realistic set of assumptions (and it matches the same set of assumptions for the unconstrained case). For the practical purpose, we think the most concerning part is that approximate realizability in Section 3.2 may not hold in practice. However, in our experience, as we usually use neural networks to approximate the function class $F$, it is likely that the approximate realizability error is small as long as we have a sufficiently rich neural network. The behavior policy is easy to achieve, even if it is not given to us, since extracting the behavior policy from an offline dataset is not difficult with behavior cloning (BC). In particular, we can estimate the learned behavior policy $\hat{\pi}\_\mu$ as follows: $\forall s \in D, \hat{\pi}\_\mu(a \vert s) \leftarrow \frac{n(s,a)}{n(s)}$, and $\forall s \notin D, \hat{\pi}\_\mu(a \vert s) \leftarrow \frac{1}{\vert A \vert}$, where $n(s,a)$ is the number of times $(s,a)$ appears in the offline dataset $D$. Essentially, the estimated BC policy matches the empirical behavior policy on states in the offline dataset and takes uniform random actions outside the support of the dataset. It is easy to show that the gap between the learned policy $\hat{\pi}\_\mu$ and the behavior policy $\pi\_\mu$ is upper bounded by $\min\lbrace 1, \vert S \vert / N \rbrace$ [R1, R2]. We can have a very accurate estimate as long as the size of the dataset is large enough. We will add more details in the revision.

[R1]: Kumar, Aviral, Joey Hong, Anikait Singh, and Sergey Levine. "Should i run offline reinforcement learning or behavioral cloning?." In International Conference on Learning Representations. 2021.

[R2]: Rajaraman, Nived, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. "Toward the fundamental limits of imitation learning." Advances in Neural Information Processing Systems 33 (2020): 2914-2924.

• Response to capability under sparse-cost setting: Maximizing the importance-weighted average Bellman regularizer aims to control the Bellman error. We believe this can also be applied to the long-tail constraint case. Our results can be generalized to a high probability bound (might be loose) using Markov's inequality, and it is possible to show some results under the CVaR objective. Of course, rigorous proofs and experiments are needed. We will explore these interesting directions in the future.

• Response to extension of the current framework to multi-constraint settings: Yes, the current approach is readily to generate to the case with multiple constraints by simply considering multiple constraints in the objective function: $f_r^k(s,a) - \sum\_{i=1}^I \lambda\_i \lbrace f_{i,c}^k(s,a) - f_{i,c}^k(s.\pi_{ref}) \rbrace\_+,$ where $f\_{i,c}^k$ is the $i$th constraint. We only have to tune $\lambda_i$, and $\beta_{c,i}$ values corresponding to constraint $i$. We will add some discussions in the revision.

• Response to the selection of weight W in the experiments: In experiments, we take $\mathcal{W} = \{0, C_\infty\}$ for computation effectiveness. Then we can reduce $\epsilon \_{D}(\pi, f)$ (Eq. (4)) to $C\_{\infty}E\_{D}[(f(s,a)-r-\gamma f(s',\pi))^2]$. In the environment, we simply set $C_\infty$ to be 1 in the neural case and 5 in the tabular setting.

We greatly thank the reviewer for considering reevaluating our paper and for raising these two interesting questions.

Extracting the Behavior Policy

In the standard offline RL setting, we assume that the dataset is generated by some behavior policy $\mu(s,a)$ and is i.i.d. Therefore, as long as the state-action space is finite, the method we provided guarantees that we can accurately estimate the behavior policy. This holds true whether we are considering safe RL or regular RL, as the only difference is the additional information regarding the cost function, while the process of generating the dataset remains the same.

For more complicated cases, such as when the state space is continuous, we can use a neural network to approximate the policy by minimizing the distance between the learned policy and the behavior policy. Alternatively, we could use DAgger (Dataset Aggregation) to achieve an even better policy. However, in such cases, the assumptions made in offline RL (not only in our paper but in the field generally) may no longer hold, particularly the data/policy coverage assumption. This is why we argue in our paper that single-policy coverage is crucial since it is much weaker than the full-policy coverage assumption.

extended to the non-tabular cases

There seems to be some confusion. We consider the function approximation setting, not a tabular setting, and the theoretical results are independent of the size of the state and action space under the given assumptions. The practical version of our algorithm is designed to develop a deep neural network approach for more complex environments. In order to handle the continuous state and action spaces, we use an actor-critic approach to solve the optimization problem (2), which aligns with the objective in our theoretical version. Specifically, we can use the aggression-limited objective to train the actor-network, which is feasible by considering two Q-value neural networks (one for reward and one for cost). The method used in our practical version for the weighted Bellman regularizer is a very simplified version. However, we believe it is possible to approximate $w(s,a)$ with another neural network such that the weighted Bellman error is minimized when the critic network is fixed. We will add more discussions and possibly some results in the revision.

Please let us know if you have further questions and comments, we are glad to have more discussions.

Thank you for your positive feedback on our paper! We appreciate your support and comments. Please find our point-by-point response to your questions below.

• Response to the reference policy: We would like to mention that extracting the behavior policy from an offline dataset is not difficult with behavior cloning (BC). In particular, we can estimate the learned behavior policy $\hat{\pi}\_\mu$ as follows: $\forall s \in D, \hat{\pi}\_\mu(a \vert s) \leftarrow \frac{n(s,a)}{n(s)}$, and $\forall s \notin D, \hat{\pi}\_\mu(a \vert s) \leftarrow \frac{1}{\vert A \vert}$, where $n(s,a)$ is the number of times $(s,a)$ appears in the offline dataset $D$. Essentially, the estimated BC policy matches the empirical behavior policy on states in the offline dataset and takes uniform random actions outside the support of the dataset. It is easy to show that the gap between the learned policy $\hat{\pi}\_\mu$ and the behavior policy $\pi_\mu$ is upper bounded by $\min \lbrace1, \vert S \vert / N \rbrace$ ([R1, R2]). We can have a very accurate estimate as long as the size of the dataset is large enough. In addition, in many applications such as networking, scheduling, and control problems, there are existing good enough reference policies. In these cases, a safe robust policy improvement over these reference policies has practical value.

[R1]: Kumar, Aviral, Joey Hong, Anikait Singh, and Sergey Levine. "Should i run offline reinforcement learning or behavioral cloning?." In International Conference on Learning Representations. 2021.

[R2]: Rajaraman, Nived, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. "Toward the fundamental limits of imitation learning." Advances in Neural Information Processing Systems 33 (2020): 2914-2924.

• Response to computationally efficient: We thank the reviewer for pointing out our statements on computational efficiency compared to existing approaches. We claim our approach is efficient and tractable compared to [19, 26] mainly because: [26] requires two FQI inner loops for policy improvement and three additional inner loops for policy evaluations, while [19] also requires an inner loop for offline policy evaluation. However, our algorithm does not have any inner loop for extra OPE. To demonstrate the efficiency, in the following Table (Table 1), we report the training times of our method and other baselines for the Car Goal environment using one NVIDIA GeForce RTX 3080 Ti. We observe that our practical version still has a faster training time, which is very time-efficient compared to others. We will make this clear in the revision.

Table 1: Training Time (seconds) for 200 steps

• Response to ablation studies: The weighted Bellman regularize, aggression-limited objective, and no-regret policy optimization together guarantee our theoretical results. We did an ablation study in the tabular setting and the results can be found in the following table (Table 2). The results of the ablation study indicate that the weighted Bellman regularization and no-regret policy optimization ensure the safety of the algorithm, while the aggression-limited objective ensures the algorithm to achieve higher rewards without compromising safety. | Components | Cost | Reward | |--------------------------------------------------------------|-------|--------| | ALL | 0.016 | 0.766 | | W/O no-regret policy optimization | 0.016 | 0.766 | | W/O Aggression-limited objective | 0.016 | 0.765 | | W/O Weighted Bellman regularize | 0.181 | 0.624 |

Table 2: Ablation study under tabular case (cost limit = 0.1)

• Response to hyperparameter sensitivity of $\beta$ : Note that our main result remains the same for SRPI as long as $\beta \geq 0$, indicating that our approach is highly robust across a wide range of $\beta$ values. We use different $\beta_r$ and $\beta_c$ in the practical version because it allows us to easily make a trade-off between rewards and costs, as different environments have varying sensitivities to rewards and costs. To address the reviewer's comments about hyperameter sensitivity, we provide the rewards and costs under different sets of $\beta_r=\beta_c\in \{1,0.5,0.05 \}$ and $\lambda\in\{[0,1],[0,2],[1,2]\}$ (since our $\lambda$ only increases, the closed interval here represents the initial value and the upper bound of $\lambda$) to demonstrate the robustness of our approach in the tabular setting in Figure 1 (Please see the uploaded file). We can observe that the performance is almost the same under different sets of parameters and different qualities of behavior policies. We will add more details in the revision.