C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning

Q1: The main novelty is redefining the reward function that only gets rewards when the policy is safe. The contribution part is not clear.

In SORL, the most important concern is the OOD problem, while existing methods can only mitigate it via policy constraining but cannot avoid it. The first novelty of this paper is proposing C2IQL to completely avoid the OOD problem in SORL.

The second novelty is that we may be the first one to propose and analyze the problem of discounted cost value formulation in detail. The contribution is illustrated in line 87, page 2 while redefining the reward function is the novelty of CPQ (2022).

For better clarity, we will add new content to line 88 page 2 and line 100 page 2:

"We first propose Constrained IQL (CIQL) to address the OOD problem, which can only be mitigated for existing RL methods in constrained settings."

"As far as we know, we may be the first one to propose and analyze the problem of discounted cost value formulation in detail."

Q2: References [A1] is not discussed/compared.

Thank you for bringing this paper to our attention. We agree that [A1] is relevant to SORL, and we will include it in the related work section and as a baseline in the experiments. Specifically, we will add the following to line 205, page 4:

"... problem. WSAC (Wei et al., 2024) focuses on the policy inferior problem and proposes a refined adversarial objective function. OASIS ..."

Additionally, we compared WSAC with C2IQL across all environments. The results are summarized below where bold results are unsafe:

C2IQL achieves better performance while satisfying constraints on all tasks.

Q3: The improvement compared to the CDT seems to be marginal.

The performance gain only appears marginal because the unsafe results in smaller constraints are averaged by safe results in larger constraints since Table 1 is averaged out 3 constraint thresholds following the style of most existing work. Thus we add Figure 2 as a supplement of Table 1 to illustrate that:

1. CDT cannot achieve safe policy for small constraints (like L<30) under some cases. This is very fatal disadvantage since satisfying the constraint is the foundation of safety.
2. CDT cannot achieve reward maximization for large constraints (like L<70) compared to C2IQL.
3. C2IQL achieves best and safe performance for all three constraints. This indicates C2IQL provides substantial benefits in handling a wider range of constraints and reward maximization accordingly.

Q4: The cost reconstruction part is not clear. How did the paper reconstruct the cost by finding the value function corresponding to different discount factors?

We would like to provide the following clarifications: The definition of the cost value function is a linear function where $\gamma$ is the constant and $c(.,.)$ are variables.

$V_c^\pi(s_t)=\sum_{j=0}^T\gamma^jc(s_{t+j},a_{t+j}\sim\pi)$

The goal is to reconstruct the non-discounted cost value:

$\hat C^{\pi}(s_t)=\sum_{j=0}^Tc(s_{t+j},a_{t+j}\sim\pi)$

To approximate the non-discounted cost value, we construct a set of cost value functions with different discount factors:

$V_c^{\pi,i}(s_t)=\sum_{j=0}^T\gamma^j_ic(s_{t+j},a_{t+j}\sim\pi),i=1,...,m$

Using these multiple discounted value functions, we train a supervised learning model to estimate the non-discounted cost value, where the inputs are estimated $V_c^{\pi,i}(s_t)$ and the output is $\hat C^{\pi}(s_t)$.

Q5: Should provide theoretical guarantee.

Please refer to Q2 of reviewer MiHo for theoretical analysis due to character limit. The mathematical proof are out of the scope of this paper. Besides, C2IQL has good empirical results including additional experiments on SafetyGymnasium from Q1 reviewer ojm5.

Q6: Question about equation (3) and (4).

Yes, equation (3) lets the infeasible reward Q-value to be 0 and equation (4) indicates only feasible policy is updated, which are the key ideas of CPQ (2022). However, this is not what C2IQL should be focused on. While your suggestion to multiply the indicator function with the reward is reasonable and could improve CPQ, it is outside the focus of our work, as we adopt CPQ’s framework without modification. Investigating this idea further would be interesting for future work.

Q7: About 4 different loss functions.

Compared with existing safe RL algorithms, we only add one loss to train the cost reconstruction model (CRM). In safe RL, value loss, cost value loss, and policy loss are essential. For the CRM, it is trained separately with the dataset and some randomly generated data. It is used in C2IQL after the loss is minimized and stable. Thus the training stage is not influenced.

Q1: An experiment of how errors/inaccuracies in cost estimation impact safety guarantees is needed.

Thank you for your suggestion. To address your concern, we have conducted additional experiments to analyze the impact of cost reconstruction error on policy performance and safety guarantees. To simulate increasing cost reconstruction error, we introduced noise sampled from a normal distribution, $N(0,Noise)$, to the input of the cost reconstruction model. When the added noise is small, the performance (reward) slightly improves as the noise level increases. This occurs because small amounts of noise act similarly to data augmentation, which enhances the generalization capability of the cost reconstruction model. This is analogous to adding noise to input data in image processing to improve robustness. As noise increases from 1.0 to 4.0, the performance becomes progressively more conservative. Despite this, the reward and cost metrics degrade gracefully, indicating the robustness of our method.

Q2: The paper suggests that C2IQL mitigates the OOD problem, there is no theoretical guarantee on how well it generalizes to unseen cost distributions beyond the dataset.

Thank you for your question.

1, we would like to clarify that C2IQL avoids the OOD problem rather than merely mitigating it. In SORL, the OOD problem is defined as "Since the policy may produce OOD actions, the Q-value may be wrongly estimated". Previous methods address it by constraining the target policy to be close to behavior policy, thus only mitigating the problem. In contrast, C2IQL, following IQL, completely avoids the OOD problem by training the value function using expectile regression, which inherently focuses on in-distribution data during training.

2, C2IQL does not require generalization to unseen cost distribution because: 1) C2IQL avoids the OOD problem entirely by ensuring that the training process operates strictly within the dataset distribution. When training, C2IQL avoids relying on any distributions outside the dataset. Even during testing phase, C2IQL naturally tends to select actions that align with the dataset distribution, as it is trained to do so. 2) As for the cost reconstruction model (CRM), they will not encounter unseen cost distribution because CRM is only used during training. In testing phase, CRM is no longer utilized—only the policy is used to generate actions based on the current state and constraint conditions.

Q3: The results in Table 1 are a bit difficult to interpret without learning curves.

Thank you for your suggestions. We will add some more learning curves to the appendix. Table 1 follows a common style in most SORL papers (e.g., CDT, FISOR, OASIS), where results are presented as final values after training. Additionally, we use color coding to improve interpretability. The primary reason for not including learning curves is due to the nature of offline RL. In SORL, the agent is trained on a pre-collected and fixed dataset without interacting with the environment during training. As a result, monitoring performance during intermediate training steps is not typically necessary, since the agent's performance is only evaluated after the training process is complete. That said, we have included some learning curves in Figure 3 to provide a visualization of policy evolution.

Q4: Evaluation of sample efficiency is needed.

Thank you for your suggestions. In Offline RL, the agent is trained on a fixed and pre-collected dataset without any interactions with environments. This indicates that all methods (C2IQL and baselines) require the same amount of data without any new data added into the dataset. The dataset in DSRL usually contains thousands of trajectories.

Q5: Algorithm 2 is a bit difficult to interpret, which could consider to improve clarity.

Thank you for your suggestion. Algorithm 2 is indeed a collection of equations previously introduced in Sections 3.1–3.4, but now extended to the constraint-conditioned version. We realize that we did not explicitly indicate their connections to the earlier sections, which may have caused confusion.

To improve clarity, we will provide additional explanations in Algorithm 2 to the corresponding sections and equations:

Line 6: Cost Reconstruction -> Obtain discounted cost values and reconstruct the non-discounted value with $R^c$ in Section 3.3

Line 9: Constraint Penalization -> Obtain the constraint-penalized Q-value function with Equation (11)

Line 11: Update value function -> Update value function with Equation (12)

Line 14: Update cost value function -> Update cost value function with Equation (17)

Q1: More complex benchmark is needed.

Thank you for your suggestion. To address the reviewer’s suggestion, we have also incorporated additional experiments on the SafetyGymnasium benchmark to diversify the test tasks. Specifically, we have selected the 4 Point and 3 Velocity tasks as additional benchmarks different from Bullet-Safety-Gym. Given limited time, we have included results of C2IQL and two strong baselines (FISOR and CDT). The bold results are unsafe cases.

The additional evaluation on SafetyGymnasium demonstrates that our method, C2IQL, achieves nearly a 10% improvement in performance for most tasks while satisfying constraints, further validating its superior performance.

Q2: No discussions for Theorems.

Thank you for your suggestion. We will an explanation of Theorem 1 in line 255 to illustrate how it can be utilized in Theorem 2: "Theorem 1 provides us with a relationship between the constraint-penalized reward function and corresponding formulation of the implicit policy. When the implicit policy formulation follows Equation (14), the update of the value function under implicit policy $\pi_{imp}(a|s)$ is equivalent to the update formulation of the value function in Equation (13) under behavior policy (the policy for collecting the dataset). Thus we can utilize the formulation of the implicitly policy obtained in Theorem 1 to derive how to update the cost value function based on its definition in Theorem 2."

Q3: How do the authors obtain the results of baselines?

For BCQ-Lag, BEAR-Lag, COptiDICE, CPQ, and CDT, we use the code provided by the OSRL library. For FISOR and VOCE, we use the official code released by the respective papers. To ensure a fair comparison, we run each method across 10 evaluation episodes, using 5 random seeds and 3 different constraint thresholds.

Q4: The proposed method has very limited improvement to be honest.

The performance gain only appears marginal to CDT because the unsafe results in smaller constraints is averaged by safe results in larger constraints since Table 1 is averaged out 3 constraint thresholds following the style of most existing work. Thus we add Figure 2 as supplement of Table 1 to illustrate that:

1. CDT cannot achieve safe policy for small constraints (like L<30) under some tasks. This is very fatal disadvantage since satisfying the constraint is the foundation of safety.
2. CDT cannot achieve reward maximization for large constraints (like L<70) compared to C2IQL.
3. C2IQL achieves best and safe performance for all three constraints. This indicates C2IQL provides substantial benefits in handling a wider range of constraints and reward maximization accordingly.

Q5: The proposed method is not beating baselines in other OSRL work [1].

Thank you for bringing up this concurrent work. We would like to address the reviewer’s concern in the following aspects:

Baseline Algorithm Selection Criteria: We carefully select strong-performing methods (e.g, CDT (2023), VOCE (2023), FISOR (2024), WSAC (2024) from reviewer qPih) from recent publications in SORL field to ensure a comprehensive comparison. Extensive experiments on Bullet-Safety-Gym and SafetyGymnasium benchmarks have demonstrated the superior performance of our proposed C2IQL. Regarding the concurrent work [1], we excluded it from our comparison because it was only posted on arXiv in 12/2024 and has not undergone the peer review process.

Compare with [1]:

1. Performance: By directly comparing the results from Table 5 in [1] and Table 1 in our paper, we observe that C2IQL provides stronger performance compared to [1]:

• C2IQL achieves better results on AR, AC, BR, BC, CC, DR, and DC, demonstrating its strong performance across a variety of tasks.
• C2IQL exhibits slightly worse performance on CarRun compared to [1].

2. Method: [1] addresses issues with poorly behaved policies under global cost constraints. In contrast, C2IQL focuses on the OOD problem and is the first method designed to avoid the OOD problem in SORL. Besides, we also address the discounted cost formulation problem, which further improves constraint satisfaction and policy effectiveness.

[1] Offline Safe Reinforcement Learning Using Trajectory Classification.

Q1: My only concern is that the threshold penalty is chosen randomly. Adding experiments on real-world problem settings is appreciated.

Thank you for your suggestion. The threshold penalty is choose based on small (<50%), middle (50%), and large (>50%) constraint of the maximum cost, which is comprehensive to cover real-world constraint settings. Besides, we sincerely agree that addressing real-world problems is important. However, in most existing SORL benchmarks [1, 2], there are nearly no real-world simulators since SORL is a newly rising field in recent years. Thus we will consider it in our future work. To address your concern, we have expand experiments on other environments in [2]. Please refer to Q1 of reviewer ojm5 for detailed results due to character limit.

[1] Gronauer, S. (2022). Bullet-safety-gym: A framework for constrained reinforcement learning.

[2] Ji, J., Zhang, B., Zhou, J., Pan, X., Huang, W., Sun, R., ... & Yang, Y. (2023). Safety gymnasium: A unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36, 18964-18993.

Q2: Theorem 1 and 2 miss quality guarantees and convergence proofs.

We appreciate the reviewer recognizing the insights provided by theorem 1 and 2, which highlight our contribution avoid the OOD problem and address the discounted formulation problem in constrained settings. Regarding the quality guarantees and convergence proofs, we think our method share the same results as CPQ theoretically. The sketch of the proof is given as follows: First, we have that as the expectile parameter $\kappa$ increase to 1, $V^\pi_r(s)$ can approach $\max_aQ^\pi_{r|c}(s,a)$ because the expectile regression is a convex function. Second, convergence proofs of our algorithm can follow similar steps as those in CPQ because CPQ utilize $\max_aQ^\pi_{r|c}(s,a)$ while C2IQL utilize $V^\pi_r(s)$.

Q3: Adding a few real-world examples to motivate constraint offline RL in introduction.

Thank you for your suggestion. We add a few real-world examples on the right part of line 24, page 1, second paragraph, to further strengthen our motivation: "… in safety-critical scenarios. For example, unsafe operations could harm patients in healthcare, unsafe driving style may lead to accidents, and unsafe decisions may incur additional costs in financial investments. In these situations, …"

Q4: Adding a brief high-level summary will be helpful to follow the theoretical results and algorithms.

Thank you for your suggestion. We will add a high-level summary at the beginning of section 3.1: "To derive a concrete CIQL algorithm, we need to answer three questions: First, how to update the constrained reward value/Q-value function following IQL style? To address this problem, CIQL formulates a constraint-penalized reward Q-value function following CPQ and utilize a value function with expectile regression to approximate the maximized Q-value function in Bellman backup procedure. Second, how to update the cost value function under the same implicit policy since it is hidden in the reward value function? To address this problem, we rederive CIQL and obtained the formulation of the implicit policy in theorem 1 following IDQL, and then derive the formulation of the cost value function with this implicit policy. Third, How to extract the policy? We extract the policy in an expectile way following Equation 18."

Q5: Runtime analysis is needed.

We record the training time of our proposed C2IQL and baselines in AntCircle scenario in the following table:

Overall, the training cost of C2IQL is reasonable when compared to other methods. While it takes slightly more time than methods like FISOR and VOCE, it is still significantly faster than CDT, which has the highest training time. Notably, C2IQL achieves a remarkable balance between computational efficiency and performance. The additional training time is justified by the significant performance gains provided by C2IQL, making it a practical and effective solution

Q6: About baseline configuration?

All baseline algorithms follow the default parameters, which are already finetuned in OSRL [1] projects for best performance. As for FISOR and VOCE, we finetuned the parameter to make sure the performance keeps consistent with the original paper. However, under the same environment but different constraints, we keep the same parameters for all methods including our proposed C2IQL.

[1] Liu, Z., Guo, Z., Lin, H., Yao, Y., Zhu, J., Cen, Z., ... & Zhao, D. (2023). Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303.