Diffusion-Guided Safe Policy Optimization From Cost-Label-Free Offline Dataset

However, several major weaknesses exists:

Motivations and paper problem setups

• A dense Markovian cost function maybe not nessary for safe offline RL. I agree with the author that designing a dense Markovian cost function that can capture the trajectory-level safety performance and can consider furture unsafe outcomes is pretty challenging. However, this does not imply that obtaining a sparse cost function is diffcult. Typically, we can straightforwardly annotate states as 0 or 1 to indicate whether they are in unsafe or safe regions, respectively.
  • While this simple sparse cost function may not offer trajectory-level safety evaluations, its associated Q-value or V-value function can propagate future information through bootstrapping. Thus, we can easily design a sparse cost function and derive its Q/V-value function to provide a trajectory-level safety signal.
  • This paper presupposes a safe dataset. From my perspective, there must be a straightforward safety judgment function used to determine the safety of this dataset. Consequently, this simple function could readily serve as the sparse cost function I previously mentioned.
  • The authors may believe that learning by sparse cost can be inefficient, since the TD signal is sparse. But in the offline setting, we can easily modify the sparse costs to sparse feasible value (1 for unsafe, -1 for safe)[1]. By doing so, learning an accurate Q/V value function becomes more easier.

Limited Experiments

The authors can criticize that learning a Q value function through bootstrapping is unstable, and meanwhile argure that the proposed safety transformer offers more stable training. However, this kind of experiments are missed in the current manucript.

• More popular safe offline RL methods should be evaluated. The evaluated baselines in this paper include nearly no methods except CDT that specifically desinged for safe offline RL, probably because no costs are available. However, the authors should compare against them using sparse costs.
  • For example, the authors should compare other popular baselines, such as CPQ[2] and COptiDICE[3] based on the sparse cost function just as I mentioned, and the recent SOTA safe diffusion model FISOR[1] using sparse feasible function, etc.
  • The authors can also provide some intuitive illustration to demonstrate that the Q value learned by bootstrapping cannot offer accurate recall boundaries in Figure 3. This can further enhance the paper quality.

Theory: strong and strange assumption

In theorem 1, the authors claim that they are minimizing an upper bound of the original intractable mutual information. However, this theorem holds on a pretty strong and strange condition.

• Specifically, the assumption is $D_{KL}(p(z,r)||q(z,r;\theta))\le D_{KL}(p(z)p(r)||q(z,r;\theta))$, where z is cost and r is return. The authors just directly claim that they are minizing an upper bound if this assumption holds. However, the authors never discuss when and why this assumption holds.
• I checked the proof of Theorem 1 and found that the authors just expand this assumption and then can derive the upper bound objective. So, this objective may not be the upper bound of the original mutual information anymore if this assumption does not hold. In this case, the authors may try to minimize a lower bound, which is incorrect.

[1] safe offline reinforcement learning with feasibility guided diffusion, ICLR 2024.

[2] Constraints Penalized Q-learning for Safe Offline Reinforcement Learning, AAAI 2022

[3] COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation, ICLR 2022.

[4] Safe Offline Reinforcement Learning with Real-Time Budget Constraints, ICML 2023

1. Limited Contribution: While the authors claim three major contributions—a more practical problem setup, a novel approach combining a safety-transformer with a diffusion model data generator, and the construction of an offline dataset suite—these contributions seem limited in significance.
   1.1. Problem Setup: Similar problem formulations, which assume the absence of ground-truth cost functions or rely on learning preferences from a few safe demonstrations, have been explored in prior works on inverse safe RL [1, 2] and meta-safe RL [3].
   1.2. Methodological Novelty: The safety-transformer has been widely explored in prior research [4], and classifier-free diffusion models in (safe) RL / RL dataset curation have also been extensively studied [5, 6, 7]. The authors do not sufficiently clarify how their approach differs from previous work, nor do they provide a comprehensive comparison with related diffusion model-based safe RL approaches, such as FISOR [9] and Safe Decision Diffuser [10].
   1.3. Dataset Suite: The proposed dataset suite is a modified version of the OSRL benchmark [8], created by selecting safe demonstrations and masking cost information.

2. Vague Experimental Validation: The main experimental results, presented in Table 1, focus only on reward returns. For safe RL tasks, it is crucial to also report detailed cost/safety metrics to ensure thorough validation.

3. Additional Comments: Please refer to my specific questions for further points.

Reference:

[1] Schlaginhaufen, Andreas, and Maryam Kamgarpour. "Identifiability and generalizability in constrained inverse reinforcement learning." In International Conference on Machine Learning, pp. 30224-30251. PMLR, 2023.

[2] Kim, Konwoo, Gokul Swamy, Zuxin Liu, Ding Zhao, Sanjiban Choudhury, and Steven Z. Wu. "Learning shared safety constraints from multi-task demonstrations." Advances in Neural Information Processing Systems 36 (2024).

[3] Lin, Qian, Zongkai Liu, Danying Mo, and Chao Yu. "An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learning." arXiv preprint arXiv:2409.09958 (2024).

[4] Kim, Changyeon, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. "Preference transformer: Modeling human preferences using transformers for rl." arXiv preprint arXiv:2303.00957 (2023).

[5] Lee, Jaewoo, Sujin Yun, Taeyoung Yun, and Jinkyoo Park. "GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning." arXiv preprint arXiv:2405.16907 (2024).

[6] Liang, Zhixuan, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. "Adaptdiffuser: Diffusion models as adaptive self-evolving planners." arXiv preprint arXiv:2302.01877 (2023).

[7] Yao, Yihang, Zhepeng Cen, Wenhao Ding, Haohong Lin, Shiqi Liu, Tingnan Zhang, Wenhao Yu, and Ding Zhao. "OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning." arXiv preprint arXiv:2407.14653 (2024).

[8] Liu, Zuxin, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu et al. "Datasets and benchmarks for offline safe reinforcement learning." arXiv preprint arXiv:2306.09303 (2023).

[9] Xiao, Wei, Tsun-Hsuan Wang, Chuang Gan, and Daniela Rus. "Safediffuser: Safe planning with diffusion probabilistic models." arXiv preprint arXiv:2306.00148 (2023).

[10] Zheng, Yinan, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. "Safe offline reinforcement learning with feasibility-guided diffusion model." arXiv preprint arXiv:2401.10700 (2024).

• The motivation of the new setup in this paper (i.e., the cost label is unavailable in most offline demonstration while a few safe demonstration are provided) is relatively weak: different from reward signal, the cost label in previous works is binary and is $+1$ only when there is an unsafe event (e.g., crash), which can be easily obtained or measured even in practical application. Moreover, even in the new setting, the criterion of safety is also determined by step-wise cost label as mentioned in line 949.
• The dataset used for experiment is built heavily based on previous work [1]. However, the authors have never cited it when introducing the dataset in main text (e.g., line 79, line 314), which is extremely improper and misleads the readers on its novelty. I urge the authors to clarify it.

[1] Datasets and benchmarks for offline safe reinforcement learning

Despite the extensive empirical results, there are quite a few weaknesses that significantly hinder the presentation of this paper:

• Unrealistic assumption: the authors claim they would like to push safe RL to more 'real-world applications', where most of the safety-critical scenarios are long-tailed (sparse cost). However, in their assumption in page 3, line 138-143, they assumed that a small number of safe demonstration and another supplementary offline dataset will be provided in offline safe RL problems. This seem to be controversial with the real-world setup that I just mentioned as it will be infeasible for any behavior policies to sample a large set of 'supplementary offline dataset' if they are essentially 'unsafe'.
• Minor contribution in SafetyTransformer: the authors mentioned about they would like to resolve the case where most of the trajectory label missing. However, in the line of works using inverse RL for constrained RL [1, 2], such question has been explored even under harder setting, yet these works were missing in the literature review.
• Missing details in diffusion model: the authors titled section 3.2 with 'DIFFUSION-GUIDED SAFE POLICY', however, they just used diffusion model to synthesize high-return trajectories and use bahavior cloning to learn the policy. Besides, the implementation of diffusion policies, e.g. the classifier-free guidance, is missing in the paper.
• Uninformative theoretical results: the theoretical results in Theorem 1 is essentially based on the non-negativity of the mutual information. This is extremely uninformative to the reader what DSPO benefits the existing offline safe RL methods, e.g., what is the benefits in the cost function.

[1] Malik, Shehryar, et al. "Inverse constrained reinforcement learning." ICML 2021.

[2] Subramanian, Sriram Ganapathi, et al. "Confidence aware inverse constrained reinforcement learning." ICML 2024.