SafeDiffuser: Safe Planning with Diffusion Probabilistic Models

Question 1: Regarding to the design choice, the TVS diffuser seems to be the most general safediffuser. Is it possible to further simplify the forward invariance equation (theorem 3.4) by fully discarding the reliance on b, and directly using sigma? For example, imagine sigma at t=0 converges to exactly b. This sigma can be used in analogy to the time-varying CBF and should impose a more relaxed restriction to the intermediate samples. I guess it would need clever design of sigma (or even learning-based), but do you think it is worth the effort?

Response: Please note that b is the original safety specification, if we can define a sigma that converges to exactly b at t = 0, then we may discard the reliance on b. However, since the TVS-diffuser also has some time-varying function, it can also relax the restriction to the intermediate samples. Learning such a sigma would be interesting, and it is similar to learning a CBF [1]. It is definitely worth doing it as it can relax the assumption that b is known (see response to Weakness 4).

[1] Robey, Alexander, et al. "Learning control barrier functions from expert demonstrations." 2020 59th IEEE Conference on Decision and Control (CDC). IEEE, 2020.

Question 2: Another question is the motivation of using CBF. From first principle, all we care about here is the safety at the final timestep, ie, t=0. It sounds more similar to a goal-reaching/liveness problem, where you don’t need to enforce every intermediate sample satisfy the safety. Compared to control Lyapunov function or reachability-based method, why is CBF used for the problem? Applying the other two concepts seem more natural to me.

Response: A control Lyapunov function (CLF) is similar to a CBF (it is arguably true in the literature that a CLF is a special case of a CBF, i.e., shrinking an unsafe set in a CBF to a point would generate a CLF). However, CLF is mostly used for system stability, while CBFs are used for safety (the motivation of using CBFs instead of CLFs). The reachability-based methods are known to be computationally expensive, and thus, it is challenging to use them in diffusion models, while the CBF methods are highly efficient (the main motivation of using CBFs). Our approaches do not need to be enforced at every intermediate step, see response to Weakness 1.

Question 3: In control, CBF can usually be compared to potential-field based method to show their effective in terms of avoiding obstacle in a longer time horizon rather than reactively avoiding. Similarly, is it possible to include [1] as a baseline to show the approach’s effectiveness?

Response: This is definitely a good baseline. We have added new experiments to compare the potential-field based method and our methods (see the table below). In summary, potential-field based methods have no strict safety guarantees (see Fig. 5 a in [1] showing there are still collisions with potential-based methods), while our methods do. The main disadvantage of CBFs (compared to potential-field methods) lies in its myopic property, as mentioned by the reviewer. However, this disadvantage may be addressed when CBFs are combined with diffusion models as those models are powerful to generate long time horizon (and potentially optimal) trajectories, as shown by the scores or reward metrics in Tables 2-4 of the paper.

Maze planning:

Weakness 4: Scaling up seems challenging. I’m hoping to see a highly nonlinear dynamics like drone/quadcopter to see whether the method could scale up. Another challenge for scaling up is from the selection criteria. Page 6 discusses the principle to select different safediffusers. As mentioned, it requires the unsafe set to be known to determine the convexity. This seems challenging for high-dimensional control tasks.

Response: Scaling up is indeed a challenge in our previous results as we need to solve QPs in the previous methods. For highly nonlinear dynamics, the computation issue is the main concern. However, We have fully addressed the computational issue by finding their closed-form solutions (referring to CH3 of the book ``Optimization by Vector Space Methods’’). The closed-form solution is given in Appendix section B of the revised manuscript. We have rerun all the experiments using the closed-form solution of safediffusers, which shows that the computation time is almost the same as the original diffuser (given in the following three tables). The dynamics of manipulators and walkers are comparable to drones, and the dimension of the state space in the walker2d (usually 17) is in fact higher than that of a drone (usually 12). This scaling capability totally depends on the diffusion models, and this has been extensively tested in the literature.

The unsafe set needs to be known. However, we may learn the safety constraints using the methods such as in [1], and combine it with the diffuser. We can then further check the convexity of the unsafe sets from learned safety constraints. This is an ongoing research direction. Please note that the safety is usually not defined on the control, but on the state. The CBFs can map state constraints to control constraints. For high-dimensional control tasks, this won’t be an issue using the new proposed closed-form solution in terms of computational complexity.

[1] Robey, Alexander, et al. "Learning control barrier functions from expert demonstrations." IEEE-CDC 2020.

Maze planning (CF is short for closed-form solution):

Locomotion (CF is short for closed-form solution):

Manipulation (CF is short for closed-form solution):

We thank the reviewer for all the constructive comments. We address all the reviewer’s concerns in the response below.

Weakness 1: The definition of diffusion invariance seems too strict. Essentially we only care about the safety at t=0, and trajectories satisfying t=0 safety don’t necessarily need to satisfy t=1 safety. However the method actively encourages the intermediate trajectories to be safe. This doesn’t mean the method is wrong, but I’m hoping to see a softer way to guarantee the safety. See Q2.

Response: If we apply the proposed methods to all diffusion steps (as in the RoS-diffuser), this is surely too strict. However, we have extensively studied how to address such an issue by proposing the ReS-diffuser and TVS diffuser. For instance, in TVS-diffuser, the size of unsafe sets is almost 0 at the initial diffusion time step T, which means that all the trajectories are within safe sets (the initial condition of diffusion invariance is satisfied). As we have defined a time-varying component in TVS-diffuser (shown in Thm. 3.4), the size of unsafe sets will grow with the diffusion time. At the final diffusion time step 0, the size of unsafe sets is the same as the original unsafe sets, then the safety is guaranteed. Therefore, the definition of diffusion invariance in TVS-diffuser is not strict due to the existence of the time component. Alternatively, we may address this issue by applying safediffusers to limited diffusion steps, and we can still guarantee the safety while not being overly conservative, as shown in the ReS-diffuser-l10 of Table 2.

Weakness 2: Along with the above weakness, the design choice seems too complex. All of them seem hyper-parameters that need careful tuning.

Response: The robust term (and all the parameters) is actually used to define a more strict safety specification compared to the original safety constraint. This is to overcome the asymptotic convergence properties of the CBF method. In practice, we may actually set the robust term to 0, and the resulting safety metrics (S-spec and C-spec in tables 2-4) are about $-1e^{-7}$ (they should be $\geq 0$), which shows a slightly violation of safety. Therefore, if we set the robust term to a small positive number (e.g., 0.01), we no longer need to tune these parameters. The design choice of parameters is to show formal guarantees from a theoretical perspective (providing a tight lower bound for the robust term, i.e., introducing less restrictive safety guarantees).

Weakness 3: CBF itself is meant to design for dynamics that are (1) time-invariant; and (2) deterministic. The diffusion process doesn’t belong to any of these two parts. For (1), the alpha bar changes over time steps. For (2), most of the diffusion sampler constantly adds noise to the dynamics (unless the approach is specifically using DDIM). Applying CBF-based technique to the diffusion seems counter-intuitive.

Response: We wish to clarify the misunderstanding of CBFs and our approaches here. CBFs have already been designed for dynamics/constraints that are time-varying (e.g, [1]) and stochastic (e.g., [2]). Therefore, CBFs can be applied to the diffusion process. In fact, our proposed Time-Varying Safe (TVS)-diffuser belongs to the time-varying category. In our approaches, since we wish to strictly ensure the safety of diffusion models, we take the diffusion rollout result (during inference, a denoising process without adding noise) at each diffusion step as a reference in our QP solution, and optimally revise the diffusion process. In such a way, we can ensure the safety of diffusion models. This has been further verified on all the experiments (with code attached in the submission). Our approaches are mainly used during the inference of diffusion, and thus are not subject to the problem in (2).

[1] Lindemann, Lars, et,al. "Control barrier functions for signal temporal logic tasks." IEEE control systems letters 3.1 (2018): 96-101.

[2] Clark, Andrew. "Control barrier functions for stochastic systems." Automatica 130 (2021): 109688.

Dear reviewer gj8X,

As the deadline of the rebuttal is approaching, could you please let us know if you have any further concerns?

Looking forward to your reply,

Best regards,

Authors.

Dear Reviewer gj8X,

Thank you once again for your time and effort in reviewing our paper. We have worked very hard to address all your concerns, including new experiments showing the advantages of our methods. Could you please kindly let us know if you still have any other questions or concerns?

Much appreciated,

Best regards,

Authors.

We appreciate the reviewer for all the positive and constructive comments. All the concerns are addressed carefully in the responses below.

Weakness 1: Safety specifications have to be differentiable.

Response: We agree that this is one of the concerns. However, we may always use differentiable constraints to slightly over-approximate non-differentiable safety specifications (e.g., using a super-ellipse to cover an unsafe set with square shape).

Weakness 2: SafeDiffuser is significantly slower (by an order of magnitude) than vanilla diffusion.

Response: We have fully addressed the computational issue by finding their closed-form solutions (referring to CH3 of the book ``Optimization by Vector Space Methods’’). The closed-form solution is given in Appendix section B of the revised manuscript. We have rerun all the experiments using the closed-form solution of safediffusers, which shows that the computation time is almost the same as the original diffuser (given in the following three tables).

Maze planning (CF is short for closed-form solution):

Locomotion (CF is short for closed-form solution):

Manipulation (CF is short for closed-form solution):

We want to further clarify that this is not a limitation for our current methods in diffusion models. First, the diffusion models we used are in discrete time (not in continuous time) [1], and the output of the model cannot be unbounded at each diffusion step. Thus, the (discrete-time) diffusion models are Lipschitz.

Second, even for continuous time diffusion models (such as those based on stochastic ODEs [2]), if the activation functions are Lipschitz, the diffusion models are also Lipschitz by construction.

[1] Michael Janner, et.al., Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pp. 9902–9915. PMLR, 2022

[2] Yang Song, et.al., Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

We thank the reviewer for the positive and insightful comments. We address the remaining concerns below.

Weakness 1: The proposed methods require solving QPs, which is computationally inefficient.

Response: We have fully addressed the computational issue by finding their closed-form solutions (referring to CH3 of the book ``Optimization by Vector Space Methods’’). The closed-form solution is given in Appendix section B of the revised manuscript. We have rerun all the experiments using the closed-form solution of safediffusers, which shows that the computation time is almost the same as the original diffuser (given in the following tables).

Maze planning (CF is short for closed-form solution):

Locomotion (CF is short for closed-form solution):

Manipulation (CF is short for closed-form solution):

Weakness 2: [Minor Weakness] The approach requires predefined safety constraints. However, I think that this limitation is relatively minor, such a setup is not a major weakness.

Response: We agree on this weakness, and have also discussed this as a limitation. However, we may learn the safety constraints using the methods such as in [1], and combine it with the diffuser. This is an ongoing research direction.

[1] Robey, Alexander, et al. "Learning control barrier functions from expert demonstrations." IEEE-CDC 2020.

Weakness 3: It would strengthen the paper if the authors included comparisons with CDT [1], which also employ sequential modeling techniques to address offline safe RL problems.

Response: We thank the reviewer for suggesting this. In fact, we have compared with similar methods in our tables (CG and InvODE). The method in CDT [1] can improve the safety, however, it cannot guarantee the safety as safety constraints are additional inputs to the model. The CDT [1] method is similar to that of [2] [3] (discussed in our related works), and we added it and discussed it in our revision.

[1] Liu, Zuxin, et al. "Constrained decision transformer for offline safe reinforcement learning." ICML 2023.

[2] Ye Yuan, et.al. Physdiff: Physics-guided human motion diffusion model

[3] Anurag Ajay, et.al. Is conditional generative modeling all you need for decision making? ICLR 2023.

Dear reviewer hg4A,

We have added new experiments of the closed-form solution to address your main concerns. Could you please let us know if you have any other questions or concerns?

Looking forward to your reply,

Best regards,

Authors.

We appreciate the reviewer for all the constructive comments. We address the remaining concerns in the response below.

Weakness 1: Some parts are a bit difficult to read and have unusual terminology (e.g., "local traps"?) and the paper contains some minor grammar and cosmetic issues which make the quite complex approach harder to understand.

Response: The term ``local traps’’ denote trajectory points getting stuck at obstacle boundaries and is due to the existence of non-convex unsafe sets, which are explicitly explained at the end of Sec. 1 and 2 where they firstly show up. We added a formal definition (Def. 3.2) in the revision to make it more clear. We revised the paper to address all the grammar and cosmetic issues.

Weakness 2: The method is actually three methods, and the user is to pick the one most suitable for their problem. This seems a bit complicated, why not go for the most general?

Response: The three methods are at different complexity and for different purposes. The RoS diffuser is simple to use but may be subject to local trap issues if there are non-convex unsafe sets, while the other two methods are more complex (e.g., designing either relaxation variables or time-varying functions) but can address the local trap issues. As far as we are concerned, there is no such a general method that can simultaneously be simple and address all the concerns. Nevertheless, these three methods are not that different from the implementation perspective, and they are backup'ed by different theoretical justification.

Weakness 3: presentation issues: excessive underlining and typos

Response: We removed unnecessary underlining and revised the paper to address typos in the revision.

Question 1: Are "local traps" = local minima?

Response: Conceptually, yes, it is similar to local minima. However, since this is for planning purposes, local traps denote trajectory points getting stuck at obstacle boundaries and are due to the existence of non-convex unsafe sets (may not be necessarily a minima, but more like a trap). Please see Fig. 6 of appendix for an example of local traps in maze planning.

Question 2: Isn't it a big problem that the method gets stuck in "local traps" between 39% and 100% of the time for some of the benchmarks? What does this mean in practice?.

Response: This is not a big problem if only few trajectory points get struck in local traps (as in our proposed ReS and TVS diffusers) as we can easily address this using smoothing techniques (e.g., by removing these trajectory points or taking the average of trajectory points within a sliding window). In practice, this means that some trajectory points get stuck at unsafe set boundaries (usually flat or nonconvex). Please see Fig. 6 of appendix for an example of local traps in maze planning.

Question 3: What is trap rate 1 vs. trap rate 2 in the table?

Response: The trap rate $r$ denotes the trap rate with the number of trapped trajectory points $\geq r$, where $r=1$ or $2$ in the table. This was defined in the caption (the last sentence) of Table 2 (highlighted in the revision).

Question 4: Could you comment on why you chose to use CBF here instead of e.g. lagrangian objectives common in safe RL?

Response: The Lagrangian objectives in safe RL can also guarantee safety. However, since it is very complex to solve, to our knowledge, it is mostly used as a loss/reward function during the training of safe RL. There are still no safety guarantees during inference. We choose CBFs since the CBF method can strictly ensure the safety (as the CBF constraints are hard constraints) while being very efficient to solve during both training and inference, as shown in Thm. 3.2-3.4. We can also find the closed-form solution of the CBF method (please see Appendix section B of the revised manuscript for details). We have done new experiments to show the effectiveness of the closed-form solution, please see the response to weakness 1 of reviewer hg4A for the new experiment results.

Question 5: Why three methods instead of the most general, or one that picks the best solution approach?

Response: As far as we are concerned, there is no such a general method that can simultaneously be simple and address all the concerns. The RoS diffuser is simple to use but may be subject to local trap issues if there are non-convex unsafe sets, while the other two methods are more complex (e.g., designing either relaxation variables or time-varying functions) but can address the local trap issues. Nevertheless, these three methods are not that different from the implementation perspective, and they are backup'ed by different theoretical justification.

Dear reviewer 4UNS,

Hope our response have addressed your concerns. Could you please let us know if you have any further concerns?

Looking forward to your reply,

Best regards,

Authors.

Dear Reviewer 4UNS,

We have got responses from all the other reviewers, and they are satisfied with our current revision of the paper indicated by raising their scores. As you have explicitly mentioned that ``I will observe the other reviewer reactions and consider increasing my score based on their responses'', could you please let us know your response as all the other reviewers have responded?

Looking forward to your reply,

Thank you and best regards,

Authors.

Dear Reviewer 4UNS:

We are truly grateful for your insightful comments and advice, which have played a significant role in enhancing the quality and clarity of our paper.

We hope that the additional details and experimental results we provided have effectively addressed your concerns. As the rebuttal period comes to an end, we kindly request your thoughts on our rebuttal and ask that you consider raising your score accordingly. If there are any remaining concerns, please feel free to share them with us.

Once again, we deeply appreciate your thoughtful review and constructive feedback.

Best,

Authors

In light of the updates and clarifications I am raising my score to 6. I think the approach shows promise but I am not entirely convinced of how practical it is to apply. The experiments are on rather simple and low-dimensional problems: 2D mazes, 2D walker and what seems like a rather simple 7DoF manipulation problem. It is not obvious to me on which problem I would prefer this over some combination of classical (e.g., MPC/MPPI, RRT*) , Safe RL or safety filter approaches. It does impose safety during run-time but you may also need special handling of parts of the trajectory getting stuck local traps, rendering the solution infeasible.