Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control

Q4 (1/4): Theorem 3 is difficult to understand. What is the purpose of giving Theorem 3?

R: Thanks for the feedback! Theorem 3 provides a way of computing an approximate projected objective function gradient ($g$) so that it does not interfere with the gradients from the constraint ($\sigma^{(m)}$).

This allows us to borrow ideas from multi-objective optimization to improve the sample efficiency as compared to the CRPO-style of constrained optimization.

To improve the clarity of Theorem 3, we have moved the formal statement of Theorem 3 into the Appendix (Theorem A2 in Appendix). Instead, we put an informal statement of Theorem 3 (Informal Theorem 3 in Section 4.3) in the revised version that more clearly communicates the above idea.

Also, we have provided additional discussion on Theorem 3 in the revised manuscript (paragraph above (14)).

---

Q4 (2/4): The orthogonality assumption in Theorem 3 is impractical.

R: While we do agree that the orthogonality assumption is impractical, it does allow us to derive a policy loss function that outperforms other alternatives (i.e., ablation study in Appendix B.6.1 of the original submission or Appendix C.6.1 of the revised version).

Due to the unrealistic assumptions, we view Theorem 3 not as a core theoretical contribution but rather as a principled way to motivate design choices that behave well empirically, as showcased by our extensive empirical results.

---

Q4 (3/4): In Theorem 3, the expression for $\sigma$ uses a state distribution $\rho$ independent of $\theta$, but the expression for $g$ uses the stationary state distribution (which is a function of $\theta$).

R: Good observation! The situation here is the same as in our response to Q3. Specifically, there is a difference between $\nabla_\theta \mathbb{E}\_{\mathbf{x} \sim \rho^{\pi_\theta}}$ which requires the policy-gradient theorem to compute the gradient, and$\nabla_\theta \mathbb{E}\_{\mathbf{x} \sim \rho} \Bigg|\_{\rho = \rho^{\pi_\theta}}$, which uses the score function gradient.

We have included a detailed discussion on this in Appendix B.2 in the revised version.

---

Q4 (4/4): What is the relationship between (13) and (14)?

R: We realize that the derivation of the policy loss $L$ from Theorem 3 could be made clearer, and have added a detailed derivation of the policy loss to Appendix B in the revised version.

Also, to make the connection more straightforward, we change the old (14) to use an indicator function instead of having two cases for the safe and unsafe cases, i.e.,

$ L(\theta) &= \mathbb{E}\_{\mathbf{x} \sim \psi(\rho^{\pi_\theta})} \mathbb{E}\_{\mathbf{u} \sim \psi(\pi\_\theta(\cdot|\mathbf{x}))} \big[ \log \pi\_\theta(\mathbf{x}, \mathbf{u}) \psi\big( \tilde{Q}(\mathbf{x}, \mathbf{u}, \theta) \big) \big], \\\\ %%%%%%% \tilde{Q}(\mathbf{x}, \mathbf{u}, \theta) &:= \mathbb{1}\_{ \\{\max_m \tilde{C}^{(m)}\_\theta(\mathbf{x}) \leq 0 \\} } \psi(Q^{\pi_\theta}(\mathbf{x}, \mathbf{u})) + \nu \max\_m \tilde{C}^{(m)}(\mathbf{x}, \mathbf{u}). $

We thank the reviewer for finding our considered problem general and challenging and for acknowledging our extensive experiments. We also thank the reviewer for the detailed theoretical concerns from closely reading our manuscript.

We hope that our responses below and our revised manuscript address the concerns raised by the reviewer.

Summary

In brief, we have:

1. Explained why the policy in this work is "distributed"
2. Clarified the difference between policy gradients and score function gradients during the policy updates w.r.t. the DCBF condition and performed additional experiments that demonstrate the difference
3. Provided proofs for the safety and generalizability guarantee of DGCBF.
4. Clarified the necessity of using deterministic rollouts to learn the constraint-value function $V^h$
5. Discussed the advantage of using DGCBF instead of DCBF

Detailed Reply

Q1: The policy in this work only takes local observations as input, such that it is decentralized. Why do you refer to it as a distributed policy?

R: Good question.

We use the definition in this work that a distributed method allows for communication among agents (e.g., [1]), though we note that this term has a different meaning in other communities ([2]).

Using this definition, whether the policy counts as distributed or not depends on whether the observation function $O_i$ uses communication or not. Since DGPPO is agnostic to both cases, we refer to it as a distributed policy.

---

Q2: The modified constraint (11b) is too strict which cannot be satisfied by a lot of policy classes, such as the Gaussian policy.

R: Even if the modified constraint (11b) is too strict to be satisfied theoretically, empirically this results in a large increase in safety compared to the original more relaxed constraint (10) without significant differences in the cost (Figure 8 in App. B.6.1, Figure 9 in App. C.6.1 of the revised version).

More importantly, this formulation enables the use of cheap gradient projections as in Theorem 3 to improve the sample efficiency, which outperforms both approaches above.

While we are currently unable to explain this gap, we believe the empirical results are sufficient and leave stronger theoretical explanations of this phenomenon as future work.

---

Q3: In (12), when the safety condition is violated, the expression for the gradient, which requires the policy-gradient theorem, is not associated with the score function gradient in (41).

R: Good catch, this is a typo. Thank you for the careful reading!

Elaborating a bit more, there is a distinction between

\nabla_\theta \mathbb{E}\_{\mathbf{x} \sim \rho^{\pi_\theta}}[ \tilde{C}\_\theta^{(m)}(\mathbf{x}) ] = \nabla\_\theta \mathbb{E}_\{\mathbf{x}\_0 \sim \rho\_0, \mathbf{u} \sim \pi\_\theta}\left[ \sum\_{k=0}^\infty \max\big(0, \tilde{C}(\mathbf{x}^k, \mathbf{u}^k) \big) \right] \tag{$\star$}

which requires the policy-gradient theorem to compute the gradient, and

$$\nabla_\theta \mathbb{E}\_{\mathbf{x} \sim \rho}[ \tilde{C}\_\theta^{(m)}(\mathbf{x}) ] \Bigg|\_{\rho = \rho^{\pi_\theta}} = \mathbb{E}\_{\mathbf{x} \sim \rho^{\pi_\theta}}[ \nabla_\theta \tilde{C}\_\theta^{(m)}(\mathbf{x}) ] \tag{$\dagger$}$$

which only uses the score function gradient.

In particular, while our original goal is to satisfy the DCBF condition everywhere, i.e.,

$$\tilde{C}_\theta^{(m)}(\mathbf{x}) \leq 0 \quad \forall \mathbf{x},$$

($\star$) will also try to minimize the stationary density at states $\mathbf{x}$ where the constraint is violated, which is not what we want. Hence, we use ($\dagger$) in (12) and in our experiments.

We have included the above explanation in Appendix B.2 in the revised version.

To verify this, we have also performed additional experiments where we try using ($\star$) instead of ($\dagger$). The results show that using ($\star$) behaves overly conservatively and converges much slower in cost. This matches the discussion above, that using ($\star$) additionally tries to avoid states where the DCBF constraint violation is high, which leads to unnecessary conservatism. We have included these new results in Appendix B.2 in the revised version.

We thank the reviewers for finding our approach innovative and well-motivated, and for acknowledging our theoretical presentations. There are important misunderstandings that we hope to have clarified in both our revised manuscript and our response below.

We hope that our responses below and our revised manuscript address the concerns raised by the reviewer.

Summary

In brief, we have:

1. Clarified that the "cost" in our paper is the same as the "negative reward" in the CMDP setting
2. Provided additional experimental results showing that DGPPO can generalize to 512 agents
3. Provided experimental results to clarify that larger $\nu$ of DGPPO only affects the convergence rate and not the converged performance
4. Discussed the sampling efficiency of DGPPO

Detailed Reply

Weakness 1, Question 1 (1/2): Without metrics like mean global success rate or reward, it is unclear if the agents are merely achieving safety or are successfully accomplishing the global objective.

R: We already report the global reward $**global reward** = - **global cost**$. This may have been a misunderstanding.

In our paper, the joint cost $l$ is task-oriented, equivalent to the negative reward, and unrelated to the safety constraints. This follows the convention in the optimal control community (e.g., [1]) and is NOT the cost in the CMDP setting.

For example, in Appendix B.2.1 of our original submission (Appendix C.2.1 of the revision), we have provided detailed equations for the cost, which are mainly distances of the agents to their goals. Therefore, minimizing costs drives the agents to their goals.

To prevent confusion, we clarify that the cost $l$ is task-oriented and not safety-oriented in Section 3.1 of the revised paper:

• "Let the joint cost function $\text{describing the desired task}$ be denoted as $l: \mathcal{X} \times\mathcal U\to\mathbb R$"

We have also added a footnote in the revised paper to explain the relationship with the CMDP setting:

• "The cost function l here is not the cost in CMDP. Rather, it corresponds to the negative reward in CMDP."

Consequently, the results (e.g., Figure 3, 4, 5, 6 in Section 5) do show metrics for accomplishing the global objective. For example, in Figure 3, since achieving the objective (by minimizing the cost) and achieving safety can be conflicting objectives, some baseline methods have high task performance (i.e., low cost) but low safety rates, while some baseline methods have low task performance (i.e., high cost) but high safety rates.

---

Weakness 1, Question 1 (2/2): Does the proposed framework present mechanisms to ensure that safety constraints do not dominate the objective?

R: Yes, but only if the safety constraints remain satisfied.

This can be seen in Equation (20) or Figure 1 in the original submission:

• At states $\mathbf{x}$ where the safety constraints are satisfied ($\tilde C^{(m)}(\mathbf{x}) \leq 0$), we use the gradient of the objective function
• Otherwise, we use the gradient of the constraint

Hence, if the safety constraints are all satisfied, then only the gradient of the objective function will be used.

We thank the reviewer for acknowledging our DGPPO as "an elegant way to solve the discrete-time distributed MASOCP (multi-agent safe optimal control problem) with unknown environment dynamics", and for acknowledging our theoretical analysis. We hope that our responses below and our revised manuscript address the concerns raised by the reviewer.

Summary

In brief, we have:

1. Explained that designing a nominal policy requires simple or known dynamics and can lead to deadlocks;
2. Added more experiments to show that the DGPPO policy also generalizes to more agents (512) after training;
3. Clarified some details regarding DGPPO including the sampling efficiency, hyperparameter sensitivity, and the definition of the avoid set;
4. Clarified the details of the environments in our experiments and the results using more figures;
5. Explained the difference in dynamics considered in our paper compared with [1].

Detailed Reply

Question 1: Why is the dependency of the algorithm on a nominal policy a bad idea in the given setting? Since it appears easy enough to construct one (say a PID controller like in [1]) for the environments given, is this the right direction?

R: Good question! There are 2 reasons why depending on a nominal policy is a drawback.

1. Requirement of Simple or Known Dynamics

Controller design usually requires the dynamics to be simple or requires knowledge of the dynamics. The PID controllers in [1] are constructed for the unicycle dynamics. More generally, PID controllers are usually only used with single-input single-output systems. For more complicated systems, one could use LQR or MPC, but this requires full knowledge of the dynamics. In addition, PID controllers are much more difficult to apply in environments with complex contact dynamics, for example, our Transport, Wheel, and Transport2 environments.

2. Deadlocks

Another drawback is that the CBF-QP approach of [1] leads to deadlocks, as discussed in Section VIII of [1] or theoretically in e.g. [2]. This is because the safety filter approach of [1] only minimizes deviations from the nominal policy at the current time step, even if this leads to a deadlock at a future time step.

In contrast, minimizing the cumulative cost directly takes future behavior into account and hence will try to avoid deadlocks. To investigate this, we have performed an additional experiment in Appendix D.1 in the revised manuscript, where the approach from [1] results in a deadlock while our method successfully completes the task (See Figure 15 in the revised manuscript).

We have added the above discussion on the limitations of assuming a nominal policy in Appendix D.1 of the revised manuscript.

---

Question 2, Weakness 1: Scalability appears limited (only up to 7 agents) compared to the continuous time setting of GCBF+ [1]. Can the algorithm generalize to larger numbers of agents? Does the algorithm need to be retrained for every new number of agents?

R: Good observation! We wish to make a distinction between the scalability (number of agents during training) and generalizability (ability to be deployed with more agents during test time).

Scalability-wise, GCBF+ also trains on 8 agents (Section VI of [1]), which is similar to the number of agents considered in our training. Generalizability-wise, GCBF+ can be deployed on 512 agents without significant performance loss after training.

We have performed new experiments and find that DGPPO is also able to generalize and be deployed on 512 agents after being trained on 8 agents. The following table shows the safety rates and the normalized cost (w.r.t. traveling distance and number of agents). We can observe that DGPPO maintains high safety rates and low costs when deployed on up to 512 agents.

From the results above, we can conclude that DGPPO does not need to be retrained when deploying for larger numbers of agents.

We have added the above results in Appendix C.8 of the revised manuscript.