Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game

We thank Reviewer UzHb for recognizing the significance, problem setting, theoretical concept, solution concept and evaluation of our paper. A detailed response is provided below.

Q1: Self-containment and conciseness.

We appreciate this valuable feedback. To address this, we have:

• Removed unnecessary discussions, reducing references to the appendix from 17 times to 8 times and decreasing the appendix length from 21 pages to 12 pages. We have meticulously revised the manuscript to ensure it remains focused on the core message. For completeness, we will host a GitHub page for additional experiment results upon acceptance.

• Included a concise proof sketch for each theorem to enhance self-containment.

• Reduced the use of mathematical notations and added more textual explanations.

Please refer to our global response and the revised manuscript for more details.

Q2: Why do adversaries condition on observations despite them not being Markovian (in contrast to the non-adversarial policies)? Is this intentional or a mistake?

Thank you for pointing out the concern. Please allow us to clarify that in our original draft, for both defender and adversary, their policies are conditioned on histories $H$, not observations $o$. For example:

• Section. 2.1, paragraph 2, "...each agent $i$ selects its action $a_t^i \in \mathcal A^i$ using its policy $\pi^i(\cdot|H_i^t)$"; "the goal of all agents is to learn a joint policy ... that maximize the long-term return "$J(\mathbf{\pi}) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r_t|s_0, \mathbf{a}_t \sim \pi(\cdot|H_t)\right]$".
• Section. 2.2, paragraph 2, "replaced by action $\hat{a}^i_t$ sampled from an adversary with policy $\hat{\pi}^i(\cdot|H_t^i, \theta)$"; "the value function can be defined as $V_\theta(s) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r_t|s_0=s, \mathbf{a}_t \sim \pi(\cdot|H_t), \hat{\mathbf{a}}_t \sim \hat{\pi}(\cdot|H_t, \theta)\right]$".

A thorough review of our paper confirms that the policy of both defenders and adversaries rely on histories, not observations.

Q3: Do you have any intuition on why M3DDPG performs so poorly in the presence of an adversary compared to the standard MAPPO without any robustness mechanism in Figure 4?

There are two key reasons for M3DDPG’s underperformance in adversarial contexts, as compared to standard MAPPO without robustness mechanisms.

First, as discussed in Proposition 2.2, the existence of a pure-strategy equilibrium (RMPBE) isn't guaranteed. M3DDPG, which derives from MADDPG, inherently generates deterministic policies. Such policies may not be optimal in robust cooperative MARL contexts where a pure-strategy equilibrium is not assured. Despite this, M3DDPG was included as a baseline due to its recognition in the field.

Second, as a part of the max entropy RL family, MAPPO employs stochastic policies that encourage diverse solution strategies, thereby enhancing resilience to perturbations. This advantage of max entropy policies is well-documented in several studies [1, 2, 3, 4].

[1] Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Ph.D. Thesis, Carnegie Mellon University, 2010.

[2] Haarnoja, Tuomas, et al. "Reinforcement learning with deep energy-based policies." ICML 2017.

[3] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." ICML 2018.

[4] Eysenbach, Benjamin, and Sergey Levine. "Maximum entropy rl (provably) solves some robust rl problems." ICLR 2022.

Minors:

Thank you for your suggestions. We have meticulously revised the manuscript to correct grammatical errors and enhanced the clarity of Figure 2 in our revised submission.

Thank you for the rebuttal and the revised version. Since I asked for a revision that affected the whole paper, I needed to completely re-read the new version to check.

The revision reads well to me now. Therefore, I am happy to increase my score since all my concerns have been sufficiently addressed.

We thank Reviewer EssV for appreciating the theoretical formulation and empirical evaluation for our paper. A detailed response is provided below.

Q1: Convergence cannot be guaranteed.

Thank you for pointing out the concern. Firstly, please allow us to clarify that we have established convergence in our original draft. As stated on page 6, Section 3.2, "Using common assumptions in stochastic approximation proposed by Borkar (Borkar, 1997; Borkar & Meyn, 2000; Borkar, 2009), we show that Theorem 3.1 converges almost surely to ex interim robust Markov perfect Bayesian equilibrium, with detailed assumptions and proof deferred to Appendix. A.8". This proof leverages stochastic approximation theory and is underscored in both our abstract and introduction.

Secondly, the term "almost sure" convergence implies that the sequence of defender and attacker policies converge with probability 1 as the number of iterations goes to infinity.

Thirdly, for greater clarity, we have added a bolded subtitle "Convergence" in Section 3.2 and restructured the relevant paragraph.

Fourthly, while our approach and methodology indeed rely on specific assumptions and mathematical techniques, the implementation of our algorithm is practical and straightforward. It involves conditioning the policy and value function on the belief of the current type, combined with a two-timescale update strategy (i.e., faster policy updates for the adversary and slower for the defender).

Q2: The paper is too long, and reference appendix for too many times. Concentrate on the most important parts.

We appreciate this valuable feedback. To address this, we have:

• Removed unnecessary discussions, reducing references to the appendix from 17 times to 8 times and decreasing the appendix length from 21 pages to 12 pages. We have meticulously revised the manuscript to ensure it remains focused on the core message. For completeness, we will host a GitHub page for additional experiment results upon acceptance.

• Included a concise proof sketch for each theorem to enhance self-containment.

• Reduced the use of mathematical notations and added more textual explanations.

Please refer to our global response and the revised manuscript for more details.

Q3: The paper is hard to read, it assumes significant knowledge of the field and is full of acronyms and citations that break the flow of the text.

Thank you for this suggestion. In our revised version, we have carefully go through the whole paper and reduced the use of mathematical notations and added more textual explanations. However, it's important to note that certain expressions equations cannot be overly simplified for correctness.

Dear Reviewer EssV,

We sincerely thank you for raising the score of our paper, the paper's quality has been greatly enhanced by your tremendous efforts. As the reviewer-author discussion period ends tomorrow, we hope that our recent updates and responses have adequately addressed your queries and concerns. Should you have any further questions, we are committed to promptly addressing them within the remaining timeframe.

Best regards, Authors of Submission 872

We thank Reviewer FinH for recognizing the significance, clarity, theoretical contribution and empirical findings of our paper. A detailed response is provided below.

Q1: The document frequently refers to its appendix, suggesting that it does not stand alone as effectively as it could. The authors should further take simplification, such as using simpler terms within the equations for greater clarity.

We appreciate this valuable feedback. To address this, we have:

• Removed unnecessary discussions, reducing references to the appendix from 17 times to 8 times and decreasing the appendix length from 21 pages to 12 pages. We have meticulously revised the manuscript to ensure it remains focused on the core message. For completeness, we will host a GitHub page for additional experiment results upon acceptance.

• Included a concise proof sketch for each theorem to enhance self-containment.

• Reduced the use of mathematical notations and added more textual explanations.

Please refer to our global response and the revised manuscript for more details.

Q2: How can the proposed method benefit real-world applications? Any concrete examples?

Let's illustrate this by examples. In robot swarm control [1, 2], individual robots may behave unpredictably due to malfunctions or external interference [3, 4]. Our approach studies the lower bound for these scenarios where a swarm faces a worst-case, non-oblivious adversary, as discussed in our introduction.

Apart from robot swarms, in traffic light control [5], individual lights might malfunction or be compromised, leading to a non-cooperative system. Similarly, in power grid control [6], nodes might fail or transmit erroneous signals. Our cooperative MARL approach is designed to adaptively respond to these challenges, ensuring efficient operation of these systems during deployment. We plan to apply our algorithm in these domains as future work, as discussed in our conclusion section.

[1] Hüttenrauch, Maximilian, Sosic Adrian, and Gerhard Neumann. "Deep reinforcement learning for swarm systems." JMLR 2019.

[2] Batra, Sumeet, et al. "Decentralized control of quadrotor swarms with end-to-end deep reinforcement learning." CoRL 2022.

[3] Giray, Sait Murat. "Anatomy of unmanned aerial vehicle hijacking with signal spoofing." RAST 2013.

[4] Ly, Bora, and Romny Ly. "Cybersecurity in unmanned aerial vehicles (UAVs)." Journal of Cyber Security Technology 2021.

[5] Chu, Tianshu, et al. "Multi-agent deep reinforcement learning for large-scale traffic signal control." IEEE TITS 2019.

[6] Xi, Lei, et al. "Smart generation control based on multi-agent reinforcement learning with the idea of the time tunnel." Energy 2018.