Safe Multi-agent Reinforcement Learning with Protection Motivation Theory

1. The motivation seems unclear to me. The author use human behavior as motivation, but failed to point out what problems exists in current works, as been discussed in Introduction section. If incorporating human behavior is a must, then it should be solving some limitations of current works, yet this part is missing.

2. The authors should consider evaluating their method on more tasks. Safe-MAMujoco, Safe MAIG and MAPDN all contains 10-20 tasks, yet the authors evaluated only 2 tasks in Safe-MAMujoco and 1 in Safe MAIG. The authors can consider adding additional 3-6 tasks on Safe-MAMujoco and Safe MAIG.

3. The gain in safety seems minor in Fig. 1, 2 and 3, especially comparing with MACPO. I would say there is a strong overlap between the curves of proposed method and MACPO. I would suggest authors to evaluate the safety measure on some more challenging tasks.

4. The problem of safe MARL is not a MDP. Typically, MDP modells the decision process of single-agent, when in multi-agent case, it's commonly formulated as a Dec-POMDP or Markov Game. So the problem formulation is incorrect. According to experiments I guess it's some sort of safe Dec-POMDP. Also refer to MACPO for their problem formulation.

5. The author should add surveys on safe MARL literature in preliminaries.

6. In Sec. 3.1.2, many derivations are based on existing literatures. Maybe it is better to focus on the central derivations.

7. What are the guarantees for "fear for safety guarantee"? I suppose it to be some type of bounds, but failed to find any.

Minor: Seems the paper do not follow the ICLR template and exceeds the page limit. Also, there are many grammar errors (eg, In this paper, we introduce PMT into the MARL to address the challenge safety. in line 067-068).

1. Some components of the algorithms are not clear, especially for the optimization of FPN and RPN.
2. The application of Lagrange dual is a main component of the proposed algorithms, while it has been used in many related works. Besides, the learning of FPN and RPN is more like the learning of cost function.
3. In the experiments, it seems that curves have not converges, or the performance of proposed algorithms is not obviously better than baselines.

(1) This paper introduces “fear” and “regret” for pre-decision risk assessment and post-decision reflection respectively. However, the mixed model doesn’t enhance performance, which contradicts real-world scenarios where humans often experience multiple emotions simultaneously. An effective framework to integrate the two emotions is lacking.

(2) The experimental analysis is relatively brief. Since the paper proposes two emotion models, it should provide a more detailed comparative analysis of their effectiveness in different scenarios and explore suitable application contexts to better guide practical use of the methods.

(3) This paper lacks a time complexity analysis, which limits the evaluation of the model’s feasibility for real-world use.

I find the paper difficult to follow; many equations are listed without interpretations. Additionally, the paper lacks a comprehensive discussion of related work. While PMT serves as good inspiration for the method, I am not entirely sure how the essence of the proposed methods differs from other traditional safe MARL methods.

However, some weakness significantly hinders readers from further evaluating the contribution and importance of the work:

1. Annotation & Mathematical Derivation: the presentation of the work, especially regarding the theoretical part (part 3), is very chaotic. First, many annotations are not introduced during mathematical derivations. For example, in your introduction of FPN, $\tilde{f}^i=F P N\left(s; \zeta^i\right)$, what is $\zeta$ here? Also, in Equation (10), what are $B$ and $T_s$ here? Each annotation should be introduced when it first appears in the paper.

2. Proposed Theoretical and Loss Function Design: I do agree introducing the fear and regret mechanism is interesting, but why should the loss function of your FPN and RPN have loss functions like Equation (4) and (14)? What is the theoretical intuition and explanation for Equation (4) and (14)? Also, in Equation (3), why does the cost function suddenly have probability distribution $p(C^i)$? In Equation (13), what does the cost function $\mathcal{C}\left(s, a^i\right)=1$ and $\mathcal{C}\left(s, a^i\right)=0$ mean?

3. Experiments and Hyperparameters: The experimental section needs more details about the hyperparameters used in your network training - what are the specific hyperparameter settings for each algorithm, including yours? Also, while you show the average costs, what's the actual constraint violation rate for each method? Additionally, I see you focus on improving the Lagrangian safe RL approach, but how does your method compare with those algorithms that claim zero constraint violation, like [1]?

4. The proposed PMT framework doesn't seem specifically designed for multi-agent settings - would it work equally well in single-agent scenarios? What's your motivation for choosing a multi-agent setting? The paper needs to better justify why the PMT framework is particularly suitable or important for multi-agent rather than single-agent problems.

[1] Liu T, Zhou R, Kalathil D, et al. Learning policies with zero or bounded constraint violation for constrained mdps[J]. Advances in Neural Information Processing Systems, 2021, 34: 17183-17193.