Robustness to Multi-Modal Environment Uncertainty in MARL using Curriculum Learning

We thank Reviewer yumG19 for the detailed comment and insightful feedback to make our paper better.

1. While the methodology and results are well-detailed, it would be better to include a brief introduction to curriculum learning and its traditional applications for readers unfamiliar with the concept. For example, in 5.1 EFFICIENT LOOKAHEAD CL, it’s hard to understand what algorithm 1 is doing, especially without adequate context of explanations. The same problem in 5.2 EFFICIENT LOOKAHEAD CL FOR MULTIPLE UNCERTAINTIES, algorithms 1a and 1b.

Thank you for the suggestion. We have added more details and clarity in our presentation.

2. The problem formulation is too straightforward that is a simple combination of the robust MARL problem formulation from the existing literature, thus it lacks a unique perspective or innovative twist that could differentiate this approach from previous works.

Thank you for your suggestion. We will add more analysis and insights to make it more innovative and unique.

3. When careful review of the "related work" section in the paper, I noticed that the author's coverage of previous research is not comprehensive. More critically, there seems to be a misreference. For instance, in the proof of existence of NE policy the author refers to Han et al. (2022), but from the context, it appears that He et al. (2023) should have been cited. Such oversight can not only confuse readers but also potentially mislead other researchers.

Thank you for pointing it out. We have fixed the issue.

4. After a thorough review of the paper, I observed that the author's reliance on the theories from He et al. (2023) and Kardes ̧ et al. (2011b) to demonstrate the existence of Nash equilibrium within their problem formulation appears to be flawed. The central issue is that both of these references do not account for multi-modal uncertainty. To be more precise, neither of them consider the three distinct types of uncertainties. Therefore, using these theories as foundational proof in the current context may lead to incorrect or incomplete conclusions.

We have not used these works to show the proof for multi-modal uncertainty. In our work we only show proof for Theorm 1 which states that if optimal Nash equilibrium exists then optimal bellman equation also exists. However the existing works (He et al. (2023) and Kardes ̧ et al. (2011b) ) were only cited to show that they do complete proofs for the respective single uncertainties and how it becomes complex for multi-modal case.

5. In reviewing the experimental section, I found several aspects unclear, which I believe need further elaboration for the reader's comprehensive understanding. Firstly, in section 6.1 titled "ROBUSTNESS UNDER UNCERTAINTY IN A SINGLE PARAMETER," the exact baseline algorithms the author is comparing with their own are not explicitly mentioned. It raises the question, are different baseline algorithms used under different uncertainties? Even if a particular baseline algorithm only considers a single type of uncertainty, isn't it worthwhile to compare this baseline with the proposed method under other uncertainties?

We use a single baseline [1] and not different baselines for different uncertainties. We have used the same baseline and integrated different uncertainties in the same baseline following the existing literature for those uncertainties and have already done comparison as you asked. [1] Kaiqing Zhang, Tao Sun, Yunzhe Tao, Sahika Genc, Sunil Mallya, and Tamer Basar. Robust multi-agent reinforcement learning with model uncertainty. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc., 2020

Furthermore, there are several RL algorithms that consider action uncertainty. Shouldn't the author consider the multi-agent versions of these algorithms as potential baselines?

It's difficult to convert single agent work to multi-agent. It can be considered as another research paper. We have only looked into existing methods for MARL in literature.

Regarding state uncertainty, while Han et al. (2022) might not offer a comparison for various uncertainty values, there are other MARL algorithms that do. Examples include "A Robust Mean-Field Actor-Critic Reinforcement Learning Against Adversarial Perturbations on Agent States" and "Robust multi-agent reinforcement learning with state uncertainty." It would be beneficial for the paper's completeness and comparative analysis if these methods were included in the evaluations.

Thank you for mentioning these papers. RoMFAC is a very interesting but it's in the regime of adversarial attacks/perturbations, however our work is in the regime of uncertainty and hence a direct comparison is not feasible. We have already used the uncertainty definition from "Robust multi-agent reinforcement learning with state uncertainty" and compared with its results.

Thank You

Authors

We thank Reviewer TdMD for the detailed comment and insightful feedback to make our paper better.

I do not understand the motivation behind Figure 1. It merely shows the consequences of different uncertainty aspects, which are not surprising when regarding them in isolation. How does it motivate multi-modal solutions, e.g., how does it approach the problem in contrast to single-modal solutions?

Yes, your are correct Figure 1 do not show the motivation for multi-modal robustness but single robustness. Multi-modal robustness is motivated by the fact that in real world situation, any or all the environment parameters can get faulty, perturbed etc. So the robustness problem should be studied in a generalized sense.

“Note that the range of value of observations and actions is quite small as compared to that of reward. Thus, the magnitude of robustness is different for different uncertainty parameters.” - How can I confirm this? There are no ranges provided, and equations 2, 3, and 4 restrict values in the same manner, i.e., through the truncated Normal distribution.

We will add the ranges in the appendix. Thank you pointing it out.

How is convergence measured in Algorithm 1?

Convergence in a particular iteration of CL is measured by evaluating the success rate for 5000 episodes. The CL convergence is measured when the model do not achieve high success rate after training for a fixed number of epochs. We then reduce the uncertainty value by one level.

Answers to Some More Issues:

• We are in the POMDP framework.
• We cannot boil it down to zero sum Markov game in case of multi-modal uncertainty since the uncertain stochastic variables are dependent on each other and not separable.
• [1,2,3,4] work are not in-line with our work since they do not talk about robustness issue in MARL.
• CL idea is not novel since CL has been used for improving robustness in many different areas since a long time. However our algorithm of including two uncertainty is useful and results are novel. We also plan to add more analysis to show the details on sample efficiency of this method.

Thank You

Authors

Thank you for the rebuttal.

We are in the POMDP framework.

This is mentioned nowhere in the revised version. The paper focuses on (Robust) Markov Games.

We cannot boil it down to zero sum Markov game in case of multi-modal uncertainty since the uncertain stochastic variables are dependent on each other and not separable.

Is there a formal example of this? The whole maximin setup suggests that the algorithm tries to optimize a policy in a worst-case setting, i.e., a zero-sum game, where all stochastic variables could be regarded as an "adversarial team".

[1,2,3,4] work are not in-line with our work since they do not talk about robustness issue in MARL.

I would have hoped for a meaningful discussion regarding state, action, agent uncertainties instead of simply discarding it (robust or not, the Dec-POMDP setting does have a relation to the setting promoted in the paper). Therefore, I am not satisfied with the answer.

After reading the revision and the other reviews, I decided not to change my assessment. The paper needs substantial revision as I do not find all of my concerns sufficiently addressed. For example, the baseline is still unclear (what algorithm, hyperparameters, etc. are actually used?), which was noted by multiple reviewers. I do not understand why the info cannot be simply provided in Section 6, so that we can adequately assess the work's significance.

I thank the authors for responding to my comments. After going through other reviews and response from authors, my concerns remain unadressed-

• Statement of Claims: Authors justify that prior works either consider single agent settings or uncertainty in a few variables. However, this justification is not exactly correct. Even within multi-agent settings, prior works address uncertainty utilizing auxilary methods which scale to multiple parameters. For instance, [2] address uncertainty using an auxilary exploration model using variational inference. [3, 4, 5] use techniques such as intrinsic motivation and adversarial learning to counteract uncertain states in multi-agent population learning. I encourage the authors to revisit their claims and discuss prior works so that the paper is well oriented.
• Writing and Presentation: Authors have made an effort to improve the writing and presentation of the paper. However, the work needs to be further refined. Authors could discuss technical details of Lookahead, the tasks they considered and intuitive reasoning behind their design choices. In its current form, the work presents a high-level overview of the approach with little technical contribution.
• Results and Baselines: Authors mention that they consider a baseline from literature. In that case, it would be beneficial to provide technical details of the baseline. How was the baseline implemented and trained? Is the baseline borrowed from a paper? Which design choices does the baseline consider? Aditionally, to make a strong comparison and highlight the strenghts of their algorithm, authors should consider additional baselines in order to evaluate the generality of continual learning approach.

In my view, the paper presents an interesting direction which requires further refinement. The work needs to be improved from a presentation and technical perspective. On a side note, authors might want to consider anonymizing the paper before the discussion period ends. I understand that this might have been an honest mistake so I encourage the authors to act on it immediately.

We thank Reviewer P2NT for the detailed comment and insightful feedback to make our paper better.

Statement of Claims: My main concern is the statement of claims. The paper significantly overstates some of its claims by stating that it is "the first to formulate the generalised problem of robustness to multi-modal environment uncertainty in MARL", "first work to handle multi-modal uncertainty in MARL" and "first work to handle action uncertainty in MARL". Note that there is a breadth of literature which addresses environment uncertainty in multiple parameters using surprise minimization [1], variational exploration [2], intrinsic motivation [3], adversarial learning [4] and generative models [5]. Additionally, a wide variety of established MARL algorithms build on uncertainty estimation from single-agent RL methods.

We cannot apply single agent approaches to multi-agent since multi-agent suffers from additional issues like un-stationary environment etc. [1, 4] is single-agent method. [2, 3, 5, 6, 7, 8] does not talk about robustness to uncertainty hence not in-line with our work. This work is thus first to handle multi-modal (2 uncertainty in our case) uncertainty in MARL specifically.

What is the reason behind using a curriculum? What is efficient lookahead? How do SkipAhead and TrainTillSuccess work? Can you please technically explain the proposed approach and its intuition?

We have added these details and clarifications in the paper.

How many tasks are a part of the currciculum? What is the complexity of these tasks? What is the training and evaluation setup used during experiments?

We will add details about this in paper. However some details are we use the standard deviation parameter ($\epsilon, \mu, \nu$) to increase/decrease the uncertainty level. We do this in both the training and evaluation phase except for reward uncertainty which we evaluate only using the training curves since rewards don't play a role during evaluation.

Can you please consider baselines from existing literature? How does the proposed method achieve state of the art performance? How was robustness and uncertainty measured using rewards/success rates? What is the importance of a curriculum in the learning process?

Our baseline is method is from existing literature. We have added more details in the paper to give clarifications on the comparisons with existing methods. We state the model to have achieved robustness to a specific uncertainty level if it gets more than 95% success rate for set of 5000 episodes. CL makes our learning sample efficient and produces higher level of robustness.

Thank You

Authors

We thank Reviewer nCir for the detailed comment and insightful feedback to make our paper better.

1. The article is poorly written and many variables are not defined in details. For example, what is the meaning of TrainTillSuccess in algorithm 1? And what is TrainToSucc and SkipAhead？

We have corrected our mistakes and added details in the paper. Some more details here for TrainToSucc: The RL model has converged for the given level of uncertainty. Specifically for our experiments, the success rate of the agent should be greater than 0.9. SkipAhead: We continuously increase the noise parameter until the model fails to converge. Then we step back to the highest value for which the RL agent converged.

2. The core contribution of this paper is just a simple application of curriculum learning in robust RL with little innovation. And there is no detailed analysis of the sensitive parameters of curriculum learning in this paper, such as the effect of different settings of \delta(\labmda) ?

Thank you for the suggestion. We will add analysis of sample efficiency of our CL method. However we wanted to know what do you mean by different settings and what kind of settings?

3. The experimental results are not convincing. Why the results are not represented using a curve of mean-standard deviation commonly used in reinforcement learning, such as fig2?

Thank you for the suggestion. We will add mean-standard deviation training curves in our future works.

4. I didn't understand what the essential difference is between the solutions for single uncertainty and double unceritainties. What is the essential difficulty caused by the two uncertainties?

As you see in equation 6 of the paper, inorder to solve for the bellman equation we have to minimize with respected to the various model uncertainties simultaneously in case of multi-modal uncertainty problem. Here the stochastic variables are interdependent on each other and cannot be separated which is needed for bellman optimality equation proof (Unless another approach can be found out). Only in the case of reward+transition dynamics uncertainty the variable are separable which is why we have not considered transition dynamics uncertainty in our work. To give an example we sample true action as $\pi^{i}(.|\bar{s}^{i})$ which is depended on stochastic variable $\bar{s}$. Eventually we don't use true but perturbed actions defined as stochastic variable $\bar{a}$ in our bellman equation. Similarly reward is dependent on stochastic actions: $\bar{R}^{i}(s, \bar{a})$, etc. Hence making the problem to solve bellman proof for double uncertainty difficult. Single Uncertainty proofs can be checked out here :

Erim Kardes ̧, Fernando Ordóñez, and Randolph W Hall. Discounted robust stochastic games and an application to queueing control. Operations Research, 59(2):365–382, 2011.

Thank You

Authors