CAMMARL: Conformal Action Modeling in Multi Agent Reinforcement Learning

Thank you for the insightful comments and suggestions. We have addressed your concerns below.

1. Scalability: Even with a small number of agents, MARL can be quite challenging to solve. Our main goal in this paper was to develop a framework for conformal action prediction for MARL. For every agent, the conformal set would grow by $|A_{other}|$. However, we can also use a common last layer for all the conformal prediction models and append that layer to the input states. That way, we can incorporate multiple agents seamlessly. Generally, even in a setting of many agents, for instance, 100 or 1000, one could also prefer modeling only a local set of agents instead of all of them, hence reducing the complexity of the scalability problem. Nonetheless, these ideas require a modified/upgraded design to be investigated and would be an interesting follow-up work.

2. Performance with respect to Baselines: While we do agree that the performance improvement is not quite significant in CN, for LBF it is quite significant especially when considering the baselines, which have more full information about the actions of the other agents before taking any decisions. However, note that for CN, the baseline that CAMMARL majorly overlaps with EAP is actually CAMMARL with set size=1. Since action prediction is quite simple in CN, conformal predictions are not much better as in environments like LBF. GIAM is clairvoyant in that it has access to all the information, so comparing it against CAMMARL is unfair, and is meant as an upper bound on performance. In comparison to the other baselines, CAMMARL does better even in CN.

3. Multi Agents: CAMMARL as an idea is not restrictive to two agents, and can be easily adapted to incorporate multiple agents. Results in mixed settings and more than 2 agents: We also show results in the non-cooperative Level Based Foraging Setting for 2 agents with 6 food locations in Section D.1 of the Appendix in the revised version of the paper. From this section, we can see that the utility of CAMMARL is not just restricted to cooperative settings, but it can be easily applied to a multitude of MARL problems. We also show results on Pressure Plate with 4 agents which is a new environment described in Appendix Section A of the revised version. In Section D.2 of the revised paper, we show that CAMMARL converges 10% faster (in terms of the number of episodes) when compared with the other baselines in the domain. This definitely lends further credibility to conformal predictions, and additionally shows the adaptability of CAMMARL to other MARL domains with more agents.

4. Does N_other's action also condition on N_self's action?: No, it does not. The other agent has its own set of observations and actions and is not influenced by the actions of the main agent directly.

Given the limited compute resources, we believe that we could answer all your doubts and clarifications as much as possible. We would be grateful if you could consider increasing your score if you find our responses adequate.

We thank the reviewer for their detailed comments and suggestions for improving the paper. We address all the concerns below.

1. Regularization Term in Conformal Prediction: We apologize for the lack of clarity in the explanation. Each time the conformal model is trained, we choose the lambda and k_reg parameters from a set that optimizes for small sizes of the action set. The k_reg values are dependent on the logits, however, the lambda values are in a set [0.001, 0.01, 0.1, 0.2, 0.5]. We have added a section in the Appendix explaining this. We apologize again, for the confusion and we have added this explanation in Section C of the Appendix in the revised version of the paper.

2. Hyper-parameters for the Baselines and CAMMARL: Thank you for pointing this out. Yes, the hyper-parameters are different for the different baseline. We performed a moderate tabular search over the hyper-parameters and chose the best hyper-parameter to represent each of the individual baselines.

3. Visualization of Policies: For all the environments, rewards provide a way to understand cooperation as well, as in all the domains, the rewards are designed to be higher for better cooperation. We show a learned policy by CAMMARL in Pressure Plate in Section D.3 of the Appendix of the revised version of the paper. We can see perfect cooperation among the 4 agents in the environment, with each agent staying behind to keep the gateway open.

4. Drawbacks and Sensitivity Issues: The main drawback would be using an extra neural network for conformal action prediction, but this would be true for other methods that model actions from observations as well. Like any RL method, our method would also be sensitive to hyper-parameters, however, for the conformal regularization parameters, we sweep over every conformal model training step to find the optimal one for conformal sets. In terms of calibration, we adopt the same procedure as in RAPS [1], and get a conformal calibration set from the data. The core conformal prediction algorithm is exactly the same as RAPS. We hope this clarifies your doubts.

5. Fictitious Play: Thank you for pointing out this interesting comparison, and yes intuitively it is kind of similar to fictitious play. All the action modeling baselines include actions from the previous observations, and so they would be intuitively similar as well. However, the experiments we tested are all non-zero-sum games and we did not find any cycling behavior or non-convergent behavior in our experiments.

6. Clarification of GIAM and CAMMARL inputs: We apologize for the confusion. GIAM is infeasible because it uses the real actions of the other agents as input for taking actions, whereas CAMMARL builds a model from replay data. GIAM is in a sense clairvoyant because before taking an action, it has access to the true actions of the other agents. So, GIAM is infeasible in real-world scenarios.

7. Different Set Sizes of Conformal Model: In CAMMARL, the conformal prediction model is a separate MLP that learns (simultaneously with the RL agent) to output conformal prediction sets. However, these sets can be of varying sizes in different situations. So, we came up with numerous ways to be able to use them to implement CAMMARL. One of the ways is to pad the output set with zeros (or any other constant value) to bring them to the same fixed size (the total number of possible actions). Another way is to pick up the last layer of the conformal prediction model’s MLP as embeddings (hence always maintaining a fixed size) instead of sending the conformal set as input for the n_self agent. We provide some more details on this in Appendix B of the revised version with more experiments showing related results.

[1] Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193, 2020.

Thank you for the thoughtful review. In our rebuttal, we have addressed your concerns sequentially.

1. Novelty: While we added conformal modeling for action predictions, we explored different ways of incorporating them into the main RL algorithm. Since the conformal predictions can have different set sizes, we explored ways of incorporating them into the algorithm including using the embeddings of the model directly. Our main goal is to highlight that conformal predictions can be a really useful tool for agent modeling in MARL and can be more beneficial than just predicting the exact actions or their probabilities. Simple Envs: Multi-Agent Reinforcement Learning is in general not an easy problem to solve, and our environments are indeed difficult to solve based on the performance of NOAM in CN and LBF which does not do quite as well.

2. Results in mixed settings and more than 2 agents: We also show results in the non-cooperative Level Based Foraging Setting for 2 agents with 6 food locations in Section D.1 of the Appendix in the revised version of the paper. From this section, we can see that the utility of CAMMARL is not just restricted to cooperative settings, but it can be easily applied to a multitude of MARL problems. We also show results on Pressure Plate with 4 agents which is a new environment described in Appendix Section A of the revised version. In Section D.2 of the revised paper, we show that CAMMARL converges 10% faster (in terms of the number of episodes) when compared with the other baselines in the domain. This definitely lends further credibility to conformal predictions, and additionally shows the adaptability of CAMMARL to other MARL domains with more agents.

3. Algorithm 1: We apologize for the lack of clarity. We have made the changes in the revised version of the paper.

4. $b_{conformal}$: $b_{conformal}$ is the replay buffer for the conformal model. It is the exact same as the replay buffer for $N_{other}$ ($b_{other}$) except the conformal model has a different update frequency than $b_{other}$.

Because of our limited compute resources, we tried to include as many new experiments as possible. We hope this rebuttal and the new experiments will clarify your doubts and answer your questions. We would be grateful if you could consider increasing your score if you find our responses satisfactory.

We thank the reviewer for the helpful comments and suggestions to improve the paper. We address all the points mentioned sequentially below.

1. Performance with respect to Baselines: While we do agree that the performance improvement is not quite significant in CN, for LBF it is quite significant especially when considering the baselines, which have more full information about the actions of the other agents before taking any decisions. However, note that for CN, the baseline that CAMMARL majorly overlaps with EAP is actually CAMMARL with set size=1. Since action prediction is quite simple in CN, conformal predictions are not much better as in environments like LBF. GIAM is clairvoyant in that it has access to all the information, so comparing it against CAMMARL is unfair, and is meant as an upper bound on performance. In comparison to the other baselines, CAMMARL does better even in CN.

2. More Domains: CAMMARL as an idea is not restrictive to fully cooperative worlds and is generalizable to mixed/competitive settings. More specifically, if an agent in a mixed or competitive setting is required to model other agents in the environment, developers can consider applying CAMMARL in such situations too.

Results in mixed settings and more than 2 agents: We also show results in the non-cooperative Level Based Foraging Setting for 2 agents with 6 food locations in Section D.1 of the Appendix in the revised version of the paper. From this section, we can see that the utility of CAMMARL is not just restricted to cooperative settings, but it can be easily applied to a multitude of MARL problems. We also show results on Pressure Plate with 4 agents which is a new environment described in Appendix Section A of the revised version. In Section D.2 of the revised paper, we show that CAMMARL converges 10% faster (in terms of the number of episodes) when compared with the other baselines in the domain. This definitely lends further credibility to conformal predictions, and additionally shows the adaptability of CAMMARL to other MARL domains with more agents.

3. Using Observations of other agents: We thank the reviewer for raising this important point. It is indeed a strong assumption particularly in the case of decentralized multi-agent learning, even with observations the problem is still quite challenging to solve. Also, in problems without ego-centric views, generally, the entire observation is available and the key challenge then is figuring out the actions of the other agents for complete information. We have added this as a limitation in Section 7 paragraph 2.

Given our limited computing resources, we hope we can address all of your concerns. Please consider increasing your score if you find our responses satisfactory. Additionally, we would be happy to answer any follow-up questions.