Attaining Human's Desirable Outcomes in Indirect Human-AI Interaction via Multi-Agent Influence Diagrams

We thank reviewer y3C4 for their suggestions and appreciate that our paper is strong and well-written. We will address each of the questions posed by the reviewer in the sections below.

Weaknesses

While the story and theory of the paper are interesting, they are less attractive in terms of method and experiments.

1. In method/algorithm 1, I could barely get information besides the alternate training of pi_pre using intrinsic rewards and pi_agent with extrinsic rewards

Answer: We agree that the algorithm does not look fancy. However, we would like to emphasize that the principle of deriving the algorithm from our theory is a more critical contribution as you observed. In other words, the main purpose of the algorithm tends to verify the validity of our theory. Furthermore, there also exist other ways to implement our theory.

2. In experiment session, I would also demonstrate the method some games where the diverse NE(conventions) are easy to tell and has abundant prior work because nothing prevents the methods to be applied to the general-sum setting.

Answer: We agree that a case that can easily demonstrate the results may make people better understand our work. In our MPE experiment, we actually have provided a theoretical analysis of the result in Appendix 11, where diverse NEs are easy to tell.

3. Possibly missing literature on multi-agent diversity. I believe the diverse convention finding literature is relevant because you can find them in the first place and select among NEs. Here is a few examples [1][2][3]

Answer: Thank you for your suggestion. We understand your viewpoint. Although diverse multi-agent behaviours are preferred in our work, how these diverse behaviours can be generated are out of our research scope. For this reason, we cannot agree with your viewpoint at this moment. Nevertheless, [1] is related to human-ai coordination which we have added to our related work and have a discussion in the revised paper.

[1] Hu H, Sadigh D. Language instructed reinforcement learning for human-ai coordination[C]//International Conference on Machine Learning. PMLR, 2023: 13584-13598.

[2] Cui, B., Lupu, A., Sokota, S., Hu, H., Wu, D. J., & Foerster, J. N. (2023). Adversarial diversity in Hanabi. In The Eleventh International Conference on Learning Representations.

[3] Rahman, M., Cui, J., & Stone, P. (2024, March). Minimum coverage sets for training robust ad hoc teamwork agents. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 16, pp. 17523-17530).

Question

1. This paper reminds me of [1]. Can the authors elaborate on how language conditioning and pre-strategy are related to each other?

Answer: Thanks for your insightful viewpoint. First, [1] focused on how to implement human-AI coordination with LLMs as a medium between a human and AI agents, to guide human-AI coordination under human's instructions. In contrast, our work stands on the broader view on studying the principle to facilitate human-AI coordination with a medium as a indirect way to convey human's explicit goal. Therefore, [1] can be regarded as a good engineering work to further verify the validity of our theory on indirect human-AI interaction, where LLMs is an specification of the AI proxies equipped with pre-policies which can generate pre-strategies. As promising above, we have added this point in the related work.

[1] Hu H, Sadigh D. Language instructed reinforcement learning for human-ai coordination[C]//International Conference on Machine Learning. PMLR, 2023: 13584-13598.

2. How do you get the intrinsic rewards? How do humans specify their preferences?

Answer: In our work, the intrinsic reward is manually crafted to encode human's desirable outcome (goal). However, the approach to represent human's desirable outcome is not the main gist of our paper as we mentioned above. We only provide a possible and initial solution. In contrast, [1] can be regarded as a kind of fancy approach to implement the algorithm we proposed, where intrinsic reward can be generated by LLMs to encode human's goal. We added an experiment to demonstrate the advantage of manually designing intrinsic rewards when it is feasible compared with the approach of learning an intrinsic reward in Appendix 18.

Thank you for your response! About the analysis of MPE, we agree that it is not a conventional case in games, but it is actually a case to show NE (since NE is just one solution concept to evaluate policies). The main reason for us to choose MPE is that this environment is more intuitive to most people with no prior knowledge of rules in Hanabi for visualization.

Automated preference design can be treated as a further extension to replace the manually designed preference reward, but how to obtain intrinsic rewards is not our main contribution of this paper. In this paper, we primarily aim to justify the necessity of intrinsic rewards to express human's intention through the framework of MAIDs. In the future work, we may investigate how to automatically infer human's intention through additional data, when the human's intention is not able to be explicitly expressed.

We would appreciate your further thoughts again and sincerely hope that you could reconsider your rating from the perspective of our main contributions. We would be highly appreciated if you could kindly reconsider to upgrade your overall rating, but we respect your decision if you choose to retain your current rating - thank you so much!

We would like to thank reviewer rjtB's feedback on our novelty, theoretical contributions as well as our idea can also useful for some general tasks. We respond to each of the reviewer's questions in the sections below.

Weaknesses

1. Make a stronger motivation for indirection human AI coordination

Answer: We agree with your discussion above. Indirect human-AI coordination can be treated as a kind of equilibrium selection problem from the layer of mathematical problem settings. From the perspective of application, we can provide another exmaple. Suppose a future scenario where a person named Bob need to interact with other AI agents (NPCs) and other AI proxies (can also be seen as AI agents to Bob in this case). Bob may not intend to expose himself to others because he would hide some private information (e.g. his facial profile). To satify this purpose, he has to hire an AI proxy.

2. Clearly contrast it with existing works and emphasize on the unique advantages that prior work cannot achieve.

The related work section on LLM alignment seems less relevant for this paper as the author makes no comparison or application into the LLM land. On the contrary, it seems to miss some of the potentially related work mentioned above. However, it could also be that I missed the main point of the paper.

Answer: [2] actually solved the similar problem setting as we proposed (indirect human-AI interaction), though they may allow AI proxies to interact with other agents not limited to AI agents. On contrary, [1] actually implemented an agent that is able to infer human's intentions in direct human-AI interaction tasks. This example could be an evidence to justify the necessity of formalizing the difference between these two paradigms. Although both paradigms share some common solutions (e.g. the case you mentioned above), it does not imply that we have to use a more complicated method when we have some exclusive benefits of the problem setting. For example, in [2] human can directly give instructions to an AI assistant, while in [1] this is not allowed. On the other hand, inspired by the proposition in [2], we suppose that indirect human-AI could be a precondition to fulfil more complicated scenarios, such as AI proxies interacting with other humans and AI proxies interacting with other AI proxies on behalf of other humans. This could be another reason to formalize this problem setting.

Second, we agree that [2] proposed to use intrinsic rewards to encode human's desirable outcomes to achieve human-desired equilibrium. However, it just treated intrinsic rewards as an approach without a justification of its validity at theoretical level. In our work, we propose to employ MAIDs, a graphical mdoel to justify the correctness in theory, which is the fundamental difference from these two previous works and a complementary work to support the reasonableness of [2]. Thanks for your so critical point. We have added this discussion in the related work. As for related work about LLM alignment, we agree with you and Reviewer Dy2f that it is with weak connection to our work. We have removed it in the revised paper.

[1] Human-level play in the game of Diplomacy by combining language models with strategic reasoning

[2] Language Instructed Reinforcement Learning for Human-AI Coordination

Questions

1. Figure3: “Figure 3 shown that our method demonstrates general faster convergence compared with the baseline.” In Figure 3.a the GNN curve looks indistinguishable from the baseline curve, and in Figure 3.b and yellow baseline curve seems to converge faster. How do you reach the conclusion?

Answer: Thanks for your suggestion. We agree that our statement is not clear enough. First, in Figure 3a as the the combination of intrinsic rewards and extrinsic rewards may induce a new solution, our method can reach this new optimal solution (a NE that complies with human's desirable outcomes) faster at about $8\times10^{4}$ timesteps. However, with no influence by intrinsic rewards, the only optimal solution of baseline algorithm is maximizing the extrinsic rewards, which has not reached at least before the end of the demonstrating time axis. To justify the existence of effects of human's desirable outcomes, we also provide the average last-step intrinsic reward evaluation for MPE in Figure 4c in the revised paper. In Figure 3b, although the yellow baseline climbs steeply in the first half of training time, our method (in green) surpasses it afterwards.

2. MPE experiments: “the greater the distance the larger intrinsic rewards” -> Shouldn’t this drive the red dot further from the leftmost landmark? In Figure 4.a it seems that the red dot is moving closer to the leftmost landmark?

Answer: Thanks for your careful inspection. This is a typo. The correct version should be "the smaller the distance the larger intrinsic rewards." We have corrected it in the revised paper.

3. Instead of using the two staged approach, how would the proposed method compare to a baseline that jointly optimize all agents while the proxy agent receives the intrinsic reward?

Answer: Thanks for your insightful suggestion. We have given the results of this experiment in Appendix 17 in the revised paper. In summary, simultaneous-update approach would be similar to or a little weaker than the two-stage approach. This is not surprising, as simultaneous update can still optimize Equation (4). However, this is not an evidence to weaken the meaningfulness of the two-staged approach, as the disentanglement of these two optimization stages can open up the possibility of pretraining pre-policy. One can suppose that the time interval between these two stages is long enough, and the pre-policy training stage can be accomplished separately (pretrained) by the collected data from the provided human feedback as intrinsic rewards.

We appreciate reviewer Dy2f's constructive feedback and positive comments on our idea is interest paradigm solve a very common real-world problem. We also thank for your recognize our paper is well-structured. We respond to each of the reviewer's questions in the following.

Weaknesses

1. The introduction of indirect human-AI interaction is not enough. From the Figure 1 (a-c) , it cannot recognized which agents represent human, which represent the AI proxies, which represent the AI agents.

Answer: Thanks for rasing your concerns to help us improve our work. First, in Definition 1 we define that AI proxies are actually on behalf of a human. As a result, human would not appear in the IHAD. As for the differentiation between a human and AI proxies, this is our carelessness in writing. Actually, the color in red indicates AI proxies while the color in blue indicates AI agents. We have clarified this point in the revised paper.

2. The author gives a proof about the existence of nash equilibrium in their game. However, they doesn't prove their proposed method, i.e., pre-strategy intervention based method can be ensured to converge to optimal nash equilibrium.

Answer: Thanks for your critical viewpoint. We first agree that giving convergence result would make the paper more complete. Nevertheless, proving the convergence of an algorithm would need extensive technical assumptions and some further clarification. This would complicate the expression of our work and weaken the main contribution of our work at this moment, as our main purpose is to propose a new angle of constructing human-AI interaction based on graphical models (MAIDs). However, we agree that this is a good future direction to extend our work.

3. The selection of baseline in experiments are not proper. I suggest the authors choose some MARL algorithms, such as MAPPO and QMIX.

Answer: We agree that adding more baselines would strengthen the quality of our work. We have added new baselines, such as MAPPO, QMIX in Appendix 16.1 in the modified paper. As we see, our method can perform well on QMIX, but does not perform well on MAPPO. This sheds light on the design of intrinsic rewards could be not only corresponding to tasks, but also related to base algorithms. This is not surprising, as a base algorithm forms an optimization technique, which in combination with optimization problems as intrinsic rewards here, naturally together affect the final converggence result.

Question

1. Whether the introduction of AI proxies is rational? Whether AI proxies can also be regarded as AI agents? If yes, the definition of indirect interaction is not true.

Answer: We understand your concerns. First, the introduction of AI proxies is reasonable, as in many cases a person is not able to interact with other AI agents (or other AI proxies on behalf other humans). In addition to the examples we gave in Introduction (related to physical constraints), we now show another example. Suppose a future scenario where a person named Bob need to interact with other AI agents (NPCs) and other AI proxies (can also be seen as AI agents to Bob in this case). Bob may not intend to expose himself to others because he would hide some private information (e.g. his facial profile). To satify this purpose, he has to hire an AI proxy.

Second, as our above example shows, the definition of AI proxies and AI agents is always dependent on the ego view of a person of interest (e.g. Bob in the above example). Although from the system-wide view AI agents and AI proxies may be overlapped in definition as you observed, this is not what we claimed in Definition 1, where we always stand on the ego veiw of a human of interest. We have emphasized this point in the revised paper.

Hope our answer has addressed your concerns.

2. Why the authors introduce LLM alignments in the related work. It seems that this research doesn't relevant to LLM alignment.

Answer: We agree with you and Reviewer rjtB that these two research fields have no connection on the surface. Thus, we have removed it in the revised paper.

Thank the authors for their response. Some key concerns still exist.

1. The response to my Question 3 was inappropriately placed within reviewer fGsJ's section without any cross-reference. The performance comparison in Figure 14 does not support the claim that "our proposed paradigm (with pre-policy) reaches the optimum more efficiently" as the difference appears negligible. The claim about handling "more complicated and realistic scenarios by separately designing pre-policy modules" seems to be questionable. In my experience, simpler approaches typically demonstrate better generalizability across different tasks.

2. Regarding Figure 1(a-c), there is no human role. I understand that if AI proxies are replaced by humans, the proposed algorithm would be applicable to Human-AI direct interaction scenarios, right?

3. It is not convincing that the proposed approach performs worse than MAPPO, especially on an easy SMAC map. The individual reward setting is indeed important for algorithm performance. I suggest the authors should investigate and identify reward settings that effectively promote cooperation between AI proxies and AI agents.

I will maintain my ratings. Thanks.

1. We sincerely apologize for the inconvenience caused by placing Question 3 in the wrong section. We agree that the claim "simpler approaches typically demonstrate better generalizability across different tasks" holds true in many scenarios. However, our method offers a novel perspective by addressing complex, realistic problems through the design of pre-policy mechanisms. The intention behind our pre-policy design is to tackle complex problems using simpler interaction strategies, which aligns with rather than contradicts this claim.

2. Yes, the graphical models we introduced in this paper can actually be extended to direct human-AI interaction scenarios. This can also be one significant novelty for our paper. In other words, albeit that the research problem is indirect human-AI interaction scenarios. The research method can also be employed to study more extensive problems. This can also be considered as one scientific contributions from our own perspectives.

3. We would like to kindly clarify that there might be some misunderstanding of our result. Although MAPPO can perform well in global rewards, it does not satisfy the human desirable outcomes (see Table 5). In summary, the main reason why the performance on task reward shown in Figure 13 is that our approach has to reach human's desirable outcomes (represented in intrinsic rewards) in addition to the task goal (represented in global rewards). This further justifies the difficulty to compromise human's desirable outcomes and the task goal, i.e., not every base algorithm can simply reach this balance, and thus the value of our research problem.

We still thank you for your response and sincerely hope you can reconsider your decision, especially depending on the third point above.

We would like to thank reviewer fGsJ's feedback on our idea is well-motivated and interesting as well as our results are promising. We respond to each of the reviewer's questions in the sections below.

Weaknesses

1. The environments used for evaluation involve discrete actions. The formula to evaluate a pre-policy involves a summation over strategy profiles. How do you expect the results to scale up to more complex environments with continuous state and action spaces, including more complex observation spaces?

Answer: Thanks for your insightful question. For continuous action spaces of pre-policy (pre-strategies), we may employ Monte Carlo sampling techniques or pick up a typical sample action. As for complicated continuous observation spaces, the key point to handle it is how to represent the observations in a compact form. We may suggest manually categorizing observations into several categories and then represent them via GNNs as we did in our first experiment. If the physical meaning of observations are obscure, we may employ some clustering techniques to acquire clusters from observations, e.g. the Principle Component Analysis (PCA). Although the above discussion is just prospect, we still directly evaluate our algorithm in a new environment in which observaitons are represented in more complex continuous values. The new experimental result is shown in Appendix 15 in the revised paper. In summary, our current method can still perform better than the baseline, but weakly. This indicates that we can improve this in the future research with the approeaches aforementioned.

2. There also isn't as much comparison of the paper's results to related works in inverse RL for multi-agent cooperation in the experiments section, although the ablations are helpful for understanding the author's strategy in developing their approach.

Answer: To our best knowledge, inverse RL is a algorithmic paradigm that learns a reward function from demonstration data and then the learned reward function can be used to train a policy. In contrast, in our paper we do not consider the case that demonstration data is provided, but we do not deny that this can be a future work to explore. Thanks for providing your viewpoint.

Question

1. Though cited in the main paper, I think it would be nice for the paper's presentation to put a motivating example of how MAIDs work in the main paper, rather than just the appendix.

Answer: Thanks for your good point. We agree with you that moving the motivating example to the main paper could ease reading. However, we may have no extra pages to realize that. For this reason, giving an indicator to link to the MAID example could be a compromise.

2. The paper could also benefit from a more extensive results discussion. I understand there is a lot of background to be laid out about causal models, etc., but I think these additions would actually help motivate and enhance readers' understanding of the work.

Answer: If we understand you question correctly, we would say that we have involved detailed discussions on specific examples about pre-strategy interventions in Appendix 9 and the MPE experimental results in Appendix 11. If our understanding is incorrect, we look forward to your clarification and then give further responses.

3. Whether the comprison experiments can be performed when pre-strategy intervention is not introduced into the approach.

Answer: We have added the comparison in Appendix 16.2 in the revised paper. Since both paradigms are trained with intrinsic rewards, both of them finally reach the similar performance. However, our proposed paradigm (with pre-policy) reaches the optimum more efficiently. On the other hand, due to the flexibility of our proposed paradigm, it is potential to handle more complicated and realistic scenarios by separately designing pre-policy modules. This is what the paradigm without pre-policy cannot achieve.

Thank you for your timely response and raising your soundness/presentation score to 4! We are glad to found the additions/responses thoroughly address your concerns : ) We would be highly appreciated if you could kindly reconsider to upgrade your overall rating, but we respect your decision if you choose to retain your current rating - thank you so much!