Opponent Modeling based on Subgoal Inference

Thanks for your valuable comments. As follows, we address your concerns in detail.

Experiments: It's a bit disappointing that in most of the presented experiments, the confidence intervals of OMG and baselines generally overlap (except in the predator-prey environment). I understand that controlling a single agent makes it hard to jump from 10% to 90%. However, I suggest choosing environments that amplify the impact of a single agent. For instance, the 3s_vs_5z environment seems well-chosen (hence the highest advantage). In either case, increasing the number of trials to make confidence intervals tighter would also help.

While OMG does not show a significant advantage in the experiments, it maintains a modest but consistent edge. This is reasonable, considering that opponent-predicting ability is not the sole factor influencing performance.

We have increased the number of runs to 10 seeds for Predator-prey, and the result presents in the PDF file of "global" response.

Equations 6-7: It's confusing that g in V(s_t, g) is the opponent's subgoal, while the notation resembles the standard goal-conditioned value function. I suggest replacing it with V(s_t, g^o), similar to a^o, or using any other indication that it's the opponent's subgoal. Additionally, to analyze the impact of choosing either (6) or (7), you should include an ablation (even a single experiment with just the results provided) in which you compare to naive choices, such as a random state or a fixed-distance state (e.g., s_t+3). This is especially important given that in Appendix D you report that 80% of subgoals are <=3 steps.

Following your suggestion, we have added ablation experiments. In these experiments, OMG-random, OMG-1s, and OMG-3s represent subgoals selected from the opponent's future states randomly, at the next step, and at three steps ahead, respectively. The results present in the PDF file of "global" response.

l.139: Technically, by modeling the opponents' subgoals, you don't gain any new information because everything is computed from the observation. l.148: Why is |G| finite? I think you didn't assume the finiteness of anything, including the action space.

Thank you for your correction. Indeed, these statements are imprecise. We are not limited to discrete spaces; in continuous spaces, the state space is infinite. We will correct this.

l.154-157: This claim needs stronger support to be that general. The experiment you show here is an illustrative example, too small to frame the takeaway so broadly.

We understand your concern regarding the strength of the statement about the extension of Q-values with subgoals. However, we believe that illustrative example prove the extension of Q-table works. Additionally, similar pattern has been used in the literature [1].

[1] Ashley Edwards, Himanshu Sahni, Rosanne Liu, Jane Hung, Ankit Jain, Rui Wang, Adrien Ecoffet, Thomas Miconi, Charles Isbell, and Jason Yosinski. Estimating q (s, s’) with deep deterministic dynamics gradients. In International Conference on Machine Learning, pages 2825–2835. PMLR, 2020

l.160: Why are there fewer subgoal tuples? I understand that both methods create exactly one datapoint for every step. If that's because of duplicates in the buffer, then this example may not be fully relevant (as in other environments, it is nearly impossible to repeat exactly the same state).

In this experiment, we adopt the Q-learning algorithm. These tuples represent the cells in the Q-table.

In the example shown in Figure 3, $(\mathcal{A} = \mathcal{A}^o = \{ \text{none}, \uparrow, \downarrow, \leftarrow, \rightarrow \})$ and $(\mathcal{G} = \{D_1, D_2\})$. In the Q-table, the number of entries for $(s, g, a)$ is $(10 \cdot |S|)$, which is fewer than the number of entries for $(s, a^o, a)$.

Due to exploration, it is practically impossible to update all cells in the Q-table. During my experiments, I observed that the number of cells used in the Q-table with $(s, a^o, a)$ was indeed fewer.

l.297: I'd also mention here which value of H you've chosen.

The hyperparameters are presented in Appendix B.In both Foraging and Predator-Prey, the subgoal horizon $H$ is 5. In SMAC, $H$ is also 10.

Appendix B lists the hyperparameters. The subgoal horizon $H$ is 5 for Foraging and Predator-Prey, and 10 for SMAC.

What is the speed of training and inference of OMG compared to Naive OM and standard RL baselines (wall time)?

The training and test times are shown in the table below. OMG requires additional training time to compute subgoals from the buffer, as described in Eq. 6 and Eq.7.

Thank you for acknowledging our novel contributions as well as raising valuable questions.

Improvements over scores in baselines are minor in Foraging. The main difference between the baselines appears to be in the reduction of the episode steps. It would be interesting to see if there is a reduction in the average time steps towards the end of training in Predator-Prey.

The Predator-Prey has a fixed episode length. The score represents the number of times the predators touch the prey. Therefore, this result already reflects a reduction in the average touch time.

L152 — missing word? “will reach the same ” → “will reach the same state”

Thank you for your correction. In this paper, goal $g$ represents a feature embedding of a future state. Therefore it is more accurate to use "will reach the same feature embedding" and we will fix it.

L159-160 — confusing sentence: “...resulting from the tuple (s, a^o, a) is more than numerous than (s,g,a) in the Q-table”

In the example shown in Figure 3, $(\mathcal{A} = \mathcal{A}^o = \{ \text{none}, \uparrow, \downarrow, \leftarrow, \rightarrow \})$ and $(\mathcal{G} = \{D_1, D_2\})$. In the Q-table, the number of cells for $(s, g, a)$ is $(10 \cdot |S|)$, which is fewer than the number of cells for $(s, a^o, a)$.

L190 — confusing preposition: “state as…” → “state of”?

This sentence implies that the selected state is being treated or used as a subgoal. Emphasizes the role or function of the state in the context of the model. We modify it in the revision to disambiguate.

L204 and L206-207 — both sentences here claim seemingly contradictory statements: “...adopting an optimistic strategy akin to the minimax strategy, which applies to cooperative games” seems to contradict the following statement “...leading to a conservative strategy similar to the minimax strategy, which is commonly used for general-sum games”. Are these both similar to minimax?

In Line 204, the maximax strategy is used, while Line 206 employs the minimax strategy. These two strategies are different.

Section 4.2: Subgoal Selector. This section is a bit hard to follow the reasoning for Eq. 6 and Eq. 7 for OMG-optimistic and OMG-conservative, even after referring to 4.1 Can you clarify in this section why you provide these two distinct manners for subgoal selection?

The maximax strategy aims to maximize the maximum possible gain. It is an optimistic approach that focuses on the best possible outcome. The formula is as follows:

$a = \arg\max_{a_i \in \mathcal{A}} \left(\max_{a_o \in A_o} (P(a_i, a_o)) \right)$.

The minimax strategy focuses on maximize the minimum gain. The formula is as follows:

$a = \arg\max_{a_i \in \mathcal{A}} \left(\min_{a_o \in A_o} (P(a_i, a_o)) \right)$

In Eq.6 corresponds to the inner "max" in the maximax strategy, and Eq.7 corresponds to the inner "min" in the minimax strategy. RL corresponds to the outer "max" of both.

Figure 7b — OMG-conserv shows step 1, 5, 6, and 8; OMG-optim shows steps 1, 5, 8, 10. Is there a reason the steps are not matched? (i.e both showing 1,5,6,8 or 1,5,8,10)

The trajectories formed by the two algorithms have different lengths and cannot be matched exactly. We will release the full trace instead of the key steps in the appendix.

L299-300 — the wording of the definition for hit ratio is a bit unclear. Is this essentially the ratio of predicted opponent trajectory to actual opponent trajectory?

Section 5.5: How do you determine the hit ratio if the subgoal is several steps ahead of the current state? For example, if a predicted subgoal is 3 steps north, the opponent could reach the subgoal in 3 steps or more steps. How many steps can an opponent take between the subgoal prediction state in order for it to still count as a hit?

For example, the opponent’s trajectory sequence from $t=0$ to $t=4$ is $(s_1, s_2, s_3, s_4, s_5)$. The agent's prediction from $t=0$ to $t=3$ is $(s_3, s_1, s_5, s_5)$. The hit ratio is calculated as $\frac{|\{s_3, s_5\}|}{|\{s_1, s_2, s_3, s_4, s_5\}|} = 0.4$. The predicted $s_1$ at $t=1$ is not counted because $s_1$ is not present in the trajectory sequence from $t>=1$. We'll add a detailed explanation in the appendix.

Section 5, Figure 5 — It is not immediately clear what the X-axis is representing. The numbers in front of “non-homologue” and “homologue” do not have context (they are only clear from reading the Appendix). Can you add a sentence to the description of the Figure? Can you expand on the sentence: “The X-axis represents the opponent’s policies, and “homologue” refers to the policy learned by the same algorithm, while “non-homologue” represents different ones”?

L294 — possible typo: “interrupted” → “interpreted”

L479 — Typo: “ovservation” → “observation”

L504 — Typo: “remained” → “remaining”

Thank you for your suggestions and corrections. We will fix it in the revision.

Part Ⅰ

Thank you for valuable comments. Below is a detailed response to your question, addressing each point individually.

I do not believe that goal and sub-goal inference for modelling agents is original. The paper is missing an entire line of research in their related works here, namely the inverse planning literature. This community has been studying inferring goals and sub-goals from trajectories through probabilistic modelling of other agents for a long time now (e.g. see [1,2] as a starting point and follow the thread of references and citations). In addition, inferring and modelling goals of others is not original also in the ad-hoc teamwork literature (see [3,4]). Others also considered goal recognition between agents actively [5]. These are only some of the papers in this line of work, as there are many more. I am surprised to see none of these works are cited. I would like to see a discussion on what is the novelty/originality left after this literature is accounted for. Conditioning the Q-function on the goal of other is also not an original idea, but this is not the main claim any way. Additionally, the problem of learning to cooperate with previously unseen teammates has been defined and named as "ad-hoc teamwork" in 2010 [6]. Since it is literally the problem setting of the authors, I am surprised there is no mention of this and the original paper is not cited.

We will add more relevant references to enhance the related works section. Below, we explain the differences between our work and the references you mentioned to highlight our contributions and originality.

Our method represents an innovation in the field of opponent modeling. As described, OMG is applicable to both cooperative games and general-sum games, and is not limited to "ad-hoc teamwork" [3,4,6]. OMG emphasizes methodological innovation rather than focusing solely on the ability to solve specific tasks.

Additionally, inferring goals is a straightforward concept and is commonly addressed in numerous papers [5, 7, 8, 9]. However, in the context of opponent modeling for autonomous agents, this method is novel. Unlike the method in [2], which requires agents to communicate with each other, our setting involves autonomous agents that cannot communicate with other agents. Model-based methods [1] require constructing an environmental model and involve extensive computation for planning during execution. This approach differs from the model-free method adopted in this work.

The experimental section only compares to two relatively simple opponent modelling works LIAM and "Naive OM". However, the authors for instance literally cite the "Machine Theory of Mind" paper and the "Modelling others using oneself in multi-agent reinforcement learning" in related works. Surely these are the closest competitors to their method. Additionally, some of the papers listed above under originality are also close competitors. I believe you need more than two methods in opponent modelling baselines. Especially the two methods compared are by no means considered the state-of-the-art.

OMG differs from traditional action prediction methods in opponent modeling by introducing a new approach based on subgoal inference. Therefore, we selected related methods as baselines. To the best of our knowledge, we have compared state-of-the-art (SOTA) methods in this domain. Below, we will explain why certain methods were not included as baselines.

[1] describes a planning algorithm using "Sequential Inverse Plan Search," which is fundamentally different from the model-free algorithm used in this paper. The only similarity is the concept of "goal inference." In [2], agents are able to communicate with each other, unlike our setting with autonomous agents that cannot communicate. References [3, 4, 6] focus on ad-hoc teamwork. The method in [3] does not use deep learning and is difficult to apply to complex environments like SMAC. [4] uses Goal-Conditioned RL (GCRL) and requires goals to be manually defined, whereas OMG does not. Additionally, goal-based reward shaping limits [4] to cooperative tasks. The problem addressed in [5] is active goal recognition, which is quite different from our approach of optimizing policies through opponent modeling. We have conducted a comprehensive survey of related work and compared all relevant baselines under fair conditions.

[1] Zhi-Xuan T, Mann J, Silver T, Tenenbaum J, Mansinghka V. Online bayesian goal inference for boundedly rational planning agents. Advances in neural information processing systems. 2020;33:19238-50

[2] Ying L, Zhi-Xuan T, Mansinghka V, Tenenbaum JB. Inferring the goals of communicating agents from actions and instructions. InProceedings of the AAAI Symposium Series 2023 (Vol. 2, No. 1, pp. 26-33).

[3] Melo FS, Sardinha A. Ad hoc teamwork by learning teammates’ task. Autonomous Agents and Multi-Agent Systems. 2016 Mar;30:175-219.

[4] Chen S, Andrejczuk E, Irissappane AA, Zhang J. ATSIS: achieving the ad hoc teamwork by sub-task inference and selection. InProceedings of the 28th International Joint Conference on Artificial Intelligence 2019 Aug 10 (pp. 172-179).

[5] Shvo M, McIlraith SA. Active goal recognition. In Proceedings of the AAAI Conference on Artificial Intelligence 2020 Apr 3 (Vol. 34, No. 06, pp. 9957-9966).

[6] Stone P, Kaminka G, Kraus S, Rosenschein J. Ad hoc autonomous agent teams: Collaboration without pre-coordination. AAAI 2010.

[7] Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. ICML, 2020.

[8] Suraj Nair, Silvio Savarese, and Chelsea Finn. Goal-aware prediction: Learning to model what matters. ICML, 2020.

[9] Menghui Zhu, Minghuan Liu, Jian Shen, Zhicheng Zhang, Sheng Chen, Weinan Zhang, Deheng Ye, Yong Yu, Qiang Fu, and Wei Yang. Mapgo: Modelassisted policy optimization for goal-oriented tasks. IJCAI, 2021.

Part Ⅱ

Q1: "However, focusing on the opponent’s action is shortsighted, which also constrains the adaptability to unknown opponents in complex tasks". What do you mean by "shortsighted"? How does that constrain the adaptability? These are strong statements but are not concretised at all.

Generally, modeling an opponent’s actions just predicts what it will do at the next step. A lot of opponent modeling papers[1,2,3] predict the next step action. Intuitively, it is more beneficial for the agent to make decisions if it knows the situation of the opponent several steps ahead. Predicting future states of the opponent have an advantage over predicting future actions. Just like our example in Sec 1 Para 3:

For example, to reach the goal point of $(2, 2)$, an opponent moves from $(0, 0)$ following the action sequence $<\uparrow,\uparrow,\rightarrow,\rightarrow>$ by four steps (Cartesian coordinates). There are also 5 other action sequences, i.e., $<\uparrow,\rightarrow,\uparrow,\rightarrow>, <\uparrow,\rightarrow,\rightarrow,\uparrow>, <\rightarrow,\uparrow,\uparrow,\rightarrow>, <\rightarrow,\uparrow,\rightarrow,\uparrow>, <\rightarrow,\rightarrow,\uparrow,\uparrow>$, that can lead to the same goal. Obviously, the complexity of the action sequence is much higher than the goal itself.

Similar conclusions are also verified by the experiments in Figure 3, and the corresponding analysis is given in Appendix A.1.

[1] Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:19210–19222, 2021.

[2] Georgios Papoudakis and Stefano V Albrecht. Variational Autoencoders for Opponent Modeling in Multi-Agent Systems. arXiv preprint arXiv:2001.10829, 2020.

[3] Haobo Fu, Ye Tian, Hongxiang Yu, Weiming Liu, Shuang Wu, Jiechao Xiong, Ying Wen, Kai Li, Junliang Xing, Qiang Fu, et al. Greedy when sure and conservative when uncertain about the opponents. In International Conference on Machine Learning, pages 6829–6848. PMLR, 2022.

Q2: "...bridge the information gap between agents". This concept of information gap appeared out of nowhere. It is not explained either. What does this even mean? It needs to be clarified.

In Multi-Agent Reinforcement Learning (MARL), "bridging the information gap between agents" refers to enhancing communication and information sharing among agents. This process involves reducing the uncertainty or lack of information each agent has about the environment, as well as the states, actions, or intentions of other agents. By effectively bridging this gap, agents can make more informed decisions, coordinate better, and ultimately achieve more optimal outcomes in their collective tasks. The following literature [1] explores the importance and impact of information sharing among agents.

[1] Tan, M. . "Multi-Agent Reinforcement Learning : Independent vs. Cooperative Agents." Proc. of 10th ICML (1993).

Q3: "Autonomous agents, different from those jointly trained, can act autonomously in complex and dynamic environments...". You seem to be conflating autonomy with decentralized training or being self-interested here. The definition of autonomy does not preclude centralised training. I can have a set of autonomous agents train with privileged information, yet deploy them to act autonomously.

We will reorganize the discussion on "Autonomous agents and jointly trained" to eliminate any ambiguity. The aim here is to distinguish our method from those that are "jointly trained," as such methods struggle to generalize against opponents with varying policies, as results in Appendix C.

Q4: "Although a lot of the existing methods concentrate on modeling the opponent’s actions, we argue that such an approach is short-sighted, pedantical, and highly complex.". Again, lots of sweeping and strong statements with no backing or clarification. What does short-sighted, pedantical, and highly complex mean? This does not sound professional or scientific. If you are going to actually complain about an entire line of literature, you need to make your statement more concrete.

These words "short-sighted, pedantical, and highly complex" are used to describe the weakness of "predicting action" in opponent modeling, which is also the advantage of our proposed "predicting subgoals". For example, A and B play the prophecy game, A can predict the next step of other agents, and B can predict the next 5 steps. Obviously, A is short-sighted compared with B.

We give intuitive examples that illustrate the advantages of predicting subgoals. That example is detailed in Sec 1 Para 3. Similar conclusions are also verified by the experiments in Figure 3, and the corresponding analysis is given in Appendix A.1.

Part Ⅲ

Q5: "Generally, modeling an opponent’s actions just predicts what it will do at the next step. Intuitively, it is more beneficial for the agent to make decisions if it knows the situation of the opponent several steps ahead". This statement seems to ignore the trajectory prediction literature entirely, and tries to get away with it by saying "generally".

To the best of our knowledge, current literature on plan recognition [1, 2, 3], including the trajectory prediction methods you mentioned, requires substantial prior knowledge, such as hierarchical plan libraries or domain models. These methods, which depend on prior knowledge, are outside the scope of our discussion.

[1] Blaylock, N., Allen, J., 2006. Fast hierarchical goal schema recognition. In: Proceedings of the 21st AAAI National Conference on Artificial Intelligence. pp. 796–801.

[2] Sohrabi, S., Riabov, A., Udrea, O., 2016. Plan recognition as planning revisited. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence. pp. 3258–3264.

[3] Vered, M., Kaminka, G., 2017. Heuristic online goal recognition in continuous domains. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. pp. 4447–4454.

[4] Tian, X., Zhuo, H., Kambhampati, S., 2016. Discovering underlying plans based on distributed representations of actions. In: Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems. pp. 1135–1143.

Q6: "Other methods that claim to predict the opponent’s goal [28 , 29], but without explicitly making a connection to the opponent’s goal...". What does this even mean? Also, see the inverse planning / goal recognition works cited in Weaknesses and their references/citations for works that predict goals explicitly.

In the domain of inverse planning, goals are indeed explicitly predicted. We want to emphasize that this paper discusses methods for predicting an opponent's goals within the context of opponent modeling. While both inverse planning and opponent modeling involve understanding other agents' behaviors, inverse planning focuses on inferring goals and intentions, whereas opponent modeling is concerned with predicting opponent behavior and gaining an advantage in the scenario.

Q7: "Unlike these methods, in this paper, we consider the most common...". I would argue that the setting where other autonomous agents are also learning and adapting to our agent is the more common setting, and the fixed opponent policy is a big simplification of it. This can be justified in certain cases, but it certainly is not the most common, except that it is more common in literature because it is simpler. But if the harder and more realistic problem is already being tackled, I do not see why this point is a plus.

Thank you for your suggestion. We acknowledge that the term "most common" might be ambiguous, and we will revise it to avoid confusion. In reality, due to the high costs of continuous learning, it is rare for deployed agents to continually learning and adapting. Instead, periodic updates are more common. We believe that the setting where opponents have unseen, diverse, but fixed policies during testing is more prevalent at present.

Q8: The $\pi^o, a^o$ notation is confusing. I would recommend using the pre-established notational norms in game theory / MARL, $\pi^{-i}, a^{-i}$ for all agents except $i$.

Thank you for your suggestion. We will revise it in the revision.

Q9: "Opponent modeling typically predicts the actions of other agents to address the non-stationary problem." At this point, I am a little confused. You have said "Unlike these methods, in this paper, we consider the most common setting where opponent s have unseen, diverse, but fixed policies during test." So the opponents have fixed policies during test. Then there is absolutely no non-stationarity during test time. In fact, since is fixed within a test episode, the problem the agent is facing in test time is in fact an MDP. So you are only dealing with the setting where at each episode the agent might be facing a new MDP. How is this different from online single-agent reinforcement learning then?

Firstly, the sources of non-stationarity differ. In single-agent RL, non-stationarity arises from changes in the environment. In MARL, an additional challenge is the uncertainty from opponents' policies. These different sources lead to varying degrees and forms of non-stationarity, which should not be conflated. Secondly, online reinforcement learning is a broader concept. In our setting, the agent must quickly adapt to opponents' policy changes, which aligns with the goal of online RL to enable agents to adapt rapidly.

Thank you for acknowledging our novel contributions as well as raising valuable questions.

''The subgoal prior model, denoted as $p_\psi$, is a pre-trained variational autoencoder (VAE) using the states previously collected in the environment'': What policy was used to collect these states to pre-train the prior model? This policy will have a huge impact on the coverage of the state space, and hence the subgoal inference model and final policy of OMG.

Training OMG requires a set of policies as the training set. During the preparation of the training set, states are collected to train the prior model. The policies used in the training set are described in Appendix B, lines 494-499. As you noted, $p_\psi$, serving as an encoder, must cover the state space as thoroughly as possible. However, it does not limit the policies used to collect states, allowing for an expanded range of state collection in practice.

I have concerns about the generalizability and scalability of this approach. The authors test their approach on SMAC as a complex domain with one easy map (8m) and one medium map (3s_vs_5z). A number of SMAC maps have been shown to be solvable by independent learning methods such as IQL [1] and IPPO [2]. Can the authors showcase performance on hard SMAC maps such as 6h_vs_8z, corridor, 27m_vs_30m which requires learning more complex strategies and have a higher number of agents?

We tested on 6h_vs_8z, and both QMIX and our method (OMG-optim + 5 QMIX agent) had a win rate of 0%.

The ability to predict an opponent's actions or subgoals is not the only factor that determines performance, especially in scenarios requiring close cooperation. We are testing on 27m_vs_30m and will provide the results later. This may be challenging due to the persistent scalability issues in opponent modeling.

3a. Why do the authors use different baselines for each environment i.e. D3QN, PPO, and IQL for Foraging, Predator-Prey, and SMAC respectively? Each of these baselines have been used for these environments. [4]

The core of OMG is opponent modeling method, which does not restrict the type of underlying RL method. Therefore, we followed the original environment's setup.

3b. What variant of PPO was used for the Predator-Prey environment? Both IPPO [2] with individual agent critics and MAPPO [3] with centralised critics have shown to exhibit strong performance in a lot of cooperative benchmarks including predator-prey.

We used IPPO. The centralized critics are not applicable to the setup of this study. Because, the motivation of this paper is to address the autonomous agent through opponent modeling, rather than training a group of agents for some task.

3c. It would be helpful to assess the impact of the approach if the authors can compare their approach to state-of-the-art CTDE techniques in MARL i.e. QMIX.

We has conducted tests where QMIX acts as the agent with opponents of test set in Appendix C.

The training paradigm employed by QMIX leads to a lack of generalization for different opponents. Using the same training methodology for QMIX as OMG leads to a degradation in IQL, which already serves as one of the existing baselines. The test results indicate that opponents trained using different methods and seeds are not homogeneous, which poses challenges for cooperation.

Could the authors provide more insight into how the subgoal predictor's performance might degrade or improve when scaling to environments with significantly more complex or numerous agents?

When modeling opponents, the difficulty of prediction usually escalates in environments with more complex or numerous agents, as the prediction space expands. From a scalability standpoint, OMG offers certain advantages over traditional action prediction methods. For example, if the state space of the environment excludes agent information, OMG's prediction space does not significantly increase with the number of agents.

Are there specific types of multi-agent environments where OMG might not perform as expected? What limitations in the model's assumptions should be considered?

We believe it is essential for opponents in the environment to have clear goals. An opponent with a purposeless, random policy is challenging to model, even though it might not directly impact the agent's rewards.

How does the model handle highly dynamic environments where the opponent's strategies evolve more rapidly than the model's training updates?

OMG does not account for scenarios where opponents rapidly evolve during interactions. Addressing such situations requires incorporating planning ideas as discussed in [1, 2], which is left for future work. Currently, OMG addresses this by ensuring that the training set includes a diverse range of opponent policies, allowing it to generalize well even when opponents rapidly evolve during testing.

[1] Zhi-Xuan T, Mann J, Silver T, Tenenbaum J, Mansinghka V. Online bayesian goal inference for boundedly rational planning agents. Advances in neural information processing systems. 2020;33:19238-50

[2] Xiaopeng Yu, Jiechuan Jiang, Wanpeng Zhang, Haobin Jiang, and Zongqing Lu. Model-based opponent modeling. Advances in Neural Information Processing Systems, 35:28208–28221, 2022.

Thank you for your rebuttal. I appreciate the additional clarifications provided and have decided to update my score, provided the additional experiments are added to the final version of the paper.