We thank the reviewer for their careful reading and valuable feedback. We are glad the reviewer found the paper “clear and well-organized” and appreciated its ”thorough theoretical analysis”.

Below, we address the reviewer’s concerns in a detailed way.

Extension to n-player setting

While our QRE-based analysis is presented in the two-player setting for clarity, the framework can be naturally extended to the general $n$-player case. Specifically:

• In Theorem 3.8 we would then have the sum of the total variation of the variations of the other players' policies.
• In Theorem 3.9, as stated below the theorem, we would get an exponential dependency in the number of agents $\mathcal{A}^n$.

We will clearly state in the introduction that we focus on the two-player case for readability, but the framework extends to $n$ players.

Average reward assumption

• The assumption that two distinct opponent policies yield the same average reward for a player can hold in various realistic settings. One example is a symmetric zero-sum game where the agent’s reward depends only on the final outcome (e.g., win/loss).

• This assumption is also consistent with identifiability conditions in the single-agent inverse RL literature (e.g., [1]), where one needs to observe the same reward function across two MDPs.

• Compared to the single-agent case, this assumption can be less restrictive as in the single-agent case, it is required to observe two different MDPs with different transition functions as well. On the other hand, in the multi-agent case, a different transition function can be induced by a different expert in the same Markov Game. Coming back to the example, one can consider a Tic-Tac-Toe game where one faces two different winning strategies; then the average reward function is the same, and the transition dynamics can be altered for the same Tic-Tac-Toe game.

Notations in the appendix

We thank the reviewer for noting this and agree with the reviewer that all the notations should be included in the main paper instead of the appendix. As the final version allows an additional page, we include the necessary notation in the main section of the camera-ready version.

Once again, we would like to thank the reviewer for their valuable feedback that helps us to improve the paper, and we are happy to answer any further questions that the reviewer might have.

[1] Rolland, Paul, et al. "Identifiability and generalizability from multiple experts in inverse reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 550-564.

We thank the reviewer for their thoughtful and encouraging feedback. We are glad that the reviewer found the paper ” is clearly written” and ”rigorous”.

Below we address the reviewer’s questions and suggestions in detail.

Introduction of QRE

We appreciate the reviewer’s suggestion to introduce QRE earlier in the paper. Our current presentation begins with NE because it is the most commonly studied solution concept in game-theoretic analysis. This allows us to clearly highlight the limitations of single-equilibrium inference. That said, we agree that introducing QRE earlier would provide a more consistent conceptual framework throughout. In the camera-ready version, we make the following adjustments:

• Introduce QRE in the Preliminaries (in the Entropy Regularized Game section).
• Clarify the transition from NE to QRE in Section 3.2, emphasizing the practical and analytical motivations for using QRE.

Average reward

We introduced the average-reward setting (Equation 2) in the context of our second research question, on when rewards are identifiable in multi-agent games. This was not necessary before, as we were mainly considering the feasible reward set, meaning all reward functions that are feasible under an observed expert equilibrium. To clarify this, we are happy to explicitly state this transition when Equation 2 is introduced.

Once again, we thank the reviewer for their constructive and positive feedback, helpful suggestions, and we hope that we have clarified the reviewer's concerns.

We thank the reviewer for their time to read our paper, providing this kind and insightful review that helps us to improve it. In particular, we are grateful that the reviewer acknowledges that our paper makes a “strong and methodologically sound contribution.”

Below, we would like to address the concerns raised by the reviewer and answer the questions.

Experiment on Identifiability

We thank the reviewer for the excellent suggestion to include an experiment validating our claims on identifiability. We will add the following to the appendix.

Experimental Setup: We test identifiability by observing the QRE policies from two environments with different, randomly generated transition dynamics satisfying the identifiability. The Markov Game has 15 states, and both agents have 5 actions for both setups. We only consider the results for the reward of the first player; it follows analogously for the second one.

• Averagel Case: We first use a ground-truth reward $R\_1​(s,a\_1​,a\_2​)$. The algorithm perfectly identifies the average reward (max difference: 0.000000). However, the full reward is not identifiable, achieving a correlation of only 0.9097 with the ground truth.
• Linearly Separable Case: We repeat the experiment with a linearly separable reward. Then, the algorithm perfectly recovers the full reward component R_1​ (max difference: 0.000000), meaning the full reward is successfully identified. We summarize the results of the correlation in the following table

Minor issues

We thank the reviewer for pointing out these minor issues, and we have addressed them separately to improve the quality of our paper:

• Fixed typo, changed equilirbium to equilibrium
• To clarify the dependence on $R$, we addressed the dependency directly in Definition 3.2 and changed it to $\mathcal{E}(\hat{\boldsymbol{\pi}};R) := \max\_{i \in [n]} \max\_{\pi^i \in \Pi^i} V\_{\mathcal{G}\cup R}^{i}(\pi^i,\hat{\pi}^{-i}) - V\_{\mathcal{G}\cup R}^{i}(\boldsymbol{\hat{\pi}}).$ Then, we can include $\mathcal{E}(\hat{\boldsymbol{\pi}};R)$ also in Definition 3.3 to emphasize the dependence.
• We thank the reviewer for pointing out the missing citations. We have already cited [1] in our related work section (line 984). Regarding [2] we are happy to include the concurrent work(it was released after the NeurIPS submission deadline), in our related work section of the camera-ready version. In particular, we note that [2] introduces identifiability in the context of linear payoff functions, where the essential goal is to estimate the parameter $\theta$ if the experts are playing a QRE. Next, the authors extend their results to Zero-Sum Markov Games with a linear reward function. In contrast, we consider the general-sum Markov Game case, and additionally, we provide insights on the ambiguity of the feasible reward set.

Extension to n-player case

• We would like to affirm this question from the reviewer. The results can easily be generalized to the n-person case: - If we denote the policy of the other player as $\pi\_{-i}$, all policies but the policy of agent i, we can simply replace $\nu$ by this joint policy. - In Theorem 3.8 we would then have the sum of the total variations of the other players' policies. - In Theorem 3.9, as stated below the theorem, we would get an exponential dependency in the number of agents $\mathcal{A}^n$. We will add these comments as a footnote in the camera-ready version.

Lower bound

This is an interesting question raised by the reviewer, and it remains an open question for future works. Constructing a lower bound is, in general, not easy and as we have shown in this work, MAIRL is hard, and therefore it is unclear if techniques used in the single-agent case [3] also apply to the multi-agent setting. Next, we lay out our conjectures:

• In the single-agent case, it was shown that Inverse RL suffers from the same lower bound as RL, except for an additional $(1-\gamma)$ factor, when given access to a generative model [3]. This could indicate that, also in Multi-agent Inverse RL, the lower bound will be of the same order as learning equilibria in the first place. This would mean that a lower bound would also suffer from the curse of multi-agents.
• In [5] the authors provide a lower bound for Zero-Sum Games for the reward agnostic case, meaning that the learner has no access to the reward function during learning. The lower bound here scales with $\vert \mathcal{A} \vert \vert \mathcal{B}\vert$, which is worse than the reward-aware case in zero-sum games, scaling linearly in the number of players.

To summarize, we conjecture that Multi-Agent Inverse RL will probably suffer from the curse of multi-agents, meaning a bound that will scale exponentially in the number of players. We will add a comment about this future direction in the appendix.

Knowledge of entropy coefficient

We thank the reviewer for this relevant question. We answer it below:

• The knowledge of the entropy coefficient is also required in the single-agent case [4].
• To check the identifiability conditions, knowledge of $\lambda$ is not required.
• If the goal is to recover a specific reward function, it is necessary to know the coefficient, as this scales the recovered reward. Therefore, a mismatch in this coefficient would also result in a mismatch of this difference for the recovered reward and true identifiablity would not be possible.
• Regarding the feasibility, one can see this as input given by the expert, as the entropy coefficient reflects the bounded rationality of the agent in a multi-agent setting. This means how much the expert believes in the rationality of the other agents.

We are happy to include a discussion regarding this in the camera-ready version of the paper.

[3] Metelli, Alberto Maria, Filippo Lazzati, and Marcello Restelli. "Towards theoretical understanding of inverse reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

[4] Rolland, Paul, et al. "Identifiability and generalizability from multiple experts in inverse reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 550-564.

[5] Zhang, Kaiqing, et al. "Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity." Advances in Neural Information Processing Systems 33 (2020): 1166-1178.

We thank the reviewer for their detailed and constructive feedback. We are pleased that the reviewer found the paper ”the paper is well written and pleasant to read” and ”makes a valuable contribution by strengthening the foundations of MAIRL and will be a relevant read for anyone entering the field.” Below, we address all comments and suggestions in detail.

Content-Sensitive Typos

• We corrected the definition of the Hausdorff metric to properly reflect the intended subscript and structure:

\mathcal{H}\_d(\mathcal{Y}, \mathcal{Y}') := \max\left\\{ \sup\_{y \in \mathcal{Y}} \inf\_{y' \in \mathcal{Y}'} d(y,y'), \sup\_{y' \in \mathcal{Y}'} \inf\_{y \in \mathcal{Y}} d(y,y')\right\\}

• Changed action $a_2$ to action $b$.
• We changed the subscript in the action space to a super script $\mathcal{A}^i$ to be consistent.
• Added the missing condition $s \sim \rho$ in the expectation on Line 110.
• We changed the alignment in line 192: $V^{i, (\pi^{\mathrm{Nash}}\_i, \pi\_{-i}^{\mathrm{Nash}}) }\_{\mathcal{G} \cup R}(s)$.
• We changed the reference in line 362 to Appendix G.

Clarity

• We added time indices where appropriate.
• Adopted a convention where random variables are capitalized and their realizations use lowercase (e.g., $S_t$, $s_t$).
• In the paragraph on entropy-regularized Markov games, we now clarify that the exposition focuses on the two-player setting, but the framework generalizes to $n$-player games.
• While we defined $\mathcal{A}^2 := \mathcal{B}$ on Line 109, we acknowledge the potential confusion and now consistently use one notation throughout the paper for clarity.

Figure 2

We move the illustration of the Markov Game to section E.1 and refer in the main text to this section for a detailed description of the setup.

Consistency and Terminology

• To align with common usage in the literature, we now consistently use Markov game
• We standardized capitalization across section and paragraph titles.
• We ensured consistent usage of acronyms and their expanded forms throughout the paper.

Total Variation

Yes, the reviewer is correct. We now introduce this explicitly in the Preliminaries (Mathematical background).

Limitations and future work section

As the camera-ready version allows us to include an additional page, we are happy to extend section 5 by including the future work and limitation setting which used to be in Appendix C.

Typos

We thank the reviewer for pointing out these typos, we have fixed them in the current version of the paper (equilibira -> equilibria, considers -> consider, agent’1 -> agent 1’s, showed -> “shown”).

Once again,we would like to thank the reviewer for their careful reading and valuable suggestions. We believe these revisions improve the clarity and quality of the paper, and we hope it now better reflects the significance of the contributions.