Thank you for your detailed feedback and questions. First of all, we would like to emphasize again that, in addition to the convergence results for regularized MFG in $O(\log (1/\varepsilon))$ you mentioned (Theorem 4.3), our contributions also include the convergence of the proximal point (Algorithm 1, Theorem 3.1) and the proposal of a tractable algorithm (Algorithm 2).

We would like to address your concerns and elaborate on the technical contributions of our work.

Our result on the exponential convergence of regularized mirror descent is not merely an application of the techniques developed in the literature, such as [1]. As you correctly point out, Zhang et al.derive the relationship between $D_{\mu^\ast}(\pi^\ast, \pi^{t+1})$ and $D_{\mu^\ast}(\pi^\ast, \pi^t)$ with some discretization error. Despite this, we would like to emphasize that their discretization error is completely different from ours in Equation (4.6).

Furthermore, obtaining the $(1 - \lambda \eta)$ factor from this relationship is non-trivial. The reason lies in the complexity of the Q function $Q^{\lambda,\sigma}_h(s,a,\pi,\mu)$ in MFG, where $\mu$ is given by $\mu = m[\pi]$, making it more complex with respect to $\pi$. To explain in more detail by comparing the zero-sum Markov game focused in [1]: In zero-sum Markov games, the value of the Q function at time $h$ is determined by the policy from future times $h+1$ to $H$. In contrast, in MFG, the Q function at time $h$ also depends on the state distribution $\mu_h$, where $\mu_h = m[\pi]_h$ is determined by the policy from past times $1$ to $h-1$. As a result, the Q function depends highly non-linearly on the policy over the entire horizon, complicating the analysis of discretization errors.

To address this difficulty, we made two key contributions: - We derived a different discretization error $D_{\mu^\ast}(\pi^t, \pi^{t+1})$ from Zhang et al., as shown in Equation (4.6). This refined expression of the error facilitates clearer subsequent analysis. - We discovered that our discretization error can be bounded by $D_{\mu^\ast}(\pi^\ast, \pi^t)$. The key insight here is the claim stated in l.369-372: “over the sequence $(\pi^t)_t$, the value function $Q_h^{\lambda,\sigma}$ behaves well, almost as if it were a Lipschitz continuous function.”

We hope this clarifies the technical contributions of our work and addresses your concerns. Thank you again for your valuable feedback.

Thank you for reviewing our paper. We will now address your concerns about our contribution compared to existing research.

• Summary: This paper studies the convergence performance in Monotone Mean Field Games. The major issue is that the focus of this paper are all well studied. Authors cannot identify the research gap and contribution in a comparison to existing study.
• Weaknesses: Position: Last-iterate convergence and learning regularized MFGs have been extensively studied.

We respectfully disagree with this argument. To the best of our knowledge, the only study that provides a last-iterate convergence guarantee is the one by Perolat et al. (2022). We would greatly appreciate it if you could provide specific references to other studies that have focused on the last-iterate convergence guarantee if you are aware of any.

• Weaknesses: However, the discussion of related work in this manuscript is somewhat superficial. The authors should conduct a thorough literature review and clearly position their work within the existing literature.

While it might have been beneficial to mention existing results on last-iterate convergence for games beyond MFGs as part of the related work, we believe that the literature we discussed in the manuscript comprehensively covers the existing research on "last-iterate convergence and learning regularized MFGs" to the best of our knowledge. In fact, even the result on time-averaged convergence (which is different from last-iterate convergence) of mirror descent for regularized MFGs was only recently established by Zhang et al. in NeurIPS 2023.

This will help avoid giving the audience the impression that their contribution is marginal, merely by applying a proximal point iteration to MFGs.

To the best of our knowledge, no previous research has applied proximal point (PP) iteration to MFGs. Applying PP to MFGs and performing the theoretical analysis is not just a straightforward application of known techniques from the literature. The dependence of the state distribution $\mu$ on the policy $\pi$ introduces unique challenges specific to MFGs, as discussed in l.180-182 and further elaborated in our responses.

• Exponential convergence of mirror descent: Using mirror descent to learn regularized mean field games (MFGs) is not a novel approach. To the best of my knowledge, previous studies have established a standard convergence rate of $T^{−1/2}$. In contrast, this work claims an exponential convergence rate of $e^{−T}$, which represents a significant improvement. The authors should clearly explain the specific challenges they addressed to achieve this enhanced convergence rate that previous work did not tackle.

First, we would like to differentiate our results from the prior work of [Zhang et al. 2023], again, to make it clear. Our improvement is not limited to the convergence rate alone. While existing research, such as Zhang et al., only demonstrated convergence for the time-averaged policy $\frac1T\sum_{t=1}^T\pi^t$, we have proven last-iterate convergence, which means the convergence of the actual policy $\pi^t$. Although we emphasize the novelty of our convergence rate, we do not claim novelty for the mirror descent method itself, as acknowledged in l.264, where we cite [Zhang, et al. 2023].

One of the specific challenges that prior research, including [Zhang et al. 2023], did not address is the non-Lipschitzness of the Q function. We have mentioned this difficulty in our manuscript, particularly in l.340 and l.347. To overcome this challenge and improve the convergence result, we established a novel claim in Claim 4.5 and l.370: “the value function $Q_h^{\lambda,\sigma}$ behaves well, almost as if it were a Lipschitz continuous function.”

• Additionally, providing numerical demonstrations of such exponential convergence is essential, given the magnitude of this improvement.

Regarding the numerical demonstration of exponential convergence, we agree that including such figures would be beneficial. We plan to add these illustrations to the paper during the discussion period.

• Presentation: the presentation of the paper is not yet suitable for scientific publication. Please avoid excessive use of colors, shapes, and unconventional layouts. Figure 1 lacks informative content. Additionally, some language and structural transitions are too casual and do not provide sufficient discussion to support the arguments presented convincingly.

Our intention was to enhance the visual clarity and engagement of the content. However, we understand the importance of maintaining a professional and conventional layout for scientific publications. We will revise the figures and layouts to ensure they adhere to presentation guidelines of ICLR.

• The numerical experiments are limited. More results regarding the algorithm performance in a comparison to baselines (OMD, FP, Multi-scale, double loop, single loop) for MFGs should be provided.

The reason for conducting experiments on this example is to experimentally verify whether our proposed algorithm (Algorithm 2) converges in a typical MFG, not to compare it with existing methods.

Thanks for authors' responses. I am not convinced by the significance of the last-iterate convergence proposed in this work. Please check following works and highlight the research gap and contributions. A comprehensive review (including Xie and 2021 Zhang 2023) needs to be conducted by authors.

1. Mao 2022. A mean-field game approach to cloud resource management with function approximation
2. Zaman 2023. Oracle-free reinforcement learning in mean-field games along a single sample path
3. Yardim 2023. Policy mirror ascent for efficient and independent learning in mean field games
4. Zeng 2024. A Single-Loop Finite-Time convergent policy optimization algorithm for mean field games
5. Huang 2024. On the statistical efficiency of mean-field reinforcement learning with general function approximation.
6. Angiuli 2022. Unified reinforcement Q-learning for mean field game and control problems
7. Angiuli 2023. Convergence of multi- scale reinforcement q-learning algorithms for mean field game and control problems

I will keep my original score at this stage, but I am open to discuss the significance of this work in a comparison to the missing papers I listed above.

Thank you for your detailed feedback and for highlighting the additional references. We appreciate the opportunity to clarify the significance of our work in comparison to the mentioned papers. Below is a table summarizing the convergence results of the referenced works and our contributions:

We have added this table to the revision as Table 1 in Appendix A.

Based on this comparison, we would like to emphasize the following points:

Significance of Our Convergence Results: Unlike many of the referenced works that require strong assumptions such as contraction to achieve convergence, our results demonstrate last-iterate convergence (LIC) without such stringent conditions. This highlights that our contributions fill a significant gap in the literature.

We hope this clarifies the significance of our contributions and addresses the concerns raised. We are open to further discussion and appreciate your constructive feedback.

Thank you for your review. We will address your comments below:

• Weaknesses: The first result holds under general assumptions but the algorithm cannot be implemented. The second result holds only for regularized MFGs, which can be quite different from the original (unregularized) MFG problem and there is not much discussion about this. As a consequence, the contributions seem relatively limited.

First of all, we would like to emphasize again that we use the regularized mirror descent of the second result to approximate the proximal point method in the first result for solving (original) MFG. Moreover, this paper is the first work to provide the last-iterate convergence guarantee for solving original MFGs under the monotonicity condition.

We acknowledge that there are differences between regularized MFGs and unregularized MFGs. In fact, in Algorithm 2, we use regularized MFGs merely as a subroutine to solve the original unregularized MFG problem. More importantly, our contribution includes the proposal of an algorithm that integrates mirror descent for regularized MFGs into a proximal point method for unregularized MFGs, whose convergence has been experimentally verified. This result extends our theoretical findings into a practical algorithm, demonstrating the applicability and effectiveness of our approach.

• Questions: Q1. In Assumption 2.1, please clarify why “It is reasonable to assume” it. For example, it is violated in many examples of the MFG literature in grid world models, where the agents cannot jump from a given state to any other given state (for any action). I think this would be true only if there is a very strong noise that can propel an agent to any other state in just one time step. It does not seem very realistic from the physical viewpoint. Please explain.

First, we guess that even the grid world model example you mentioned satisfies Assumption 2.1. In this case, we can exclude the states from which no transitions are possible from the state set $\mathcal{S}$. By doing so, the remaining states will satisfy the assumption.

Assumption 2.1 states that "for any state $s^\prime$, there exists a state-action pair $(s, a)$ such that $s^\prime$ is reachable." It does not require that "for any state $s^\prime$, it is reachable by any state-action pair $(s, a)$". This distinction is crucial, because it only requires the existence of at least one state-action pair that can reach a given state, not that all state-action pairs can reach the state.

• Q2. In Definition 4.2, are regularized equilibria unique? Otherwise, please explain how you can say “the regularized equilibrium” in Theorems 4.3 and 4.4.

Yes, the regularized equilibria are unique. This uniqueness is discussed in l.245 and Proposition B.1 of our manuscript.

To elaborate further: Suppose there are two different regularized equilibria $(\mu^\ast_1, \varpi^\ast_1)$ and $(\mu^\ast_2, \varpi^\ast_2)$. If we assume $\varpi^\ast_1 \neq \varpi^\ast_2$, the following contradiction arises: - From Lemma A.2, which uses monotonicity, we have $J^{\lambda,\sigma}(\mu^\ast_1, \varpi^\ast_1) + J^{\lambda,\sigma}(\mu^\ast_2, \varpi^\ast_2) \leq J^{\lambda,\sigma}(\mu^\ast_1, \varpi^\ast_2) + J^{\lambda,\sigma}(\mu^\ast_2, \varpi^\ast_1) .$ - Additionally, from Proposition B.1, we know that $J^{\lambda,\sigma}(\mu^\ast_1, \varpi^\ast_1) \gneq J^{\lambda,\sigma}(\mu^\ast_1, \varpi^\ast_2)$ and $J^{\lambda,\sigma}(\mu^\ast_2, \varpi^\ast_2) \gneq J^{\lambda,\sigma}(\mu^\ast_2, \varpi^\ast_1)$. Adding these two inequalities gives us $J^{\lambda,\sigma}(\mu^\ast_1, \varpi^\ast_1) + J^{\lambda,\sigma}(\mu^\ast_2, \varpi^\ast_2) \gneq J^{\lambda,\sigma}(\mu^\ast_1, \varpi^\ast_2) + J^{\lambda,\sigma}(\mu^\ast_2, \varpi^\ast_1) .$

Therefore, $\varpi^\ast_1 = \varpi^\ast_2$. Moreover, by the definition of regularized equilibria, $\mu^\ast_1 = m[\varpi^\ast_1] = m[\varpi^\ast_2] = \mu^\ast_2$. This contradicts the assumption that the two equilibria are different. Thus, the equilibrium is unique.

• Q3. Algorithm 2 and Theorem 4.3: it seems to rely very strongly on the RMD of Zhang et al. (2023). How does the proof of convergence of Theorem 4.3 differs from theirs?

Before clarifying how Zhang et al.'s arguments in their proof differ from ours, let us also emphasize how the results differ.

In contrast to us, Zhang et al. (2023)'s result does not imply last-iterate convergence. It would also be difficult to obtain a result of last-iterate convergence with rate $\mathcal{O}(T^{-1/2})$ using Zhang et al. (2023)'s argument.

In order to overcome this difficulty and achieve exponential convergence, we came up with a completely different strategy to that used in the proof of [Theorem 5.4, Zhang et al. 2023]. The key to the proof of Theorem 4.3 lies in the claim stated in l.369-372: “over the sequence $(\pi^t)_t$, the value function $Q_h^{\lambda,\sigma}$ behaves well, almost as if it were a Lipschitz continuous function.” In contrast, Zhang et al. do not utilize this Lipschitz continuity. Instead, they bound the difference in $Q$ by a (potentially very large) constant; see [Eq. (E.1), Zhang et al.]. By leveraging the Lipschitz-like behavior of $Q_h^{\lambda,\sigma}$ over the sequence of policies, our approach provides a more refined analysis, leading to the convergence results stated in Theorem 4.3.

• Q4. Theorems 4.3 and 4.4: Could you please confirm whether these results hold for any λ and σ, simply under Assumption 4.1? I just want to make sure there is no requirement to take λ large enough as e.g., in Cui & Koeppl (2021).

Yes, Theorems 4.3 and 4.4 hold simply under Assumption 4.1. Additionally, there is no need to take $\lambda$ to be large enough, as required in Cui & Koeppl (2021). We would like to note that Cui & Coeppl adopt different assumptions from ours. For example, they do not impose the monotonicity of reward.

• Q5. Theorems 4.3 and 4.4: Here it seems that the convergence only holds for the regularized MFG. How far is it from the true MFG equilibrium?

Roughly speaking, the distance between the two equilibria is $\mathcal{O}(\lambda)$. More precisely, for the regularized equilibrium $(\mu^\ast, \varpi^\ast)$ and the original equilibrium $(\mu^\star, \pi^\star)$, we have $J(\mu^\star, \pi^\star) - J(\mu^\ast, \varpi^\ast) = \mathcal{O}(\lambda)$.

• Q6. Could you please explain whether the example considered in the numerical part satisfies all the assumptions required for your convergence results? Also, is it not strictly monotone (so that it does not fall in the case covered by Perolat et al. (2022))?
• Q7. To really see how your method compares with existing one, please compare your algorithm with other baselines, such as the algorithms considered in Perolat et al. (2022) (in particular the mirror-descent-type one, which has last-iterate convergence, as you mentioned).

The example used in our experiments satisfies all the assumptions required for our convergence results. Specifically, it is a monotone example. The reason we conducted experiments on this example was to experimentally verify whether Algorithm 2, which is a novel combination of proximal point with tractable approximation, converges in a typical MFG setting rather than to compare it with existing methods.

Thank you for your additional feedback. We would like to address your points in detail:

Regarding the theoretical results feeling somewhat incremental:

• the theoretical results feel somewhat incremental from a technical stand point. E.g. the use of the regularized variant of MD to achieve fast convergence in monotone settings feel very similar to e.g. [1]. Clearly, there are more technicalities involved to adapt the results to this precise setting but arguably these technicalities do not feel conceptually/technically novel. On the somewhat positive side, this makes parsing the proof rather straightforward.

We respectfully disagree with the assertion that our theoretical results are incremental. The work you referenced, [1] by Perolat et al., discusses the convergence of a continuous-time algorithm (FoRL) for Sequential Imperfect Information Games. Our setting and their setting differ significantly in the following ways:

• Sequential Imperfect Information Game vs. Mean Field Game (MFG): While both frameworks define a probability distribution relative to a policy $\pi$, Perolat et al. focus on the reach probability $\rho^\pi$ over histories, whereas we focus on the distribution of states $\mu=m[\pi]$. The dependency on $\pi$ is fundamentally different: $\rho$ depends on $\pi$ in a linear-like manner, while our $\mu$ has a highly nonlinear dependency on $\pi$ as shown in equation (2.1). Addressing this nonlinearity necessitated the development of novel techniques that exploit the inductive structure of equation (2.1) with respect to time $h$.
• Continuous-time vs. Discrete-time Analysis: Perolat et al. primarily analyze the convergence of the continuous-time FoRL algorithm. In contrast, our work extends beyond continuous-time convergence (Theorem 4.4) to also address discrete-time convergence (Theorem 4.3). Generally, proving convergence in discrete-time is more challenging than in continuous-time. For example, the proof of Theorem 4.3 requires precise evaluation of discretization errors, which do not arise in continuous-time analysis; see equation (4.6) and Claim 4.5 for details.

Regarding equilibrium selection:

• the experiments could be expanded. For example, prior work that studied last-iterate convergence in (non-strict) monotone settings used the experiments to study the effects of equilibrium selection [2]. Although the possibility of diverse equilibria is advertised in the paper as an important element of monotone games, not present in strictly monotone games, the implications of which equilibria get selected is not addressed. I feel that this is a missed opportunity that would make the paper significantly stronger.
• Questions: Can you say anything about the effects of your algorithm to equilibrium selection? Either theoretically or experimentally?

Within the scope of our Theorem 3.1, we can guarantee convergence to the set of equilibria, where the distance to this set converges to zero. Experimentally, we can observe that our algorithm indeed selects specific equilibria, converging to one of them. We hypothesize that the equilibrium closest to the initial policy $\pi^0$ in terms of KL divergence is selected. This hypothesis and its implications for equilibrium selection represent a promising direction for future research.

We have revised the manuscript to make the above points more clear. The revised parts are highlighted in purple. The main revisions are as follows.

• In Appendix A, we have listed a wider range of related works. Specifically, we have also compared our paper with previous research on MFG algorithms that are not limited to mirror descent.
• In Figure 2, we have experimentally confirmed that Regularized Mirror Descent converges exponentially.