Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics

First of all, we would like to sincerely thank the reviewer for their detailed review, which highlights our contributions and originality ("first method to tackle non-stationary MFGs with continuous state and action spaces"), and for the constructive suggestions to improve our manuscript. As suggested, we have added a theoretical discussion to the manuscript concerning the convergence of DEDA-FP.

W1 (Theoretical guarantees): Please see the detailed answer to Q1 (Error analysis) below. We are happy to provide more details if needed.

W2 (Real world scenarios): We agree that real-world scenarios are the ultimate target and this is the very reason why we develop an algorithm for continuous space MFGs instead of discrete space (or LQ) as a the rest of the literature. The examples we solve are of comparable complexity to those presented in other well-established papers such as: [Guo et al., 2019, Subramanian and Mahajan, 2019, Elie et al., 2020, Fu et al., 2019, Cui and Koeppl, 2022, Angiuli et al., 2022] cited in our introduction, as well as other recent works on RL for MFGs, such as: [Zhou et al., ] (ICML'24) which studies a simple one-dimensional model, [Zhang et al., 2024] (ICLR'25) which studies only a finite-state MFGs with very few states, or [Ocello et al., 2025] (ICML'25) which contains a 2-dimensional example similar to our Example 3 except that they only consider a finite state version while we solve a continuous space model (which is muc more realistic for crowd motion modeling).

Q1 (Error analysis): Thank you for raising this interesting point. Although a complete analysis is beyond the scope of our paper, we explain below that there is a clear path to analyze the different sources of errors and how they compound to yield an approximate Nash equilibrium. (We will include the following discussion in the revised paper.)

On the convergence of DEDA-FP:

While a full theoretical analysis is left for future work, we can provide strong arguments for the convergence of DEDA-FP, based upon existing theory.

At each iteration, there are three sources of errors: (e1) best response computation (using DRL), (e2) average policy computation (supervised learning), and (e3) mean-field approximation (with CNF). One can expect convergence towards an approximate mean-field Nash equilibrium, i.e., an $\epsilon-$Nash equilibrium, where $\epsilon$ depends on the errors. A convergence proof can be carried out by analyzing how the errors propagate through iterations.

While most of the literature on FP for MFG focuses on perfect computations, the work by [Elie et al., 2020] have carried out such an error propagation for FP when there is only the first type of error, namely, on the BR computation. We argue that a similar proof technique can be used to track the impact of other errors on the final approximate equilibrium. In this direction, we now outline why these three types of errors can be made arbitrarily small:

• (e1) Best Response Error: This error comes from the DRL algorithm. We expect this to be the most serious source of errors because solving an MDP is more difficult than a supervised learning task or a distribution-approximation task. Fortunately, this is precisely the type of error analyzed in [Elie et al. 2020], which have shown that it does not prevent convergence to an approximate equilibrium. Furthermore, to make sure that the inner loop of DRL computes a very good policy, we have carried out sweeps over hyperparameters (see Fig. 14 in Appendix for an example). This guarantees that our choice of hyperparameters yields a small error for (e1).

• (e2) Average Policy Error: This is a (relatively standard) supervised learning task. Thanks to the universal approximation properties of neural networks, the error in representing the true average of past policies can be made arbitrarily small provided the neural network is sufficiently deep/large, the training set is sufficiently large, and there are sufficiently many training steps.

• (e3) Distribution Error: The specific architecture we use, Neural Spline Flows, has been shown to be a universal distribution approximator [Durkan, 2019]. This guarantees that the error (e3) can be made arbitrarily small if the model is expressive enough.

Q2 (average policy in time dependent case):

Thank you for raising this interesting question. We address the two points:

1. Yes, it is sound to use an average policy in time-varying MFGs, as has been shown in [Cardaliaguet and Hadikhanloo, 2017] and [Perrin et al., 2020], which both analyzed fictitious play in finite-horizon MFGs. However, there is a twist, which is that the weights depend on the population distribution, as shown in the formula at the bottom of page 3 in [Perrin et al., 2020]. This is the proper definition of average policy in time-dependent MFGs.

2. You are right that we do not know any theoretical foundations for NFSP in time-dependent games. However, we are not using exactly NFSP. What we really use is the idea of supervised learning for the average policy, and this part is sound because we make sure that the policy learns the time dependence on top of the state-dependence.

It is important to notice that the way we build the replay buffer of state-action pairs ensures two key properties: (1) we store an equal number of samples at each iteration, which implies that sampling uniformly from the replay buffer amount to sample uniformly from the past iterations; and (2) the samples are generated by using the current best response, which implies that the probability to see a given state-action pair is directly linked to the mean field generated by the best response (this is what is required in the aforementioned formula of [Perrin et al., 2020]).

We hope that this addresses the reviewers' questions about the theoretical foundation of our DEDA-FP algorithm and that the reviewer will consider raising their score. Otherwise, we are happy to discuss these points further.

References:

[Ocello et al., 2025] Ocello, Antonio, et al. "Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean Field Games." Forty-second International Conference on Machine Learning.

[Zhou et al., 2024] Zhou, Fuzhong, et al. "Graphon Mean Field Games with a Representative Player: Analysis and Learning Algorithm." International Conference on Machine Learning. PMLR, 2024.

[Elie et al., 2020] Elie, Romuald, et al. "On the convergence of model free learning in mean field games." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 05. 2020.

[Durkan, 2019] Durkan, Conor, et al. "Neural spline flows." Advances in neural information processing systems 32 (2019).

[Zhang et al., 2024] Zhang, Fengzhuo, et al. "Learning regularized graphon mean-field games with unknown graphons." Journal of Machine Learning Research 25.372 (2024): 1-95.

[Cardaliaguet and Hadikhanloo, 2017] Cardaliaguet, Pierre, and Saeed Hadikhanloo. "Learning in mean field games: the fictitious play." ESAIM: Control, Optimisation and Calculus of Variations 23.2 (2017): 569-591.

[Perrin et al., 2020] Perrin, Sarah, et al. "Fictitious play for mean field games: Continuous time analysis and applications." Advances in neural information processing systems 33 (2020): 13199-13213.

We sincerely thank the reviewer for the constructive suggestions to improve our paper. We are grateful for their acknowledgment that "the advantages of the generality of DEDA-FP... compared to prior works is evident", "The exposition in section 3 nicely builds up to DEDA-FP", and that "the numerical experiments are very clearly described and well presented visually". We provide detailed responses to each of your questions below.

Weakness (Lack of theoretical guaranteed): We thank the reviewer for their interesting question. We address this point in Q1 (Error analysis) of Reviewer ZLz3, where we have provided a detailed discussion on the convergence of DEDA-FP. We would be glad to engage in further discussion on this topic..

Q1 (Drawbacks compared to previous works): Thank you for this question. We will stress the limitations of our work in the revised paper.

1. The primary drawback of our approach does not relate to the scope of problems it can handle, but rather to the slow convergence of the underlying Fictitious Play (FP) algorithm in some cases. While other methods such as those based on entropy-regularized fixed-point iterative methods [Cui et al., 2021] or Online Mirror Descent (OMD) [Perolat et al., 2022] may converge faster, they have so far only been demonstrated in less complex, finite state-space settings and present their own implementation and convergence challenges. We chose FP because it offers a robust and stable framework for tackling the main challenges we set out to solve continuous spaces, non-stationary dynamics, and learning the mean-field density, prioritizing stability over convergence speed.
2. A second limitation is that DEDA-FP requires more training time per FP iteration compared to the simpler Algo 1 and Algo 2. Specifically, training the Conditional Normalizing Flow (CNF) to accurately model the time-dependent mean-field distribution adds computational overhead to each iteration. While this leads to a richer and more accurate solution, especially in problems with local density dependence, it comes at the cost of increased single-iteration training time.

Q2 (Discrete action space): Yes, DEDA-FP can be applied to discrete spaces. This is because we can embed discrete spaces in continuous ones and then use our algorith. However this naive implementation would be highly inefficient. A tailored implementation would be far more effective than a naive one.

Indeed, a naive application would require embedding the discrete space into a continuous one and projecting the outputs, which is inefficient. A proper, adapted implementation would be more powerful and would involve two key changes:

1. The policy network would use a softmax output layer instead of modeling a Gaussian distribution to directly handle a probability distribution over discrete actions.
2. The Conditional Normalizing Flow (CNF), designed for continuous densities, would be replaced by a model suited for distributions on discrete state spaces. Here again, one could use a neural network with a sofmax on the last layer.

Q3 (Components of the algo): Thank you for giving us the opportunity to justify our choices. Both the overall orchestration and the individual components are chosen for a specific purpose. While other choices could be made, we explain below the rationale behind our choices.

• Fictitious Play: this is the backbone of our method. The main advantage is that it is known to converge in larges classes of games. One drawback is that convergence can be lower than some other methods, but we prefered to sacrifice the convergence speed and ensure robustness rather than the opposite.

• DRL for Best Response: Policies are functions defined on the continuous state and action spaces so they are infinite dimentsional. Hence we had to approximate them using parameterized functions. We chose neural networks due to the empirical success in a variety of machine learning tasks. As for the training, model-free RL has the advantage to avoid exact dynamic programming and hence scale well to highly complex problems. In the implementation we chose SAC and PPO but other choices could be made, depending on the specific MFG at hand.

• Supervised learning for average policy: In general, convergence results for Fictitious Play are not for the last iterate policy (the best response computed in the last iteration) but only for the average policy. So computing the average policy is crucial to ensure convergence. However, our policies are neural networks and averaging neural networks is hard due to non-linearities. We thus have to train a new neural network for the average policy. Here, we chose to use supervised learning, drawing inspiration from Neural Fictitious Self-Play [Heinrich et al., 2016]. To the best of our knowledge, this is a state-of-the-art method but we are open to suggestions.

• Conditional Normalizing Flow (CNF) for the Mean-Field: This is a critical design choice that directly enables one of our paper's main contributions. To solve MFGs with local density dependence (e.g., congestion), we require a model that can both (1) sample from the population distribution and (2) compute its exact probability density at any given point in time and space ($p(x|t)$). Normalizing Flows (NFs) are well-established generative models that have been introduced precisely to provide both of these capabilities without time and CNFs are an extension of NFs which allow us to take time into account in a natural way. Other generative models, like GANs or score-based diffusion models, can sample effectively but do not allow direct density evaluation, making them unsuitable for our goal.

We will include a discussion along these lines in the main text if space permits, or in the appendix of the final paper. We hope these arguments, especially on the theoretical foundations of DEDA-FP, clarify our approach and that the reviewer will consider raising their score. Of course, we remain open to further discussion on these points.

References:

[Cui et al., 2021] Cui, Kai, and Heinz Koeppl. "Approximately solving mean field games via entropy-regularized deep reinforcement learning." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

[Perolat et al., 2021] Pérolat, Julien, et al. "Scaling Mean Field Games by Online Mirror Descent." Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022.

[Heinrich et al., 2016] Heinrich, Johannes, and David Silver. "Deep reinforcement learning from self-play in imperfect-information games." arXiv preprint arXiv:1603.01121 (2016).

I appreciate the thorough response of the authors, in particular the discussion on the convergence of DEDA-FP. I also appreciate the expanded rationale on the choices in the design of the algo. I feel more confident in my previous assessment and will maintain it as is. Thank you again.

We appreciate the reviewer's valuable feedback on our manuscript. We are grateful for the fact that the reviewer acknowledged our paper is "clear in its exposition", that our DEDA-FP shows "promising experimental results that the sample efficiency far outperforms prior methods". Below, we address the weaknesses and questions.

W1 (Lack of theory): Sorry for the misleading statements. We will rephrase them to clarify that our approach is heuristic and that the theoretical analysis is left for future work. We will also include high-level explanations in line with the answer provided to Q1 (Error analysis) of Reviewer ZLz3. Although this is not a full proof, we hope it will help the reader get some intuition about why the algorithm is expected to converge to an approximate Nash equilibrium, and how the different sources of error contribute to the final approximation.

W2 (Different types of time dependency): In the examples, non-stationarity arises from both the noise and, in particular, the evolution of the mean field distribution, $\mu_t$. For instance, in the Linear-Quadratic model (Sec. 4.2), the dynamics depend on the time-dependent first moment (mean) of the population distribution, $\bar\mu_t$. This time-dependence through the evolving mean field presents a significant difficulty compared to papers focusing solely on stationary MFGs. The Conditional Normalizing Flow (CNF) component is precisely what allows us to learn and represent this time-dependent population distribution. It would also be possible to consider models in which the transitions are intrinsically time-dependent. For instance, still in the LQ model, we could let the coefficients $(A,B, \bar A)$ depend on time. We expect this to present less difficulty than the fact that the mean-field depends on time. We remain open to exploring other forms of time-dependency and would be interested in any further thoughts the reviewer may have.

W3 (NE policy learning): Intuitively, the equilibrium is (approximately) learnt by the average policy $\bar\pi^K = \frac{1}{K}\sum_{k=1}^K \pi^*_k$ computed by $K$ iterations of Fictitious Play (FP). This has been proved in the finite-space case (see e.g. [Elie et al., 2020]) and for our algorithm, although we lack a full analysis, we expect a similar analysis to be possible; see again the answer to Q1 (Error analysis) of Reviewer ZLz3.

Now the question is: why does Algo. 2 indeed (approximately) compute the average policy, as required in FP?

The main idea is that sampling an action from $\bar\pi^K(\cdot|x)$ is equivalent to first sampling an iteration number $k \in \{1, \dots,K\}$ and then sampling an action according to the best response computed in iteration $k$, namely $\pi^*_k(\cdot|x)$. This is what is done in Algo. 1 but this requires keeping in memory all the past policies, which is inefficient.

Instead, Algo. 2 does the following: at iteration $k$, it stores a batch of state-action pairs generated with $\pi ^*_k$ and keeps them in a replay buffer. Then, a policy is trained to learn the distribution of all the state-action pairs in the replay buffer.

Since there is an equal probability to pick a pair generated by $\pi^*_k$ for any $k\le K$, the policy trained in this way really produces the same distribution as the theoretical average policy $\bar\pi^K$. Technically, Algo. 2 achieves this through the Negative Log-Likelihood (NLL) loss. For more details, we refer to line 794 of the Appendix, where we present the pseudocodes of the two algorithms.

W4 (Assumptions): Thank you for raising this point. Here is a more precise and strongly justified response (that we are going to add in Section 2) regarding our assumptions, based on our discussion and the provided literature.

• Existence: Our work assumes the existence of a Mean Field Nash Equilibrium (MFNE). This assumption is grounded in established results for related problems, as existence proofs typically rely on fixed-point arguments under specific regularity conditions on the coefficients not yet proven for our exact setting. Specifically, our justification relies on the work of [Saldi et al., 2020], who established the existence of an equilibrium for a broad class of discrete-time MFGs in an infinite-horizon, risk-sensitive setting. Crucially, their work also provides an argument that this infinite-horizon, risk-sensitive cost can be approximated by a finite-horizon (risk-neutral) game. Since our finite-horizon setting is a well-posed approximation of a problem where existence is proven, it is standard to assume that an equilibrium also exists in our case. Furthermore, we expect that existence holds under mild assumptions (typically: continuity) using the Schauder fixed point in a suitable functional space. Under more restrictive assumptions (typically: Lipschitz continuity with a small Lipschitz constant), the Banach fixed point yields both existence and uniqueness. While a full proof is beyond the scope of the present paper, which focuses on a DRL algorithm, we included this discussion in Section 2 of the paper, after the definition of MFNE (Definition 1).

• Uniqueness of Equilibrium: For our theoretical convergence discussion, we also assume uniqueness. This is not required for the DEDA-FP algorithm itself to run. Following the approach of [Elie et al., 2020], uniqueness can be ensured by adopting the standard Lasry-Lions monotonicity assumption. In simpler terms, wherever a distribution $\mu_t$ places more agents compared to another distribution $\nu_t$, the cost to be there under $\mu_t$ must also be higher.

Please let us know if any further technical details or the precise Assumptions paragraph would be helpful. We would be glad to discuss this topic further

W5 (Infinite dimensional spaces): Thank you for this suggestion. We added a sentence in the abstract and at the beginning of the introduction to stress that, since the spaces are continuous, the class of policies is infinite-dimensional and hence using function approximators for the policy is unavoidable. Likewise, since we aim at learning a model for the mean-field, which is an element of $\mathcal{P}(\mathcal{X})^{N_T+1}$, using function approximators is necessary. Neural networks are natural choices due to their empirical success in a wide range of machine learning tasks.

W6 (Algo. 3 comparison and figures): We thank the reviewer for their insightful comments and questions regarding the experimental comparison. We will address the two points separately.

1. Comparison: The full comparison results for Algo. 3 on the simpler problems are available in Appendix C (supp. material), but we moved them to the main text in the revision.

2. Sampling Efficiency: As the reviewer correctly hypothesized, plots are smoother because our method's efficiency allows for more samples within a fixed time budget, which we argue is the most practical basis for comparison. As shown in the table at line 266, DEDA-FP generates trajectories over 10x more efficiently. This allows it to produce a much higher-fidelity environment approximation in the same amount of time, a core advantage of our approach. To directly address the reviewer's curiosity, we produced a new plot using an equal number of samples, and the resulting distributions are indeed visually more similar. However, we believe the comparison based on computational time remains the most relevant testament to our method's advantage.

We hope that our detailed response regarding time-dependence and experimental details has fully addressed the reviewer's concerns. We hope the reviewer will consider raising their score. We look forward to any further discussion.

References:

[Elie et al., 2020] Elie, Romuald, et al. "On the convergence of model-free learning in mean field games." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 05. 2020

[Saldi et al., 2020] Saldi, Naci, Tamer Başar, and Maxim Raginsky. "Approximate Markov-Nash equilibria for discrete-time risk-sensitive mean-field games." Mathematics of Operations Research 45.4 (2020): 1596-1620

Thanks for your response! I think it is valuable to have a deeper conversation about the nature of nonstationarity in MFGs, and the examples you have provided are definitely interesting for future consideration. That being said, the convergence guarantee you have given in the latest response to Reviewer Zlz3 is definitely more crucial to be added to the present draft. The proof idea seems correct, and relies on similar techniques in the literature (e.g. Elie et al 2020). I do have one small question -- are there any known results for how to estimate or control the distribution error induced by the CNF? This could motivate the study of how to appropriately bound each of the error terms in a statistical learning sense, which has been explored before in the RL literature (e.g. [1]). This general line of inquiry is, of course, very challenging from a theoretical perspective but certainly something to consider for future work.

Overall, adding this to the paper would improve the quality of the submission and give credence to the experimental results. For this reason I am happy to increase my score to an accept.

[1] Farahmand, A. M., Szepesvári, C., & Munos, R. (2010). Error propagation for approximate policy and value iteration. Advances in neural information processing systems, 23.