We sincerely appreciate reviewer's valuable comments and support. We first respond to the suggestions on the conclusion part of the review:

1. Enhancing the figures

We have updated figures according to your suggestions. Examples are provided through the anonymous link.

---

2. Expanding the literature review

We will add an extended version of the following paragraph to the Related Works.

Differential Equations for Optimization. Differential equations are powerful tools for studying optimization algorithms. In the following, we present a very brief summary, and future related works can be found in the references therein. For minimization, [1,2] purposed ODEs to investigate momentum in convex minimization. The use of SDEs for the stochastic setting was developed by [3] and future developed by [4] recently. For min-max games, [6] introduced methods for deriving high resolution ODEs to study algorithms' convergence behaviors. [5] purposed mean dynamics for Robbins–Monro algorithms to study algorithms' long-time behaviors. For the stochastic setting, recent work of [7] derived SDEs to model the behaviors of several min-max algorithms under the weak approximation framework.

---

3. Experimental setup for the GANs

We will add the following paragraph into the Experiments on GANs training part.

Experimental Setup. The experimental setup generally follows the Wasserstein GANs training framework of (Gulrajani et al., 2017).

• Neural network architecture: Both generator and discriminator use the ResNet-32 architecture

• Learning rate: Both generator and discriminator use the learning rate 2e-4, with a linearly decreasing step size schedule

• Gradient penalty coefficient: 10

• Batch size: 64

• During the training, We update both the generator and discriminator in each iteration, which is consistent with the algorithms investigated in this work.

---

4. Experimental validation that the derived ODEs do track the respective optimizers

We provide experiments in all the three examples provided in Figure 1 of (Compagnoni et al., 2024). Results are provided through the anonymous link.

Our continuous-time equations can accurately approximate algorithms' trajectories. For exmaple, for test function 1, trajectories of ODEs and algorithms converge to the same limit cycle. Under the step size $h =0.001$, the maximal errors are around $0.01$ after $10^{5}$ iterations.

---

---

In the following, we will address your concerns in the other parts of the review.

5. It is unclear whether they used mini batching while training the GANs or not: If this is the case, the results are even stronger.

We used the mini batching with batch size of 64 for the experimental results presented in the GANs training (Figure 4). Please refer to Question 3 for future details.

---

6. ... that of SEG is very similar to that of HB ...

We will add extended version of the following discussions in Section 3:

It is worth noting that the Hessian-gradient product type of implicit regularization terms in the ODEs for heavy ball momentum is similar to those discovered in the SDEs of Extra-gradient algorithm [7].

---

7. What stopped you from tackling the stochastic setup?

We specifically study momentum in min-max games under a deterministic setting, following the established line of research that examines momentum in minimization problems under deterministic setting [1,2]. This approach allows us to directly compare the role of momentum in min-max games with its role in minimization, thereby emphasizing their fundamentally different behaviors observed in this paper. We believe that integrating the SDE framework from [7] with our current analysis would be a promising direction for future research. We will highlight this point as a future direction in the conclusion part of this paper.

Reference:

[1] Su et al., A differential equation for modeling Nesterov's accelerated gradient method: theory and insights, JMLR 2016

[2] Wibisono et al., A variational perspective on accelerated methods in optimization, PNAS 2016

[3] Li et al., Stochastic modified equations and adaptive stochastic gradient algorithms, ICML 2017

[4] Compagnoni et al., Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise, ICLR 2025

[5] Hsieh et al., The Limits of Min-Max Optimization Algorithms: Convergence to Spurious Non-Critical Sets, ICML 2021

[6] Lu, An $\mathcal{O}(s^r)$-resolution ODE framework for understanding discrete-time algorithms and applications to the linear convergence of minimax problems, Mathematical Programming (2022)

[7] Compagnoni et al., SDEs for Minimax Optimization, AISTATS 2024

We sincerely appreciate reviewer's valuable comments and support. Please see our itemized responses below:

1. Do you think that you could get better analysis using the framework proposed in Lu 2022?

We thank the reviewer for highlighting the potential connection between our current paper and (Lu 2022). We will add (Lu 2022) to the Related Works section. We believe that exploring the possibility of combining the $\mathcal{O}(r^s)$-resolution technique of (Lu 2022) with heavy ball momentum is a highly promising direction. This could potentially lead to continuous-time equations for heavy ball momentum in min-max games with improved accuracy. In particular, the global convergence property of the ODE proposed by (Lu 2022) would be especially interesting if it can be applied to study the heavy ball momentum algorithm, whose global convergence behavior in general min-max games remains unclear.

---

2. How do you connect the flatness of saddle-points to the flatness of ERM minima in neural nets? Why do you think the GAN gets better FID scores?

We thanks the reviewer for raising this important question. As the experiments in the current paper suggest, the superior performance of GANs trained with negative momentum is related to their implicit regularization effect. This effect guides the algorithm's trajectory to explore regions with shallower slopes in the GANs loss landscape. However, we would like to emphasize that training dynamics of min-max games exhibit subtle differences compared to ERM minima in neural networks. One notable distinction is that first-order methods are guaranteed to converge to local minima in ERM problems [1]. In contrast, for min-max games, such methods might lead the algorithms to converge to limited sets, such as cycles, rather than saddle points [2]. Therefore, we believe the relationship between the "flatness" of the min-max loss landscape and its machine learning applications could be more complex and multifaceted than in minimization problems. We also highlight building a solid theoretical understanding of this relationship as an important future research direction in the conclusion section of the current paper.

From a high-level perspective, we believe that the shallower slope regions of the GANs loss landscape may represent a certain level of "robustness". In these regions, perturbations to the parameters of the generator and discriminator are less likely to significantly impact the values of their loss functions. This offers a potential explanation for why parameters from the shallower slope regions of the GANs loss landscape tend to perform better. We believe that further exploration in this area could be an interesting direction.

---

3. Is there any other implication of the flatness of the minima? How would you connect it to (Ozdaglar et al 2022)?

From the experiments presented in the current paper, we observe that shallower slope regions in the GANs loss landscape tend to exhibit better performance when measured by FID and Inception Score. These two metrics evaluate the quality, diversity, and similarity of individual samples between generated and real data. Therefore, we believe that in the context of GANs, flatness can bring benefits in these aspects. We also believe that further exploring the implications of the flatness of the min-max loss landscape in other applications could be an intriguing area for future research.

We thank the reviewer for pointing out the literature by (Ozdaglar et al 2022). We find the metric of primal gap to study generalization of minimax algorithms purposed by (Ozdaglar et al 2022) is particularly insightful. It would be interesting to explore whether these tools could be connected to the flatness property of the loss landscapes. Additionally, we notice that the theoretical results in (Ozdaglar et al. 2022) are primarily proven for non-convex-concave or convex-concave cases, which differ from the non-convex-non-concave nature of GANs or other pratical applications of min-max games in machine learning, like adversarial training. We believe there is significant potential for further exploration in this area.

Reference:

[1] Lee et al., First-order Methods Almost Always Avoid Saddle Points, Mathematical Programming 2019

[2] Hsieh et al., The Limits of Min-Max Optimization Algorithms: Convergence to Spurious Non-Critical Sets, ICML 2021

We sincerely appreciate the reviewer's valuable comments. Regarding the concern about the practical implications of the results from continuous-time equations (CTEs), we provide a clearer explanation of the importance and relevance of continuous-time analysis:

Continuous-time analysis is a widely accepted methodology for analyzing optimization algorithms. It not only provides novel insights into the behaviors of corresponding algorithms but also proves to be indispensable in certain contexts.

We now turn to a detailed discussion.

Widely Accepted Methodology: We list several papers that have a similar approach to our paper: rather than analyzing the original algorithms, they derive well-designed CTEs for the algorithms and analyze these equations to understand the algorithms' behavior. In minimization, these papers include [1,2,3]. In min-max games setting, papers include [4,5,6]. Although the results in these papers only obtained rigorous proof for CTEs, they are also crucial for understanding the behavior of the original algorithms and are well accepted by the community.

Novel Insights: As the reviewer noted, the results in this paper "reveals fundamental differences in how HB behaves in min-max games versus standard minimization problems." We sincerely appreciate this observation and would like to emphasize that it is precisely the insights gained through analyzing the CTEs that lead us to discover such novel phenomena. While the reviewer comments that " what we have here is merely a continuous dynamical system that aligns with the observed behaviour", we respectfully suggest that this alignment underscores the strength of well designed CTEs in uncovering novel insights into original algorithms.

Indispensable: CTEs are indispensable for studying implicit gradient regularization effects (IGRs), which is the focus of Section 5 in our paper. The foundational work in this area [7] introduced CTEs for gradient descent, demonstrating that these algorithms implicitly favor flat minima, which are referred to as IGRs. IGRs only become apparent through the analysis of the regularization terms present in the CTEs. Several subsequent works, including [4,8], have also relied on the same approach.

We fully agree with the reviewer's point that deriving results for the original algorithms is important, and we will list it as a future work. At the same time, we hope the above discussion demonstrates that results obtained from continuous-time analysis play a crucial role in understanding algorithmic behavior.

Below, we address additional concerns raised by the reviewer.

1. " ... as there is already a remark suggesting that the existing work of Gidel provides an analysis for the discrete case..."

Due to the nature of the proof technique they adopted, the results of Gidel et al. for discrete-time algorithms are only applicable to bilinear games, which is a special case of the games considered in our paper.

---

2. "... momentum with a different parameter setting is, in essence, a variation of Optimistic Gradient Descent (OGD) ..."

We thank the reviewer for pointing out the omission of the discussion in OGD. OGD and HB methods are based on different approaches. OGD is to "incorporate predictions within the gradient oracle framework". In contrast, the mechanism of momentum methods focuses on simulating specific physical processes [9]. These differences are also reflected by the qualitative difference of the two approaches in simple bilinear games, Sim-HB diverge while Sim-OGD converge.

In the refined version we will incorporate a detailed discussion on the differences between OGD and HB.

---

3. "...implicit regularization section, there is no clear mathematical statement ..."

Due to the complexity of the optimization dynamics, findings in the research area of implicit gradient regularization are presented as qualitative descriptions. For example, in previous work [7] of this direction, the results are formulated as a Prediction supported by experimental results. Similarly, we state our results as a Thesis, and support with by experimental results.

Reference:

[1] Kovachki & Stuart, Continuous-Time Analysis of Momentum Methods, JMLR 2020

[2] Muehlebach & Jordan, Continuous-Time Lower Bounds for Gradient-based Algorithms, ICML 2020

[3] Romero & Benosman, Finite-Time Convergence in Continuous-Time Optimization, ICML 2020

[4] Rosca et al., Discretization Drift in Two-Player Games, ICML 2021

[5] Hsieh et al., The Limits of Min-Max Optimization Algorithms: Convergence to Spurious Non-Critical Sets, ICML 2021

[6] Compagnoni et al, SDEs for Minimax Optimization, AISTATS 2024

[7] Barrett & Dherin, Implicit Gradient Regularization, ICLR 2021

[8] Ghosh et al., Implicit Regularization in Heavy-ball Momentum Accelerated Stochastic Gradient Descent, ICLR 2023

[9] Qian, On the Momentum Term in Gradient Descent Learning Algorithms, Neural Networks 1999