Continuous Approximation of Momentum Methods with Explicit Discretization Error

1. Definition 2.1 seems not to be defined explicitly. For example, I cannot understand what "for a constant $C(T) > 0$ ..." is, since $T$ is not defined. Could you provide the explicit definition of an $\mathcal{O} (\eta^\alpha)$-close continuous approximation? (I guess that "for a constant $C(T) > 0$ ..." is replaced by "$\forall T \exists C(T) > 0$..."). I also guess that $\sup\_T C(T) < \infty$ is assumed, since HB seems to not close to HBF when $\limsup\_{T \to \infty} C(T) = \infty$. Anyway, we need mathematically explicit Definition 2.1.

2. Could you provide $C(T)$ in Theorem 3.1? I do not find that Appendix section (Proof of Theorem 3.1) includes the definition of $C(T)$ in Theorem 3.1. The explicit form of $C(T)$ is significant to check the gap between HB and HBF. Also, I cannot follow that HBF is $\mathcal{O} (\eta^\alpha)$-close continuous approximation of HB, since Definition 2.1 is not well-defined.

3. $\gamma_k (\beta)$ in (3) or (5) is a function of $\alpha$. Can we replace $\gamma_k (\beta)$ by e.g., $\gamma_{k,\alpha} (\beta)$?

4. Even if Theorem 3.1 is correct, I cannot understand why the theorem is important, that is, what the contribution of the paper is. Since I do not know the explicit form of $C(T)$ in Theorem 3.1, the following comments and questions are based on my unsupported claims:

• It is useful that the gap $\mathcal{O} (\eta^\alpha)$ should be small to train DNNs. Is it correct? Although I do not understand any significance and usefulness of Definition 2.1, the following are based on an objective such that the gap $\mathcal{O} (\eta^\alpha)$ should be small.
• If $C(T)$ is finite for any $T$ and if $\alpha \geq 1$ is fixed, then the learning rate $\eta > 0$ should be small so that the gap $\mathcal{O} (\eta^\alpha)$ can be small. Is it correct?
• Let $\eta \in (0,1)$. Then, the parameter $\alpha \geq 1$ should be large so that the gap $\mathcal{O} (\eta^\alpha)$ can be small. Is it correct? If it is correct, then why did the authors use only small $\alpha = 2, 3$ (see also Sections 3 and 4)?
• What are new/significant insights (e.g., appropriate setting of $\eta$, $\mu$, and $\alpha$ to train DNNs) obtained from Theorem 3.1?
• If $\limsup\_{T \to \infty} C(T) = \infty$ or $C(T)$ is very large for $T \geq t_0$ (where $t_0 \in \mathbb{N}$ is a certain step/time), then what is the finding obtained from Theorem 3.1? Remark: If it is theoretically guaranteed that $C(T) < \infty$ in Theorem 3.1 is small, then this question can be omitted.

5. Section 4 said there is a relationship between the bias of HBF and the generalization. I cannot understand the claim. The function $L$ is the empirical loss. Hence, HB and HBF using $L$ can be applied to only empirical risk minimization (ERM). Please explain what the connection between the bias of HBF (Theorem 4.1) based on ERM and generalization for expected risk minimization. As a result, I strongly believe that the authors can explain the relationship between Theorem 4.1 and the generalization. Moreover, what are new/significant insights (e.g., appropriate setting of $\eta$, $\mu$, and $\alpha$ to have better generalization) obtained from Theorem 4.1?

6. According to PyTorch (https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD), one practical HB is defined by (2) with a negative $\mu$ (torch.optim.SGD(params, lr=0.001, momentum=0.9, dampening=0, weight_decay=0, nesterov=False, *, maximize=False, foreach=None, differentiable=False, fused=None). Here, I have a question. What is HB used in the experiments? Just (2) with a positive $\mu$ such as: torch.optim.SGD(params, lr=0.001, momentum= - 0.9, dampening=0, weight_decay=0, nesterov=False, *, maximize=False, foreach=None, differentiable=False, fused=None) ? Please define HB and HBF used in the experiments explicitly.

Each of the points raised in weakness-1 raises a question. In line-261, the authors mention that their derivation strategy is different than that of https://openreview.net/forum?id=ZzdBhtEH9yB. Can the authors specify how is it different, the proof strategy appears to be exactly the same.

1. In Figure 1(a), I noticed that the $\alpha=2$ case gives a better approximation away from the convergence point while $\alpha=3$ is the opposite. I wonder if the authors have some intuition for this behavior.

2. This analysis can be generalized to the stochastic case by for example replacing $\nabla L$ by $\tilde{\nabla} L$ where $\tilde{\nabla}$ denotes the approximate gradient. The approximation can be done by either introducing a diffusion term [1] or an index switching process [2]. I think this might be an interesting future direction.

[1] Stochastic modified equations and dynamics of stochastic gradient algorithms I: Mathematical foundations, Li et al.

[2] Analysis of Stochastic Gradient Descent in Continuous Time, Latz.

• Can the authors elaborate on whether such arbitrary precision continuous approximation is possibly only for first-order ODE? Indeed, as the authors have discussed in the Introduction, second-order ODE approximations for HB have been extensively studied, with their own benefits. For instance, Kovachki and Stuart (2022) stated that second-order (Hamiltonian) ODE can better capture the intuitions of HB's transient regime than first-order ODE approximation.

  • Could this be a further solution? Defining the velocity component $v_t = \dot{\beta_t}$, one could consider an autonomous first-order ODE of the form $(\dot{v_t}, \dot{\beta_t}) = (- v_t - \nabla F(\beta_t), v_t) + \text{correction term}$. Could this give more benefits than the currently proposed HBF?

• I believe the authors have missed this prior work: https://ojs.aaai.org/index.php/AAAI/article/view/26209, which also considers a second-order ODE approximation of HB.

• [low priority] As in the above AAAI'23 paper, it would be interesting to see if HB leads to new implicit bias results in overparametrized linear regression, even just empirically.