Thank you very much for the feedback. Please first see our summary rebuttal above.

Weaknesses:

C is not guaranteed to exist. (Line 70)

Thanks for this comment. This is a well-known fact in the study of symmetries and the Noether theorem. It is required that $J$ needs to be conservative. We will add a note on this. That being said, for the type of symmetries (Def 4.1) we focus on in our work, $C$ always exists.

Usually, the stability of the learning is determined by the largest eigenvalue of the Hessian... The trace may not be an upper bound on the largest eigenvalue as it can contain negative eigenvalues.

Thanks for this question. For (deterministic) GD, the training stability is indeed determined by $\lambda_{\max}(Hess)$. However, this does not apply to SGD. In fact, [1][2] demonstrates that the stability of SGD is impacted by $Tr(Hess)$, rather than $\lambda_{\max}(Hess)$. [1][3] suggest that $Tr(Hess)$ significantly determines the properties of SGD’s solutions.

[1] Wu and Su. The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent. (ICML 2023)

[2] Ma and Ying. On the linear stability of SGD and input-smoothness of neural networks. (NeurIPS 2021)

[3] Li et al. What Happens after SGD Reaches Zero Loss? –A Mathematical Framework. (ICLR 2022)

I think it’s missing dependence on γ in proposition 5.3.

We will fix it in the revision. In the experiments, we let $\gamma=0$ and so we did not use the weight decay term.

The authors proved that in deep linear networks, the noise will lead to a balanced norm for all intermediate layers. This is an important theorem, but it lacks empirical verification....

Thanks for this very constructive feedback. We have added an extensive set of experiments for deep linear nets. We show that for MNIST, a deep linear net (with five hidden layers), the intermediate layers converge to the same norm, whereas the input and output layers are special and move towards different norms. This effect is also shown to be robust across different types of initializations. See Figure 8. In addition, see the additional experiments in Figure 7 and Figure 9.

There is a lack of clarity in how the analyses in Section 5.4 (Equation 24) relate to Figure 4.

Thanks for this criticism. Section 5.4 (and Eq 24) shows that if a model approximately has a symmetry (for example, when it can be approximated by a model with symmetry), then its solution will be biased towards the noise equilibrium of this symmetry model.

Figure 4 shows exactly this. A tanh network can be approximated by a linear network of the same depth and connectivity (because $tanh(wx) \approx wx$ to leading order). For a linear network, it is true that the noise equilibrium will satisfy $W\Sigma_xW^T = U^T \Sigma_\epsilon U$, and so one would expect that under SGD training, a tanh network will also be biased towards satisfying this relation, and this qualitative prediction is confirmed by Figure 4. We will add this explanation to the revision.

Questions:

In line 124-128, the authors state that the SGD dynamics can be decomposed into two independent parts. Why does equation (5) have no effect on the loss?

This is a good question. This statement is made assuming that the fluctuation is not significant. This is supported by the experiment in Figure 3 and the mid panel of Figure 2, where, at the end of training, the fluctuation in the converged solution is small.

Also, there is a deeper mathematical reason for this statement. The fluctuation can also be decomposed into the parts in the subspace parallel to the gradient and orthogonal to it. The fluctuation in the orthogonal subspace (which is our main focus of study) will not affect the loss.

For the double rotation symmetry, why it is hard to write the matrix A explicitly?... I am wondering if the results can be generalized to a general matrix A

Thanks for this question. It is just cumbersome and space-taking to write. First of all, to write it in the form 4.1, one needs to view both matrices (U and W) as vectors, and then the corresponding symmetry matrix will be composed by a sequence of block matrices: A=diag(B,...,B,B^{-1},...,B^{-1}), where each block corresponds to a column or row of $U$ or $W$.

Requiring $A$ to be symmetric is actually to ensure that a Noether charge of the form $\theta^T A \theta$. Extending the result to antisymmetric matrices is highly nontrivial, and we know very little about this type of symmetry. On the one hand, noether charge does exist for some antisymmetric matrix. For an example, consider rotation symmetry for (u,w) in 2d. The antisymmetric matrix is ((0,-1),(1,0)). The symmetry implies the following dynamics for GD: $wdu - udw = 0$. Simplifying the relation leads to (ignoring the sign problem for now) d(log(u) - log(w)) = 0. We have thus constructed a Noether charge for an antisymmetric-matrix symmetry: log(u) - log(w); however, it is no longer in a quadratic form.

On the other hand, we do not know a general formula for them or whether they exist for an arbitrary antisymmetric symmetry, which is an important open problem in the field.

I don’t understand why one needs to assume $C^\*$ to be a constant in line 163...

This is a good question. This is because $C^*$ depends on the value of $\theta$ in the nondegenerate subspace, and $\theta$ has a non-zero time derivative in general. To see this mathematically, note that by definition $C^*= C(\theta_{\lambda^*})$, but $\lambda^*=\lambda^*(\theta)$ is a function of $\theta$ as well, and so $C^*$ sometimes depends on time through $\theta$'s dependence on time (which is often non-zero due to gradient or random noise).

Therefore, a good mental and intuitive picture of the process is that there is a run-and-catch game between $C^*$ and $C$: $C^*$ moves around randomly (due to the motion of $\theta$), and $C$ chases it systematically due to symmetry.

I would like to know new insights that come from the theoretical results in this paper. This is because I am not surprised at the results. For example, can we apply the results to training more practical deep neural networks on practical datasets? Then, can we say new insights for training practical deep neural networks... Anyway, I would like to know contributions of this paper..

Essentially, the answer will have to depend on the subjective opinion of the reader. A contribution for researchers in one field may not seem like a contribution for researchers in other fields. The subjectiveness here is clearly reflected in the fact that the other reviewers rate our contribution as "excellent" while you rate it as "poor." How does understanding an insect's brain help us understand the training of diffusion models? It probably does not, and the problem is not with the brains of insects but with this question being ill-posed. Also, keep in mind that NeurIPS is not just a conference for practical deep learning but also a conference for neuroscience and many other interdisciplinary fields of science.  

With these caveats in mind, we clarify our contribution from different angles below. In short, our theory is interdisciplinary, requiring knowledge and expertise from many different fields to understand, and it could be significant to these related fields. We believe that our research has the potential to inspire new research in pure mathematics, neuroscience, deep learning, physics, and practices in deep learning.

1. Pure Mathematics: Lie-group symmetries have been leveraged to solve ordinary differential equations (e.g., see the book Elementary Lie group analysis and ordinary differential equations). However,  leveraging the Lie group to solve stochastic differential equations is a novel concept and tool for pure and applied mathematics, and our result advances our understanding of stochastic systems. The tool we developed is sufficiently general that it may even be applied to problems outside the field of deep learning.

2. Neuroscience: Understanding how latent representations are formed in animal brains is a fundamental scientific problem in neuroscience. Until now, there has been a very limited understanding of this, and our theory suggests one mechanism of how latent representations form and can inspire neuroscientists to understand biological brains. The fact that symmetry in the solutions can be a core mechanism for latent representation is a novel and potentially surprising message. For example, compare the latent representations in our paper with the biological data in Figure 7B in www.cell.com/neuron/fulltext/S0896-6273(18)30581-6

3. Theory and Mathematics of Deep Learning: The most important contribution of our work is on the mathematical tool side. We established a new mathematical tool (the formalism of the noise-equilibrium fixed points of SGD), and this formalism has the potential to be applied to many problems in deep learning. For example, we showed its relevance to flattening and sharpening effects during training, the warm-up technique, and the solution of deep matrix factorizations. We emphasize that developing new mathematical tools is an important progress per se. Understanding deep learning is a difficult task, and without new mathematical tools, this would be impossible.

4. Physics: Symmetry is a fundamental concept of physics. The standard understanding was that when a continuous symmetry exists, a conserved quantity exists during the dynamics. This concept has also played an important role in machine learning; for example, see the well-known result in https://arxiv.org/abs/1806.00900, where the symmetry and its related conserved quantities are leveraged to prove the convergence of the matrix factorization problem. Our theory, however, shows that this picture is no longer right when the dynamics are stochastic -- conserved quantities no longer exist in general, and, in place of them, we now have unique fixed points where the gradient noises must balance. This is a novel and potentially surprising new concept. 

5. Practices in Deep Learning: Lastly, our theory also has implications for the practices of deep learning, although we do not claim this to be a main contribution. The first contribution is that our theory helps us understand when SGD prefers flatter solutions. The concept of flatness is of practical importance (for example, practitioners leverage it to design algorithms: https://arxiv.org/abs/2010.01412). Another practical insight our theory directly offers is the importance of data preprocessing because our theory shows that the data distribution has a major impact on the learning trajectory and the final solution it reaches. For example, see the newly added experiments in Figure 7, where we directly show that the orientation of the data distribution has a direct effect on the balancedness and sharpness of the learned solution.

Thank you for your criticisms. We believe that your criticisms are due to misunderstanding of mathematics and the related literature and not paying attention to the details given in the immediate context. We will answer concisely due to space constraints. However, please ask us to elaborate on every point that you need more detail.

Questions:

the expected risk minimization or the empirical risk minimization...

Thanks for this question, and we apologize for the confusion. In our formalism, it is not essential to differentiate between the population risk and the empirical risk. The loss function can be either the population risk (if the training proceeds with online SGD) or the empirical risk (if the training proceeds with a finite-size dataset). This is because $\mathbb{E}$ denotes the expectation over the distribution of the training data, which can be arbitrary and is problem/algorithm-dependent.

Line 19...

We are referring to the limit $S\to \infty$. Within the SDE framework, this limit is the GF/GD algorithm.

Line 21: I do not understand what σ2≪1 is.

This is a common notation meaning that "$\sigma$ is far smaller than 1," for example, $\sigma = 0.1$, $0.01$, etc.

Line 58: Please define Σv(θ)...

This is defined and explained in line 58. Colloquially, $\Sigma_v$ is a shorthand for the gradient covariance for a subset of all parameters $v$.

Eq (7)...

Yes. We will fix it.

Line 142: I do not understand the definition of "fixed point...

In the study of dynamical systems (DS), the fixed point is defined differently. In DS, a fixed point is defined for differential equations of the form: $\dot{x} = f(x)$ as the set of $x$ such that $f(x) =0$. See standard textbooks of this field (e.g., Introduction to the modern theory of dynamical systems, by Katok et al.). In this terminology, Def 4.5 is an explicit definition of what "fixed point" means in our context. This definition is related to the fixed point of mappings. Here, the underlying mapping is one step of GD, which maps $\theta_t \to \theta_{t+1}$.

Eq (8): relationship between ℓ(θ,z) and ℓ...

$\ell$ and $\ell(\theta,z)$ are always the same thing. The $z$ argument is always abbreviated for concision. This has been explained in line 61.

Eq (9): We cannot define (9) when γ=0.

This is a good point. It is better to write Eq (9) as $2\gamma C =\sigma^2 Tr[\Sigma A]$, which is now well-defined. We will fix this.

Please define $C(θ^\*)$ explicitly.

This quantity is defined in the line 183. $C(\theta^*)= (\theta^*)^T A \theta^*$, and $\theta^*$ is whatever quantity that satisfies $G(\theta^*) = 0$ (whose existence is guaranteed by Theorem 4.3).

Eq (11): θ′ is not defined.

Thanks for pointing this out. $\theta'$ are the other parameters that are irrelevant to the double rotation symmetry. We will include this in the revision.

Section 5.1: The assumption of A seems to be strong such that A is close to the identity matrix I....

We are afraid to say that this is a major misunderstanding of the problem setting of Section 5.1. Here, the goal is to find $U$, $W$ such that $UW=C$ for a fixed matrix $C$ (note that this $C$ is not $A$). This problem has a lot of degeneracy in the solution. For any invertible matrix $A$: $UA$, $A^{-1}W$ is also a solution to $UW=C$. Hence, this problem has "symmetries." The matrix "A" in our original text has to do with these arbitrary invertible transformations, not with the target matrix $C$.

There is no assumption at all. To derive proposition 5.1, one focuses on the type of symmetries that are close to identity: $A \approx I$; this is because these matrices are the generator of the Lie group for these symmetries. This is not an assumption because $A$ can be an arbitrary matrix.

Proposition 5.1: The assumption "the symmetry (11) holds" (based on A≈I) seems to be strong. Please provide practical examples of empirical risk minimization and SGD satisfying (11).

See the answer above. This is not an assumption. Eq (11) is satisfied by any (deep) matrix factorization problem.

Figure 2...

The experimental details are given in appendix A2. The loss function is MSE and given by Eq (13). The matrix size is 30 by 30, the learning rate is 0.1, and the batch size is 1. We did not specify learning and batch size because the result is highly insensitive to them (as any learning rate and batch size would work and give a similar figure). See Figure 9 of the pdf. We will include these details in the revision.

Line 222

$\epsilon$ is a noise on the label. We will fix it.

Theorem 5.2: Is the theorem based on (11)? If so, please provide some examples of C˙B=0 under (11).

We are not sure if we understand this question. Theorem 5.2 (and Eq. (14)) enumerates all examples where $\dot{C}_B=0$.

Figure 3

They are the same as in Figure 2. $d_x=d_y=30$ and $\epsilon \sim \mathcal{N}(0, I_{30})$ is an independent Gaussian noise. We did not specify learning and batch size because the result is highly insensitive to them. We will clarify this in the revision.

Moreover, I do not understand why a warm-up with $η_{min}=0.001, η_{max}=0.008$, and the switching steps 5000 is used...

Like the previous experiment, the result is robust to these choices, and the specific values of these quantities are not essential at all. The specific values are just chosen out of convenience. For example, a switching step of $5000$ is arbitrary, and we observe that choosing it to be any number larger than 1000 will give a similar result. $η_{min}=0.001$ is also arbitrary and nonessential. Choosing it to be any value such that the model does not diverge at initialization will also work.

...Anyway, I would like to know contributions of this paper.

Thanks for this important question. This is a good question but admits many different answers. Due to space constraints, we will have to answer this in the comment.

Thanks for the update and additional question.

Since $S$ is a fixed batch size, we should replace $S$ with $S_t$?

$S$ is fixed and does not change with time, and so one should not write it as $S_t$. Here, this limit is a mathematical limit of differential equations. One mathematically rigorous way to construct this limit is to define it as a limit of a sequence of differential equations. Operationally, this (roughly) corresponds to running the experiment first with $S=1$. Then, run an independent experiment with $S=2$. Then, with $S=3$, and ad infinitum. During each of these running, $S$ is held fixed, and so does not change with $t$.

I do not know why $\sigma$ is less than 1.

This factor is just a hyperparameter determined by the user of SGD. If the user uses a learning rate of $0.1$ and a batch size of $100$, one naturally has $\sigma=0.1/(2\times 100) \ll 1$.

Thanks for your review. Please first see our summary rebuttal above. We will answer concisely due to space constraint. However, please ask us to elaborate on every point that you need more detail.

Ref [1]...

Ref. [1] shows the existence of an invariant set, mainly due to the permutation symmetry. Our work focuses on continuous symmetry. We will include a reference to clarify this.

Ref [2] ...

This is a misunderstanding of our result. Our result shows that SGD can bias the solutions towards both sharper and flatter solutions, and this depends strongly on the data distribution. See our answers below on this point and the new experiment in Figure 7. We will add this reference and discussion to the manuscript.

Questions:

Most derivations hold for symmetric Lie algebra elements...

This is due to your misreading. In the starting Definition 4.1, the exponential symmetry is explicitly defined with respect to a symmetric A. The reason is exactly what you point out -- if A is antisymmetric, the existence of Noether charge is questionable. We will add a note to clarify this below Def 4.1.

Also, we point out that antisymmetric matrices can have noether charge -- they are currently very poorly understood and hidden from simple analysis. As a counterexample, consider a rotation symmetry for (u,w) in 2d: the symmetry implies the following dynamics for GD: $wdu - udw = 0$. Simplifying the relation leads to (ignoring the sign problem for now) d(log(u) - log(w)) = 0. We have thus constructed a Noether charge for an antisymmetric-matrix symmetry: log(u) - log(w).

line 49: ref [40] wrong...

We will fix this.

Existence of C

Here, a technical remark on the antiderivative C is that one needs to assume that J is conservative, which is well known and so we omitted its discussion. We will add a note on this in the revision.

line 78: eq 5:... a. What is your argument for this G being the correct G?

Yes. You are correct.

For point a, this is a good point that needs to be clarified. We made the specific choice that $G=\sqrt{\Sigma}$ in Eq. (1), but the goal was only to avoid introducing unnecessary technical details. Generally, one should use a general $G$ in the place of $\sqrt{\Sigma}$ such that $\Sigma = GG^T$, and, had we made this choice in Eq (1), what you suggested in point b will be the proof and so the result is the same. We will clarify this point.

"I don't agree with your statement..."

Thanks for this very deep question and suggestion (also, see the answer to point a). What you suggested is a more technical (and more correct) way of proving the relation, but there are multiple routes to get to Eq 31, and our original proof is correct. We also used a hidden condition that is rather obvious: the eigenvalues of $\sqrt{\Sigma}$ are the square roots of the corresponding eigenvalues of $\Sigma$. Thus, a zero eigenvalue of $\Sigma$ must correspond to a zero eigenvalue of $\sqrt{\Sigma}$. We will add a note on this.

line 83...

Our statement in line 83 does not contradict this statement. A point is still a "connected manifold" but of dimension zero.

line 104...

We will fix it.

eq 6...

A is defined to be symmetric.

line 114: why weight decay...

Thanks for this question. Here, the (fundamental) question is not "why" we include weight decay, but "what is the most general setting for which Theorem 4.3 holds correct?" It turns out that Theorem 4.3 holds true for any nonnegative $\gamma$, and this is the main reason why Def 4.2 allows for a weight decay.

Eq 7: the cross term...

We are not sure if we understand this question. There is no contribution from the weight decay term to $\Sigma$ because $\Sigma$ is a covariance of the gradient, whereas the weight decay only affects the expectation of the gradient, not its variance.

line 118...

By definition, A is symmetric. See our answer above.

line 214-215:

Thanks for this question. This example assumes the standard MSE loss for matrix factorization and is misplaced. We will move it below Eq. (13), and include a derivation of it.

eq 13: When $\gamma \neq 0$...

Yes, but this is not a problem. Because (please first see our reply above) Eq 13 can be decomposed into a sum of a symmetry term and weight decay (which does not have the A symmetry). So, this loss function obeys Def 4.2 and so Theorem 4.3 is applicable.

line 235...

These are not functions of the learning algorithm -- they are simply properties of the data points.

eq 39,40...

Matrix A is by definition symmetric.

line 531...

A is by definition symmetric.

Limitations:

...alignment of weights in matrix factorization is already known under weight decay

We are afraid to point out that you have misunderstood our result, and our insight is not known. There are two aspects of alignment:

1. Alignment in norm: under SGD, the norm ratio is determined by the input and label covariance -- and so the balance in norm in general will NOT happen (even with weight decay), and this is a crucial new message we offer. Please examine Eq (14): only for very special $\Sigma_x$ and $\Sigma_\epsilon$, a balance in norm will happen.
2. Alignment in eigenvectors. Our crucial result is again that using SGD, the alignment will not occur even if weight decay is used.

See Figure 7 in the pdf for additional experiments.

"and bias toward less sharp minima (again, known). "

This is also a misunderstanding. See the summary rebuttal and Figure 7-right for an experiment.

I would have liked to see a more direct test of the mechanism the paper...

Besides the result in Figure 7, another new insight is precise balance in the norm only happens in networks with at least two hidden layers, and, for a two-hidden layer network, only the two intermediate layers will always have balanced norms -- See Figure 8.

Hi! We noticed that you have not communicated with us during the discussion period. It would be really helpful to us when revising and improving our manuscript if we could hear more from you. We would be happy to hear any additional thoughts or feedback you have on our work and on our previous reply!