A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models

We thank the reviewer for their detailed and insightful comments. We appreciate that the reviewer finds our results novel and potentially impactful. We address the questions as they detail the concerns raised as weaknesses. We hope this clears some of the confusion.

Questions

1. This is a very good question and is the heart of how we define hidden symmetries. You are absolutely correct that for any $\tilde{\theta}\in\tilde{\Theta}$ and $p\in P$ we have $f_{p\tilde{\theta}}=f_{\tilde{\theta}}$. The part that can break is line 128 in Definition 2.7, we require $p$ to be a homeomorphism. While originally $p:\Theta\to\Theta$, when restricting to $\tilde{\Theta}$ all we know is $p\vert_{\tilde{\Theta}}:\tilde{\Theta}\to p(\tilde{\Theta})$. The range of $p(\tilde{\Theta})$ is not necessarily $\tilde{\Theta}$! So $p|_{\tilde{\Theta}}$ is not necessarily a homeomorphism! We mention that this is the issue in line 180-181. A concrete example is a permutation transforming the weights in Figure 6a and 6c. Both weights describe the same function but the weight sharing structure is different.

   We do not quite understand the question about the paragraph before Definition 2.7. We only start using $\tilde{P}$ in Section 3.1. We would appreciate clarification about what the reviewer meant.

2. Thanks for asking! First we clarify what happens when permuting neurons. Suppose the $W_\ell$ are the ingoing weights to neurons in layer $\ell$ and the outgoing weights are $W_{\ell+1}$. Then if we permute the neurons, we transform $W_\ell\to PW_\ell$ and $W_{\ell+1}\to W_{\ell+1}P^T$, that is we permute the rows of the ingoing weights and columns of the outgoing weights, two different matrices. Perhaps the reviewer is confusing this parameter symmetry with equivariance constraints, in which we transform both rows and columns of the same matrix.

   As an example, in Figure 4, $W_\ell$ are the depicted weights and $W_{\ell+1}$ is a constant matrix unaffected by any permutations. In particular, permuting the first 3 rows of $W_\ell$ is a hidden symmetry and would bring us away from the constrained subspace (see difference between Figure 6a and 6c, both would give the exact same function but 6c has a different weight sharing scheme and does not lie in the original equivariant subspace). It is only where $\theta_1=\theta_2$ that this permutation has no effect.

3. We apologize for the confusion. Your interpretation of neuron point is correct. We originally borrowed terminology from [2] which uses the term neuron points and defines irreducibility of neuron points. What we intended for a neuron block was a neuron point that satisfies the equivariance constraints corresponding to a specific choice of transitive permutation representation on a set of neurons. Perhaps the following is more clear:

   Suppose we define a transitive permutation rep on a block of $m$ neurons. Consider a $m$-neuron point which satisfies the corresponding equivariance constraints. If this $m$-neuron point is reducible, then the reduced $n$-neuron point satisfies the equivariance constraints for a corresponding transitive permutation rep on the reduced block of $n$ neurons. Further $n|m$.

   Yes, you are correct that the subspace is no longer $\theta_1=\theta_2$ but $\sqrt{n-1}\theta_1=\theta_2$. In Figure 5b, this is why the boundary line does not have slope $1$ but instead slope $\sqrt{2}$.

4. Thanks for bringing up this work! In [1], the main results are how critical points of unconstrained, equivariant, and augmented networks are related, followed by a discussion of their respective stabilities. However, there is no discussion on how these critical points arise which is why this result is different from Lemma 3.2.

   In our work, the focus is on describing a general mechanism (hidden symmetries and hidden fixed point spaces) which gives rise to critical points (Lemma 3.2) and can even imply the existence of additional minima (Theorem 3.3). We view the hidden symmetry mechanism in Lemma 3.2 as a generalization of obtaining critical points from regular parameter symmetries [2,3]. The hidden symmetry mechanism is then applied to equivariant MLP layers built with permutation representations (Theorems 3.4 and 3.5 + experiments).

   One additional note we want to make is that [1] focuses on a particular equivariant subspace since it assumes a specific choice of representation on the hidden layers. In our work, we have the interesting empirical observation that unconstrained models can learn to change the representation in the hidden layers.

5. This is a great question! What we mean is that spherical signals satisfy a generalized version of permutation representation.

   In a more general definition, we could consider an abstract permutation representation as homomorphism $G\to\mathrm{Aut}(X)$ where $X$ is some object (maybe a set, or maybe it has some additional structure). In the case where $X$ is a finite set, then $\mathrm{Aut}(X)\cong\mathrm{Sym}(|X|)$ and we recover the usual definition of an abstract permutation representation. In the case of spherical signals, we take $X$ to be $S^2$ and the abstract permutation representation is the action of rotations on the sphere.

   To move from an abstract perm rep to a perm rep, we then construct a vector space (over field $\mathbb{R}$ for us) with elements of $X$ as the basis. Then any point in the space is of the form $\sum_{x\in X}c_x x$ where $c_x\in\mathbb{R}$ are the coefficients. But we can also interpret $c_x$ as a function $X\to\mathbb{R}$. For finite $X$, the induced representation on $\\{X\to\mathbb{R}\\}$ gives us the usual $|X|$-dimensional perm reps. For infinite $X$ such as $S^2$, we instead have an induced representation on the infinite dimensional vector space of $\\{S^2\to\mathbb{R}\\}$ which corresponds precisely to the spherical signals.

   We can make this point more clear in the appendix.

Limitations

Thanks for this suggestion. Indeed the dimension condition is somewhat restrictive. We plan to add a discussion of how restrictive each of these conditions are.

However, we would like to point out that Lemma 3.2 is very general and can be applied for any linearly constrained model regardless of width and depth. We believe the identification of these hidden symmetry induced critical points is very important even if they are not minima or imply minima (as happens for the additional criteria of Theorem 3.3). This is because saddle points are a known problem for optimization [4].

We thank you again for your time in reviewing our work. Please let us know if you have any other concerns.

References

[1] Nordenfors, Oskar, Fredrik Ohlsson, and Axel Flinth. "Optimization Dynamics of Equivariant and Augmented Neural Networks." Transactions on Machine Learning Research.

[2] Simsek, Berfin, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. "Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances." In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.

[3] Fukumizu, Kenji, and Shun-ichi Amari. "Local minima and plateaus in hierarchical structures of multilayer perceptrons." Neural networks 13, no. 3 (2000): 317-327.

[4] Dauphin, Yann N., Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization." Advances in neural information processing systems 27 (2014).

We thank the reviewer for their insightful comments. We appreciate that the reviewer finds our work well motivated, novel, theoretically interesting, and well written. We hope our response addresses the weaknesses and questions raised.

Weaknesses:

• Thank you for letting us know and for pointing out some of this in the questions! We plan to add a section in the appendix explaining our notation. We will also double check and update the main body to make sure we define everything.

• Thanks for raising this point! We will make sure to address how strong our assumptions are in the revised version.

  In particular, we would like to highlight that Lemma 3.2 is very general and can be applied for any linearly constrained model regardless of width and depth. We believe the identification of these hidden symmetry induced critical points is very important even if they are not minima or imply minima (as happens for the additional criteria of Theorem 3.3). This is because saddle points are a known problem for optimization [1].

Questions:

• Here, $V_n$ is supposed to be the vector space on which we define our group representation and you are correct for $\Theta$. We mean the usual dimension of a vector space by $\mathrm{dim}$. That $\Theta_H$ is a subvector space is elementary enough that we believe it can be omitted. However, your comment did make us realize we assume a linear group action which we will add in the revised version. We have written a short proof below.

  For any $\theta_1,\theta_2\in\Theta_H$, any $a_1,a_2\in\mathbb{R}$, and any $h\in H$ we have $h(a_1\theta_1+a_2\theta_2)=a_1h\theta_1+a_2h\theta_2=a_1\theta_1+a_2\theta_2$ where we used linearity of the group action. Therefore $a_1\theta_1+a_2\theta_2\in\Theta_H$ so $\Theta_H$ is a vector space.

  This is a very good question! The assumption is used in line 616 where we state the boundary consists of $S_1$ and $\bar{B}_C(0)$.

• This is a very good suggestion! The dimension condition is quite strong. However the other conditions are quite weak. We would also like to highlight that the dimension condition is only necessary for Theorem 3.3. Lemma 3.2 only requires a mild geometric condition which is broadly applicable.

• This is a great question! We expect increased width to mitigate the spurious minima issue. First, in some cases the dimension condition is less likely to be satisfied so the spurious minima is no longer guaranteed. Secondly, with wider layers, there is a greater chance that some components of the model are initialized in the “correct” half so the model can learn to prioritize those components.

  Finally, we would like to add that Lemma 3.2 and Theorem 3.3 lays the foundation for future work which may investigate how the number of non-global minimum critical points grows as we overparameterize, similar in spirit to [2]. This could potentially give insight on differences in scaling behavior between equivariant and nonequivariant models.

• Thanks for this suggestion! We will make sure to do this, if accepted, with the extra content page

Limitations

• We want to emphasize that the hidden symmetry mechanism in Section 3.1 is quite general and independent of equivariance. While we do focus on applying this to finite groups and permutation representations in the rest of the main body, we do also have some results for rotation symmetry in Appendix C.3. In that case, we show $S^2$-activations which use spherical signals (an analog of a permutation representation) can also suffer from the same problems identified in the paper. Note that $S^2$ activations are used in real models such as EquiformerV2.

  Finally, the main difficulty for other architectures is figuring out the right way to embed it as a constrained subspace of a more general model class. For permutation representations with pointwise nonlinearities, this is easy since MLPs naturally give the more general class. For translation symmetry, we note that convolutions can also be thought of as a restriction of a MLP and we at least expect Lemma 3.2 to apply. However, finding a useful unconstrained model class for layers with tensor products is more difficult.

We thank you again for your time in reviewing our work. Please let us know if you have any other concerns.

References

[1] Dauphin, Yann N., Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization." Advances in neural information processing systems 27 (2014).

[2] Simsek, Berfin, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. "Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances." In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.

I thank the authors for their reply to my questions.

I thank the authors for clarifying my questions, especially the one stating (in their global answer):

"Mild assumption. This just says if we go in any direction of parameter space far enough, the loss will keep increasing. This is reasonable as we expect very large parameter values to give divergent predictions. Further, the purpose of this assumption is to show we can draw a region with no boundary critical points, which can happen even if the assumption is false."

To clarify that, if I understand well, any regularization term can enforce this condition. Otherwise, without regularization, it does not seem trivial to me why it should be true? Can you comment on that? I believe that it's not because the weights are large that the loss is also large? If for example only two weights are very large, but whose contribution could cancel, it may not hold?

Also just to clarify by "for any direction \hat{r}\in \tilde{\Theta}" you mean any unitary vector? It could be good to clarify it if yes.

We thank the reviewer for their positive evaluation of our work! We appreciate that the reviewer finds our explanation of the novel hidden symmetry phenomenon clear and the experiments well constructed. We hope our response addresses the weaknesses and questions raised.

Weaknesses

• First, while we do focus on models with permutation representations, we emphasize that the results in Section 3.1 are independent of equivariance and these results apply to any architecture we can view as a constrained version of a more general model.

  Whether hidden symmetry induced minima happen in practice is a very good question. First, we would appreciate a reference for degenerate Hessians in real GNNs, is this an empirical observation or provable one? In addition, the proof of Theorem 3.3 does not actually require all Hessians to be degenerate, just that the one used in the hidden fixed point space is nondegenerate. The current assumption that all critical points are degenerate was made to simplify the statement of the theorem but can be changed in the revised version. The assumption on dimensions is the most restrictive one.

  As for practical usefulness, we do also have some results on $S^2$ activations (Appendix C.3), a component used in real equivariant models such as EquiformerV2. There are conditions where spurious minima arise and we sometimes empirically still see such minima even when the dimension condition of Theorem 3.3 is not satisfied.

  Finally, while complaints about applicability of Theorem 3.3 are more justified due to the dimension assumption, we emphasize that Lemma 3.2 is very general and has very mild assumptions. We believe the identification of these hidden symmetry induced critical points is very important even if they are not minima or imply minima (as happens for the additional criteria of Theorem 3.3). This is because saddle points are a known problem for optimization [1].

Questions

• See weaknesses

• This is a great question! Again, we emphasize that the hidden symmetry idea of Section 3.1 is independent of equivariance and could be applied to other types of networks.

  Further, we would like to add that Lemma 3.2 and Theorem 3.3 lays the foundation for future work which may investigate how the number of non-global minimum critical points grows as we overparameterize, similar in spirit to [2]. This could potentially give insight on differences in scaling behavior between constrained and unconstrained models.

• This is a very interesting question! Before answering, we want to emphasize that our work focuses on describing the loss landscape by characterizing critical points and minima rather than navigating optimization on the landscape (eg. gradient descent). The latter has a number of subtleties including a crucial dependence on parameterization [3].

  As for the usefulness of weight decay and gradient normalization, we expect these will not help. If we model weight decay with L2 regularization, we note that the parameter symmetries will still be symmetries of the loss function and so it should not affect our analysis. For gradient normalization, we expect no component perpendicular to the boundary between good and bad regions in the first place and normalizing should not affect this. We also tried using Adam instead of gradient descent and found qualitatively similar results to those in Figure 5 and can add these results in the appendix.

We thank you again for your time in reviewing our work. Please let us know if you have any other concerns.

References

[1] Dauphin, Yann N., Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization." Advances in neural information processing systems 27 (2014).

[2] Simsek, Berfin, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. "Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances." In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.

[3] Kristiadi, A., Dangel, F., & Hennig, P. (2023). The geometry of neural nets' parameter spaces under reparametrization. Advances in Neural Information Processing Systems, 36, 17669-17688.

We thank the reviewer for their positive evaluation of our work! We appreciate that the reviewer finds our work timely, the presentation clear, and the experimental setup supportive and insightful. We hope our response addresses the weaknesses and questions raised.

Weaknesses:

• Thanks for this feedback! Indeed the current work focuses on identifying an interesting mechanism (hidden symmetries) which generates critical points and even spurious minima in constrained models (e.g. equivariance). While providing explicit guidance on relaxation techniques is beyond the scope of our work, we can provide suggestions for mitigating the effects of the spurious minima.

  In practice, we expect increased width to overcome the spurious minima issues in constrained networks. For wide networks, we expect there to be enough transitive blocks initialized on the “good” side so a network could learn to only use the “good” blocks and ignore the “bad” blocks. Further, it turns out if a previous layer is wide, the dimension condition of Theorem 3.3 is less likely to hold.

• We would like to clarify that the theoretical portion focuses on the loss landscape which is independent of optimization methods. Figure 2 depicts a gradient argument to give intuition for Theorem 3.3 but the proof does not rely on gradients. Assuming a nice parameterization, we do not expect SGD to cross the hidden fixed point space. This is because the loss landscape generated from a subset of samples is still subject to the same parameter symmetries which generated the fixed point space. Whether SGD can help for other parameterizations (equivalently changing learning rates on individual parameters) is an interesting question. However, we still expect that a network gets stuck when initialized too close to a bad minima.

  In addition, we ran some experiments using Adam instead of gradient descent and obtained similar results to Figure 5 already in the paper.

Questions

• See weaknesses 1

• See weaknesses 1. We also want to add that the hidden symmetry mechanism described for permuting neurons applies to every layer so we expect increasing depth to have minimal impact on those results.

• See weaknesses 2

  Finally, we would like to add that Lemma 3.2 and Theorem 3.3 lays the foundation for future work which may investigate how the number of non-global minimum critical points grows as we overparameterize, similar in spirit to [2]. This could potentially give insight on differences in scaling behavior between equivariant and nonequivariant models.

We thank you again for your time in reviewing our work. Please let us know if you have any other concerns.

References

[1] Dauphin, Yann N., Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization." Advances in neural information processing systems 27 (2014).

[2] Simsek, Berfin, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. "Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances." In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.

We thank the reviewer for their insightful comments. We appreciate that the reviewer finds our work compelling, principled and well written. We hope our response addresses the weaknesses and questions raised.

Weaknesses:

• We agree that it would be nice to include proof sketches but were unable to due to space constraints. Instead, we provided Figure 2 giving an intuitive argument for why Theorem 3.3 is reasonable and Figure 3 which tries to visually explain the results of Theorems 3.4 and 3.5. We felt this provided a reasonable balance especially since some of the proofs are rather mechanical. However, we are open to further suggestions!

• Thanks for this feedback! The high level idea is that the fixed point subspace for a parameter symmetry of permuting two neurons corresponds to those two neurons having the same input and output weights. From Lemma 3.2 and Theorem 3.3, we care about how this fixed point space intersects with an equivariant subspace. What Theorems 3.4 and 3.5 show is that within an equivariant subspace, equivalent neurons cannot happen in isolation. Instead, the existence of a single pair of equivalent neurons must imply a number of other neurons also be equivalent in a structured way as depicted in Figure 3.

• Thanks for this suggestion! If accepted, we will add figures to motivate/describe the hidden symmetry idea which underlies our main results. We hope the reviewer finds the existing Figure 2 insightful for motivating Theorem 3.3.

  In addition, we tried to highlight the key takeaways in the current manuscript at the end of both the abstract and introduction. Upon reflection, we believe the following modifications for the first 2 points will make the ideas more clear.

  • By viewing a given architecture as a constrained version of a larger function class, we prove that the previously hidden parameter symmetries of the larger class can imply the existence of critical points and even spurious minima for the constrained network
  • Hidden symmetries induce multiple symmetrically related but different constrained subspaces. Empirically, we find a spurious minimum in an equivariantly constrained subspace is often a saddle in the unconstrained space and escapes to a better minimum in a symmetrically related subspace.

Questions:

• Line 24-25: This is meant to be a parody of the famous first sentence of the Tale of Two Cities novel which inspired our title. This can be removed if the reviewer feels strongly.

• Line 85: Good suggestion! We will add references to standard group theory texts for Schur’s lemma. In addition, we will add a discussion of Schur’s lemma in the group theory section of the appendix.

• Line 94: It is not required that $G$ be finite, however the permutation representations are finite in this definition. In principle, we can extend the definition to cover infinite dimensional representations by replacing $\mathrm{Sym}(n)$ with $\mathrm{Aut}(X)$ for some object $X$ in Definition 2.4. We can add a discussion of this extension in the appendix.

• Line 105: Yes, although we would like to only use $\rho$ to simplify the writing. We will make sure to clarify this in the revised version.

• Line 107: Yes, exactly!

• Line 109: We did put this in the appendix (but forgot to fix the numbering). It starts on Line 545 or also Lemma A.15.

• Line 128: You are right that bijection is all we need. We originally wanted to only consider “nice” transformations which is why we chose homeomorphisms.

  However, upon closer inspection, we realize that we assumed linear transformations in our main theorems without mentioning it and will update our manuscript to include this. This is not a major restriction as most commonly considered parameter symmetries (permutations, scaling) are linear transformations.

• Line 152: We believe changing this to “If $\Sigma$ is …” should fix this

• Line 179: No, we mean a subspace satisfying some other linear constraints (e.g. equivariance constraints)

• Line 180: Thanks for pointing this out! We meant for $p$ to be a parameter symmetry of the unconstrained networks.

• This is a good question! We do indeed expect the spurious minima problem to mostly disappear as we increase width. For wide networks, we expect there to be enough transitive blocks initialized on the “good” side so a network could learn to only use the “good” blocks and ignore the “bad” blocks. Further, if a previous layer is wide, the dimension condition of Theorem 3.3 is less likely to hold.

  Finally, while the dimension condition of Theorem 3.3 restricts its applicability for wider networks, we emphasize that Lemma 3.2 has very mild conditions and will still imply hidden symmetry induced critical points! This is still very important because saddle points are a known problem for optimization [1]. Further, this Lemma lays the foundation for future work which may investigate how the number of non-global minimum critical points grows as we overparameterize, similar in spirit to [2]. This could potentially give insight on differences in scaling behavior between equivariant and nonequivariant models.

We thank you again for your time in reviewing our work. Please let us know if you have any other concerns.

#References

[1] Dauphin, Yann N., Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization." Advances in neural information processing systems 27 (2014).

[2] Simsek, Berfin, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. "Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances." In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.

We thank the area chair for these suggestions and bringing up other related work. We separately posted a general comment highlighting our main contributions and discussing the practicality/limitations. We primarily focus on comparison to other works on teacher-student setup in this response.

As pointed out, the two-layer teacher student setup has been the subject of numerous studies. However, the focus of our work is not on understanding such toy models but rather using them to illustrate the main theoretical mechanism described in our work. As such, we believe our work is quite distinct from such studies.

However, works focused on two-layer teacher student setups do contain many interesting ideas which complement some of our own.

In [1], the focus is on understanding SGD dynamics in two-layer networks. In particular, they derive a very interesting set of differential equations to describe training behavior. However, this is quite different from our goal of understanding loss landscapes. Further, our training setup is a bit different as we employ weight sharing to enforce equivariance constraints.

In [2] (and a series of related works by the same authors), the key goal is understanding minima of two-layer ReLU networks. In particular, they introduce a very interesting symmetry based technique to characterize families of minima. However, our understanding is that the teacher network in their setup is fixed, with the identity as the first weight matrix and a constant 1 matrix as the second weight. This affords a nice comparison since their teacher network is equivariant and our Theorem 3.3 also applies.

In particular, we might expect an additional minima in the constrained space where the diagonal value is less than the off diagonal, this seems to correspond to the family 1(b) in Theorem 1 of [2]. Similarly, by considering equivariance to subgroups of $S_d$ and invariance of the input distribution to $S_d$, our Theorem 3.3 would also imply existence of the other families. Of course, what is powerful about their Theorem 1 is it not only shows existence but completely characterizes what these minima look like.

While a very impressive and comprehensive characterization, it is unclear to us how easily the techniques of [2] generalize to other architectures and less symmetric teacher weights. In contrast, our Lemma 3.2 and Theorem 3.3 package the hidden symmetry idea into a relatively general statement.

References

[1] Goldt, Sebastian, Madhu Advani, Andrew M. Saxe, Florent Krzakala, and Lenka Zdeborová. "Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup." Advances in neural information processing systems 32 (2019).

[2] Arjevani, Yossi, and Michael Field. "Analytic study of families of spurious minima in two-layer relu neural networks: a tale of symmetry ii." Advances in Neural Information Processing Systems 34 (2021): 15162-15174.

Since the window for discussion closes soon, I would like to share my impressions briefly and openly. Please read the following in the spirit of constructive feedback.

You have acknowledged errors in the proof of Lemma 3.2, and you indicate that you would like to restate and change the assumptions.

• Indeed, the proof is incorrect by items 2 and 3.

• In fact, the correct reasoning reduces to a straightforward one-line exercise:
  $x \in V^H \Rightarrow \nabla f(x)= \nabla f(hx) = h \nabla f(x) \Rightarrow \nabla f(x) \in V^H$.

• Notably, no group averaging is involved here (and you never defined the action on the tangent space).

You have acknowledged serious errors in Theorem 3.3, both in the statement and in the proof.

• Here, the proof is not fixable, because the statement is false.

• If you plan in the future to add the condition from Lemma 3.2, then Lemma 3.2 holds, of course, but the entire discussion of the codimension-one condition (needed to invoke Lemma 3.2), both in the paper and in the reviews, becomes irrelevant, as does the experiment section in its present form.

You have acknowledged that although the paper is centered on MLPs, Theorem 3.3 does not apply to networks with more than one trainable layer, and that a fix is beyond the scope of this work.

• Indeed, already if the second layer is trainable, the degeneracy condition fails.

You have acknowledged that the assumptions do not hold for the teacher–student setting in the experiment section.

• In fact, it is not even differentiable everywhere, let alone $C^1$. In particular, the condition relying on $\nabla L(C\theta) > 0$ is trivially false (this is explicitly used in the proof of 3.3), and the degeneracy condition has never been checked.

You write: “we believe experimental results are more interesting if they agree with theory even without satisfying the necessary assumptions.”

• Then, what is it that one can reasonably hope to learn from an experiment?

• The two minima, family 1(a) and family 1(b), were detected following On the Principle of Least Symmetry Breaking in Shallow ReLU Models, which also studies different targets and distributions. The families of minima shown in the experimental section are not a contribution of the present work, as the authors have acknowledged.

It is my considered opinion that the core idea has potential for further development. However, serious flaws are present in every aspect of the current execution. While the five reviewers assigned positive scores, I believe the extensive list of critical issues in execution all but obscures the value of the core idea.

Given the scale of errors and the revisions required, I wonder if you might consider resubmitting to the next conference after addressing all comments I outlined earlier.

Note that the list of major issues I have raised is far from exhaustive; given the short time window, I have omitted other serious problems. One additional example of the ones omitted: in your terminology, a “hidden space” is the fixed-point space of a subgroup $H$. In the teacher–student setting, $\tilde{\Theta}$ is clear, but what exactly is $H$? Your paper does not allow this question to be answered, because the group action is not properly defined.

In addition, here are a few examples of the writing issues I had to omit:

• In the literature, “hidden symmetry” refers to a group action that is indeed acting on the space but has yet to be detected.
• Direct sums are denoted by $\oplus$, not $+$.
• Fixed-point spaces are denoted $V^H$, not $V_H$.
• The principle of symmetric critically not used nor needed in this context
• …

I agree with the authors' concerns. The last comment of the area chair was very unfair towards your contribution, and is overexaggurating and/or misrepresenting relatively minor weaknesses/inaccuracies of the work. An additional discussion thread has been opened, where I think that the authors have not been made readers, and I have expressed this view also in that thread.

I support the senior area chairs taking a look at this case.