Remove Symmetries to Control Model Expressivity and Improve Optimization

Thank you for the detailed feedback. We will address both the weaknesses and the questions below.

Experiments are limited to small- or medium-scale models. The method’s applicability to real-world, large-scale tasks is insufficiently demonstrated. Benchmarks on high-impact datasets, such as ImageNet, and large-scale architectures, like transformers, would add credibility.

Thanks for this criticism. First of all, we would like to point out that some of our experiments are quite large-scale. For example, each reinforcement learning experiment takes roughly a week to run, longer than training CNNs on imagenet (often 1-2 days on our machines). The scale and difficulty of this experiment might be difficult to perceive unless familiar with RL. We also included a new experiment that implements syre on the vision transformer trained on the CIFAR10 dataset. Figure 12 in the revision shows that our methods work as expected on transformers, avoiding low-rank structures and achieving higher accuracy.

The impact of completely removing symmetries on generalization is not well-discussed. Symmetry sometimes acts as a regularizer, and the absence of analysis on this potential downside is a gap.

Thank you for your thoughtful question. Indeed, symmetries can sometimes act as implicit regularizers that benefit generalization (this point has been discussed in https://arxiv.org/abs/2309.16932, e.g., a point we have acknowledged in the conclusion), and the cases where having symmetry helps, of course, exist. The point we are making is that there are also cases where removing symmetry can be necessary: in our experiments (see Figures 3, 5, 6, and 11), we provide clear evidence of cases where our approach (syre) consistently achieves better generalization performance compared to the standard weight decay baseline ("vanilla" models).

This improvement can be attributed to the fact that in our experimental settings, overfitting is not a significant concern. Instead, removing certain symmetries increases the model's expressiveness by enabling it to capture richer patterns in the data. By doing so, syre enhances both training and testing performance. We agree that in scenarios where overfitting is more pronounced, symmetry-induced regularization might indeed be beneficial. However, in the cases we analyzed, increasing the representation power by breaking these symmetries leads to superior generalization without causing overfitting. A deeper exploration of this trade-off in different regimes is an interesting direction for future work.

How does syre scale to large models like transformers? Have you tested its efficiency or computational impact on such architectures?

Thanks for this question. We have implemented syre on vision transformers in Section B5 of the revision, which shows that syre works well for transformers. Moreover, we test the memory and time consumption in Section B6 of the revision, verifying that syre has a neglectable effect on the training efficiency of various models (with an overhead of less than 5% and gets smaller as the training batch size gets larger).

Would an adaptive bias approach improve the method compared to using a static bias throughout training?

This is a good idea and might be true. There is a caveat to this; however, making the biases learnable also has the potential to introduce more symmetries to the loss function, and these symmetries may, in turn, reduce the effectiveness of the method. Therefore, making the biases adaptive needs to be done carefully, and this could be an interesting future problem.

How sensitive is syre to hyperparameter choices, and do you have heuristics to guide practitioners in tuning them?

This is another good question. We find the method to be quite robust and not so sensitive to the choice of hyperparameters. Since we have not found scenarios whether using a random weight decay $D$ is required, we mainly focused on studying the sensitivity of parameter $\sigma_0$.

We discussed how the parameters could be chosen in Section 5.4. Often, we find that even a small bias (usually one or two orders of magnitudes smaller than the initialization scale) is sufficient to remove the negative effects of symmetry, while having an almost negligible effect on the performance of the model. This is what Figure-3 middle shows for Resnet18. A similar result is shown in the contrastive learning, where $\sigma_0=0.1$ or $0.01$ both improves the performance significantly and do not differ much from each other. However, using a very large $\sigma_0$ certainly hurts performance as it biases the parameters towards a Gaussian distribution.

Do you mind letting the authors know if they have addressed your concerns and questions? Thanks!

Gradient Analysis: Can you clarify the argument around equation (2) about vanishing gradients? How does this differ from the natural vanishing of gradients along symmetry orbits? In the expansion around fixed points, do you consider gradients in directions orthogonal to the symmetry?

Thanks for this question. They are completely different, and this has been studied by many works. The vanishing gradient in Eq (2) implies the existence of saddle points. Again, see https://arxiv.org/pdf/1612.09296 and https://arxiv.org/abs/2309.16932 for good explanations. In contrast, the gradient vanishing along the orbits is not directly related to saddles (and is thus unrelated to entrapment, which you have been correct about).

Initialization and Fixed Points: For a simple network F(x)=UVx, the origin (0,0) is a fixed point, but why is this relevant when standard initialization (Glorot, etc.) avoids such points? Do your arguments about trapping at fixed points apply to more realistic initialization schemes?

Again, as we explained above, there are many many other fixed points, apart from the origin. If U and V are d by d matrices, then this model contains the $S_d$ finite subgroup, which is believed to have (super) exponentially many finite distinct subgroups. For example, if $d=10$, $S_n$ has $29594446$ many finite subgroups (https://groupprops.subwiki.org/wiki/Subgroup_structure_of_symmetric_groups)!

There are many mechanisms for SGD or GD to be trapped in these solutions. On the one hand, these solutions tend to be attractive for SGD at a large learning rate through a mechanism called "stochastic collapse" (https://arxiv.org/abs/2306.04251). Another mechanism is that GD/SGD are attracted to these solutions when weight decay is used. See https://arxiv.org/abs/2309.16932. This is relevant to, for example, why transformers tend to learn low-rank solutions (see https://arxiv.org/abs/2410.23819). Therefore, one does not need to start close to these saddles to be trapped there.

Regularization Effects: Can you provide ablation studies comparing your method with standard weight decay? For the D-norm modification, how does making weight decay ellipsoidal fundamentally differ from standard weight decay in terms of symmetry breaking?

Thank you for your question. To clarify, in Figures 1, 2, 3, 5, 6, 7, 8, 9, 10, and 11, the term "vanilla" corresponds to models using standard weight decay with identical \gamma. We have systematically compared our method (syre) against these standard weight decay baselines in various experimental setups to show our improvement. In terms of the D-norm modification, it is only a theoretical guarantee that we can break infinite-many symmetries together, while empirically, we can effectively break all symmetries without the D-norm.

Theoretical Claims: The condition P\theta’=\theta’ seems unnecessarily restrictive. Can your results be generalized without this condition? For continuous symmetries (like GL(h) in linear networks), can you make precise what subset of symmetries your method breaks?

Thank you for this question. Please first see our answer to the related question above. Essentially, our method removes the symmetry of any group for which a fixed point projection exists -- which is guaranteed as long as this group contains a finite subgroup.

Empirical Validation:Could you provide standard weight decay baselines for all experiments? In scenarios where your method outperforms standard weight decay (like continual learning), what mechanism explains this improvement?

Thank you for your question. To clarify, in Figures 1, 2, 3, 5, 6, 7, 8, 9, 10, and 11, the term "vanilla" corresponds to models using standard weight decay with identical \gamma settings. We have systematically compared our method (syre) against these standard weight decay baselines in various experimental setups. The consistent improvement observed with syre arises from its ability to avoid convergence to low-rank solutions, which is a common limitation of standard weight decay. In particular, we empirically demonstrate that syre preserves higher ranks in the learned representations, leading to better performance across different tasks, including but not limited to continual learning.

Questionable arguments about gradient vanishing: The paper suggests symmetric points trap models due to vanishing gradients. However, in the UV example, fixed points along U=V line have nonzero gradients except at origin and minima. Gradients naturally vanish along symmetry orbits (they're level sets of the loss), but this doesn't imply trapping as claimed.

We are afraid to say that this criticism is based on incorrect assumptions. The fixed points of the UV model are not just the origin and minima.

Any low-rank solution is also a (or multiple) fixed point. If $U$ and $V$ are square matrices with dimension $d$, then for any $d'<d$, there is a low-rank saddle and escaping these low-rank solutions is impossible for gradient descent. This is due to the permutation symmetry (which is a discrete subgroup of GL(n)) of the UV model. This is a well-known fact. For example, see the reference we give above; also, see https://arxiv.org/pdf/2105.12221; also, see the discussion of rotation/permutation symmetry in https://arxiv.org/abs/2309.16932; in fact, this kind of results appear as early as in https://www.ism.ac.jp/~fukumizu/localmin3.pdf

Usually, continuous groups per se do not cause entrapment. It is the discrete/finite subgroups of the continuous groups that actually cause these entrapments. In fact, the GL(n) group contains many discrete subgroups. Actually, this implies that if we removed any discrete subgroup of $GL(n)$, we must also have removed $GL(n)$ itself.

Limited analysis of regularization effects: The D-norm modification merely changes spherical weight decay to ellipsoidal weight decay. The \theta_0 shift changes the center of the regularization. While these modifications may improve optimization, it's not clear they "remove symmetries" as claimed. In the UV example, these modifications may select preferred points along symmetry orbits (like weight decay does) but may not fundamentally break the symmetry structure

This is a good question, but again, it is due to a misunderstanding of our theory. Our method can indeed remove the symmetries in this case. Also, according to our theory and previous works, it is not the orbits that hurt the optimization process but the fixed points introduced by the group projections. Also, as we explained above, your understanding of the role of \theta_0 and the D-norm is incorrect. As addressed in our previous responses, in the standard UV model with weight decay, solutions often converge to low-rank configurations due to the inherent symmetry structure. In contrast, our modifications (\theta_0θ shift and D-norm) alter the geometry of the optimization landscape, effectively preventing convergence to these low-rank, symmetric solutions.

Experimental limitations: Lack of systematic comparison with standard weight decay baseline. Only a few experiments (particularly continual learning) show clear improvements over weight decay. Results suggest their method might be a useful variant of weight decay, but not necessarily for the theoretical reasons provided.

Thank you for your valuable feedback. We believe there may be a misunderstanding regarding our comparisons with standard weight decay. In fact, we systematically compare our method (referred to as syre) with vanilla weight decay across Figures 1, 2, 3, 5, 6, 7, 8, 9, 10, and 11. In these figures, the term "vanilla" refers to models using the standard weight decay with the same \gamma. The results consistently demonstrate that syre outperforms vanilla weight decay across various settings, avoiding low-rank degeneracies and achieving superior performance.

Moreover, as we have addressed in our previous responses, syre effectively breaks different kinds of symmetries. This mechanism prevents convergence to or entrapment at low-rank solutions, leading to the observed full-rank models, which contrasts clearly with the behavior under standard weight decay. Therefore, the improvements are not just incidental but align with our theoretical claims on symmetry removal, as demonstrated empirically in the figures.

Fixed Points vs Symmetries: Do you mean fixed points (invariant sets) when discussing "symmetric points" where P\theta_0=0? If so, consider revising terminology to avoid confusion with symmetry orbits. Why focus on fixed points rather than the full symmetry orbits which characterize the degeneracies in the loss landscape?

Thank you for this question. You are correct that what we refer to as "symmetric points" are indeed invariant sets where $P\theta_0 = 0$. We will stick to the terminology of "fixed point" to keep consistency.

Thank you for the detailed feedback. We will address both the weaknesses and the questions below.

Conceptual confusion about symmetries: The paper discusses "removing symmetries" but primarily analyzes fixed points where P\theta_0=0. This misses the key aspect of symmetries - their group orbits and the associated degeneracies.

Thank you for your question. We believe this criticism is both due to a misunderstanding of our theoretical framework and due to a lack of knowledge of the nature of the fixed points in the UV model (or the matrix factorization model).

We point out that it is a foundational result in the theory of finite groups that any finite group has a fixed point (which can be constructed through the group average $\sum_g g$. E.g., see chapter 2 of: https://www.math.columbia.edu/~khovanov/finite2016/RepTheoryFinite2.pdf). This immediately implies that any group that contains a finite subgroup will also have these fixed points, and thus, removing these fixed points is the same as removing these groups. While the orbits and degeneracies are important, removing the fixed points is sufficient to remove these group structures.

Lastly, it is not about "missing the group orbits" -- the orbits per se do not tend to create optimization entrapment (as you also point out below). It is the fixed points that do lead to these difficulties, and this is why removing fixed points is the focus of our work. See our answers below and references related to this point.

In the simple case of a two-layer linear network F(x)=UVx, the important symmetry structure lies in the GL(h) orbits, not in fixed points.

Unfortunately, we cannot agree that the fixed points are not important. For example, see https://arxiv.org/pdf/1612.09296, which shows that these fixed points are precisely the cause of the saddle points in the loss function, which needs to escape very carefully. From an optimization perspective, this is as important as (if not more important than) the orbits.

Secondly, our focus on the $(\theta', P)$-reflection symmetry addresses a fundamental Z2 group symmetry (and any finite group structures, as Theorem 5 shows). Note that GL(h) symmetries must contain Z2 subgroup symmetries. We can thus remove the GL(h) symmetry by removing the symmetry of its Z2 subgroups. In the context of a two-layer linear network, we argue that breaking these GL(h) symmetries can be systematically approached by first removing the symmetries of its Z2 subgroups. For example, in linear networks with h hidden units, there exist h! permutation symmetries that can be removed together. Each permutation corresponds to a distinct fixed point, and by removing these, we must also remove the GL(h) structure.

Moreover, although the linear network has infinitely many Z2 symmetries, we can effectively remove all of them by the method introduced in Section 5.2. More generally, Theorem 5 implies that we can remove any symmetry which contains a finite group symmetry.

Overly restrictive theoretical framework: The condition P\theta’=\theta’ for (\theta’,P)-reflection symmetry is artificially restrictive. Simple examples show valid reflections that violate this condition. For instance, in 2D with \theta’=(0,1) and reflection in x-direction P=((1,0),(0,1)), we have P\theta’\neq\theta’, yet this is a perfectly valid reflection symmetry.

We are afraid to say that this is an understanding. The $(\theta’,P)$-definition is NOT a restriction but the most general way to define these fixed points/surfaces. To specify a mirror (or any surface), one needs to specify its (1) orientation and (2) location. The $P$ gives the orientation, and $\theta'$ gives the location of this surface.

One can see that when $P$ is a surface, the choice of $\theta'$ is not unique. There exists $\theta'\neq \theta''$ such that $(\theta', P)$ and $(\theta'',P)$ refers to the same symmetry. This is exactly what Proposition 1 proves: If $P \theta' = P \theta''$, then $(\theta', P)$ and $(\theta'', P)$ are referring to the same symmetry! Thus, the requirement that $P\theta' =\theta'$ is not a restriction but is nothing but a notational simplification that is without loss of generality.

To clarify with your example: the reflection you describe in the x-direction corresponds to what we define as a $(0,P)$-reflection, where $P = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$. In this context, our framework would express the reflection as\theta \rightarrow (I - 2P) \theta for any vector \theta.

On the other hand, the authors only very loosely test their explanations in Prop. 2/3. Presumably, if this explanation is correct, shouldn’t models with more/less symmetries be less/more stuck in neuron-collapsed solutions (e.g. in the spirit of the Zivin 2024 experiments)? Fig. 1 only shows one case. If this claim around the science of models is to be taken as a main contribution of the paper, more experimental evidence of the propositions is needed.

Thanks for raising this question. We have added more experiments validating Propositions 3 and 2 (Section C). Here, we plot the rank of the gradient and NTK of a neural network before and after training. The results are shown to agree with the theoretical prediction of Propositions 2 and 3, where the ranks of these matrices decrease as more and more neurons are initialized at a symmetry-entrapped subspace. Also, we would like to point out that it is the case that having more symmetries leads to more entrapment, not less.

Typo on line 65-66: “ |\theta|_D^2=w^TDw is the norm of w with respect to D” should be “|w|_D^2=w^TDw is the norm of w with respect to D”.

Thank you for pointing this out. We have corrected this typo.

How does your syre do on the (Lim 2024) LMC experiments?

Thanks for suggesting this. While we think Lim et al. (2024) have achieved very interesting results in the LMC experiments, we point out that there is really no need nor fair for us to compare the two methods. See our explanation of this point in the summary rebuttal.

The two papers really have different focuses. Our focus is on how to remove an agnostic symmetry. The focus of Lim et al. is on some of the consequences if one removes the permutation symmetry. There is also no need to test our method for the LMC experiments, as our algorithm is not designed for this purpose. The comparison will also not be fair for two reasons. First of all, our method is symmetry agnostic, whereas Lim's method is particularly designed to tackle permutation symmetries. It does not make much sense to compare a general method to a specific technique. This is essentially comparing apples to oranges.

Also, it is very easy to construct cases that Lim et al. (2024) completely fails, while our proposed method can still run robustly. We presented one such example in Figure 4. Another rather extreme but illustrative example is a deep network with width 1. Here, Lim et al. (2024)'s method fails completely (as it simply prevents the whole model from training), whereas our method can still be applied and can work well in principle.

That being said, we do provide one such comparison with the two baselines in Lim et al. (2024) on an MLP network. Our proposed method is shown to work much better than $\sigma$-asym, which also has a single hyperparameter to tune, like our method, and performs similarly to W-asym, which has $8$ hyperparameters to tune for this simple model.

Do you have any (even at a very high level) design rules for how a training-algorithm designer may usefully modulate the symmetries in a model?

Thank you for your question. We suggested a high-level design guideline in the conclusion section. Specifically, models with excessive symmetry may fall into low-capacity traps, while models with too little symmetry may be prone to overfitting. Thus, one possible design logic is to gradually reduce symmetries until the onset of overfitting.

Thank you for the detailed feedback. We will address both the weaknesses and the questions below.

The authors make strong claims around the ability of symmetry-removal to improve performance for a) SSL, b) a supervised continual learning problem, and c) an RL continual learning problem. They do not implement proper baseline comparisons for any of these settings nor go beyond toy experiments.

We believe that these are misinterpretations of our claims. Please first see our explanation of this point in the summary rebuttal. Our actual claims are careful and substantially supported by experiments. The only major claim we have is that removing symmetries can avoid convergence to low-capacity states and nothing more.

We do not claim that the proposed method achieves the SOTA performance of these methods. We only claim that the proposed method removes the symmetries in the loss function and thereby removes the low-capacity solutions -- these claims are justified by the experimental result that rank collapse no longer happens after applying our method. The point of these experiments is to show that the symmetries are indeed removed, as our theory claims.

Also, as there is no method for agnostic symmetry removal at all, there are really no baselines to implement. The proper and primary comparison is the vanilla weight decay because our method can be interpreted as a variant of weight decay. Empirically, we demonstrate that these symmetry removals correlate with improved performance, which is validated with experiments.

If they want to claim that (some degree of) symmetry removal is helpful for training models in practical settings, they need to either show that a) symmetry removal is reasonably competitive in mildly (compute/FLOP-controlled or at least aware) realistic settings or b) symmetry removal explains some % of the success of methods that do well in these settings.

We apologize that we do not fully understand this question. On the compute-FLOP control side of the question, we have included a new experiment showing that the proposed method incurs essentially no overhead over the standard training method in MLP, resnets, and vision transformers. In terms of realistic settings, we have included a new experiment on vision transformers, which shows that the proposed method indeed makes the model much more robust to collapses. Note that the rank collapse problem in transformers has been identified to be due to the symmetries in the attention layer (see https://arxiv.org/abs/2309.16932 and, more recently, https://arxiv.org/abs/2410.23819). Again, the only major claim we have is that removing symmetries can avoid convergence to low-capacity states -- which have been fully experimentally and theoretically justified.

Aside from performance gains, benchmarking their method against the (Lim 2024) methods on linear mode connectivity would also help better validate their set-up. Likewise, benchmarking on other approaches on symmetry-removal.

We are afraid to say that we find this criticism unfair. There is really no method for agnostic symmetry removal, and our method is really the first of its kind. While it is not quite reasonable or fair to compare it with Lim et al., we included a vision transformer training task that we also compared with the methods in Lim et al. in Appendix B.5, and the performance of our methods on linear mode connectivity in Appendix B.7. Our method performs competitively against the methods in Lim et al.

Thank you for the detailed feedback. We will address both the weaknesses and the questions below.

Some of the theoretical explanations are dense and might be challenging for readers not familiar with the detailed aspects of symmetry in neural networks. Having decent knowledge of symmetries in neural networks I still struggled a bit with some sections like Section 5. Summarizing early on what the main theoretical result of each section might make it easier for reader to appreciate the contributions of the paper better.

Thank you for your constructive feedback. In response, we have added brief summaries at the beginning of Sections 5.1, 5.2, and 5.3 to clarify the main theoretical results. For instance, at the start of Section 5.1, we now state: “This subsection will show that when the number of reflection symmetries in the loss function is finite, one can completely remove them using a simple technique,” which highlights the main implication of Theorem 1.

The limitations of the method are not thoroughly discussed, which could be beneficial for practitioners to know when to use this technique with caution.

Thank you for pointing this out. We discussed the limitations briefly in the conclusion section. Specifically, while our method efficiently removes symmetries, it does not always guarantee improved performance, as “a model completely without symmetry may have undesirably high capacity and be more prone to overfitting.” Thus, practitioners should selectively retain beneficial symmetries while removing undesired ones.

Also, another limitation is demonstrated systematically by Figure 4: shifting the initialization creates a small error proportional to the amount of shifting, and thus, there is a tradeoff between how much shifting one wants to achieve and how much error in the objective function one can tolerate. There could be other limitations that we are not yet aware of, and these could be interesting directions for future research.

The formatting of the paper seems off. All the headings are pretty close to the previous paragraph hurting the readability.

Thanks for the criticism. We have fixed the spacing of the headings.

It would be beneficial to see further exploration into the limits of the method, especially in more complex or less controlled environments than those tested. Also, maybe trying some transformer architecture based methods for SSL might make the architecture agnostic point stronger?

Thanks for suggesting this experiment. We have now included an experiment of vision transformers trained on CIFAR10. Our code is adapted from github.com/kentaroy47/vision-transformers-cifar10. Figure 12 in the revision shows that our methods work as expected on transformers, avoiding low-rank structures and achieving higher accuracy.

Could the authors elaborate on any potential computational overhead or scalability issues when implementing syre in larger or more complex models?

Thanks for this question. Theoretically, the method has the same computational complexity as the standard training methods, and so there is no difference in principle.

Practically and empirically, we notice almost zero (<5%) memory overhead over the standard training methods. The reason is perhaps that during training, the majority of the memory is for the dynamic computation of minibatch losses, whereas our method only adds to the static part of training. There is also no significant overhead for computation speed, perhaps for a similar reason. See the newly added experiment in Section B6, which shows that both the computation time overhead and memory overhead are less than 5% of that for vanilla training with a large batch size.

Another major criticism comes from JSoA, which we believe are due to misunderstandings of our contribution and claims.

1a. The claims are too strong. JSoA argues that if we want to make strong claims about the performance, which we did not. Our primary claim is that we proposed the first algorithm that is capable of removing almost all symmetries in a neural network (or in any model), and this is justified in all of our experiments. We then showed that removing these symmetries correlates strongly with improved performances. Removing symmetries per se is an important and fundamental problem, a point acknowledged by reviewer akeV, for example. Our main claim is thus substantially justified by the experimental results and theory. At the same time, we never claimed that our method achieves SOTA performance in these tasks, and we do not think it is fair to criticize us for making strong claims.

1b. Comparison with Lim et al. (2024) and other baselines for symmetry removal. We believe that this is an unfair request for the following three reasons.

(A) The comparison is inappropriate: it is essentially comparing apples to oranges. The method in Lim is a special-purpose method designed for removing permutation symmetry and for improving linear mode connectivity. In comparison, our method is designed to remove any generic symmetry, and its usage is far beyond the case of linear mode connectivity problems.

(B) The comparison is unfair. Our method only has one hyperparameter to tune, whereas the method of Lim et al. requires two hyperparameters per layer (for W-asym). Even in the simplest five-layer MLP experiment, their method involves tuning 10 hyperparameters. A method with many hyperparameters cannot be compared fairly with a method with only one hyperparameter. Lim et al. also proposed a one-hyperparameter alternative ("$\sigma$-asym"), which we show below to not work as well as our method.

(C) There is no other baseline. To the best of our knowledge, our method is the only one that removes almost every symmetry, and there is no other baseline that achieves this functionality. Rejecting our work based on comparisons that our method is not designed for is unreasonable. In fact, the only fair baseline is just the vanilla weight decay method, as our method can be seen as a simple variant of weight decay.

(D) Even if we take a step back, the work of Lim et al. is a contemporaneous work to ours, and the ICLR rule does not require a comparison with Lim et al. The ICLR rule one comparison with previous work states that "We consider papers contemporaneous if they are published within the last four months. That means, since our full paper deadline is October 1, if a paper was published (i.e., at a peer-reviewed venue) on or after July 1, 2024, authors are not required to compare their own work to that paper." (https://iclr.cc/Conferences/2025/ReviewerGuide). The work of Lim et al. will be published in Neurips this year, and the neurips conference only starts in December (and the official publication date is after the conference). Even if we take the neurips author notification date as the publication date, it is still within a month of the ICLR submission deadline. Asking for this comparison is not fair, even according to the submission rules of ICLR.

(E) In spite of all above, we compare syre with Lim et al. (2024) for linear model connectivity and vision transformers. Our experiments suggest that syre works similarly to Lim et al. (2024) for linear model connectivity with much fewer hyperparameters and works better ifor vision transformers.

Lastly, we encourage and welcome the reviewers to raise additional questions, and we are happy to address them accordingly.