The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof

We thank the reviewer for their appreciation of the novelty of our methods, thoroughness of our discussion of previous work, organization of the paper and methodology, and comprehensiveness of our proofs. Also, we thank the reviewer for putting effort into understanding the empirical setup of our work, and making suggestions to improve the $\sigma$-Asymmetric networks.

On Reproducibility: As mentioned in the paper checklist, we plan to open source the data and code to the public, and we are sharing the code with the reviewers now. We have sent the AC our data and code to reproduce the experiments, which they will share with you.

We have also run useful ablations that the reviewer suggested, and found that the results do not significantly change! In fact, some of the reviewer’s suggestions actually improved the results (e.g. shorter warmup period improved W-Asymmetry interpolation while making Git-ReBasin interpolation worse).

“The main issue is that the paper relies heavily on experimental rather than theoretical results, but the strongest results are for a single method (W-asymmetry) with settings much more different than "standard" training … lower learning rates, longer warmup, and larger fixed weights that all differ from standard training by order(s) of magnitude.”

We have rerun our ResNet20 experiment on CIFAR10, and find that using a more standard warmup schedule (1 epoch) yields similar results for linear interpolation. In fact, using a 1 epoch warmup has a lower loss barrier (.691) than the 20 epoch warmup (.931). Performance is also similar on GNNs, see our 1-page results PDF for more.

Our learning rates used for our ResNet training are pretty standard for the Adam optimizer (1e-2 peak). When using 1e-3 learning rate for W-Asymmetric nets, we actually get a lower barrier of $.285 \pm .065$, but test loss that is significantly worse. Perhaps the reviewer is referring to the higher learning rates that people use with SGD (such as 1e-1 in Git-ReBasin), but these learning rates are normal for Adam (e.g. Git-ReBasin uses 1e-3 for Adam on their MLP, and the folklore 3e-4 of Karpathy is common).

Indeed the magnitude of the fixed weights we use are very large but that is by design. We note that we require high magnitude fixed weights in order to effectively break both approximate and exact parameter symmetries. Nevertheless, we agree that the large fixed weights are not ideal for W-Asymmetry. Our work is fairly novel, in that others have not really worked on making Asymmetric networks like this. We envision that future works will develop new types of Asymmetric networks that get around some of these issues.

“There should be a comparison with standard models initialized to the same weights as the W-asymmetric networks…

Standard networks initialized to the same weights have similar performance ($.79 \pm .2$ loss barrier with 1 epoch warmup, versus $.67 \pm .3$ for W-Asym, for ResNet20 on CIFAR-10). We also find that these networks are comparably asymmetric though have slightly worse linear interpolation. We find this not too surprising since the symmetry-breaking weights have such high magnitudes and thus are similar before and after training.

“how are $n_{\mathrm{fix}}$ and $\kappa$ determined for each architecture? … what do the names of the blocks refer to? There are not enough for a 1-to-1 mapping onto ResNet20 layers or permutations, so more explanation is needed.”

Our theory suggests that $n_{\mathrm{fix}}$ should be at least $\log_2(dim)$ so that the conditions in Theorem 3 hold. The exact values of $n_{\mathrm{fix}}$ and $\kappa$ are chosen manually (these are hyperparameters).

The ResNet20 code we use is in lmc/models/models_resnet.py in our attached code. It consists of an initial convolution followed by 3 blocks. Each block contains 6 convolutions (and 36 convolutions for ResNet110). The parameters we give in Tables 6 and 7 apply to each convolution of the described block.

“Are the results in table 1 for networks with different random initializations, or the same identical initialization? … are the masks and fixed weights different or the same between the two networks? ...”

The learnable parameters are from different random initializations, but the masks and fixed weights are the same between the two networks. See also the general comment for more on this.

“... If the fixed linear transform F in $\sigma$-asymmetry is similarly shifted in distribution, would that make its results closer to W-asymmetry?”

Good question. We have tried this as well (as noted in the paper, we tune the standard deviation that $\mathbf{F}$ is drawn from for $\sigma$-Asymmetry as well), but this was not sufficient.

“Why not fix biases to get asymmetry instead?” We tried this and it didn’t work. See the results in our results PDF. Intuitively, there is only one dimension for the biases to vary, so for instance if you sort the biases, then two neighboring biases may be very close to each other. This causes two neurons to be approximately automorphic, so there are still approximate permutation symmetries.

“for $\sigma$-asymmetry … I wonder if the network's activations mainly stay in the approximately linear region of the sigmoid … Some other comments or experiments on why $\sigma$-asymmetry is less effective in practice would be helpful …”

We thank the reviewer for this suggestion. This is a plausible cause, but we investigated it, and it does not seem like this is the main issue. For instance, to make the nonlinearity more nonlinear, we switched to cosine nonlinearity instead of sigmoid, but this did not work well either. Also, we have already tried changing the magnitude of the fixed weights (via tuning $\kappa$), to move the activations into different parts of the sigmoid. See general comment for more.

We thank the reviewer for appreciating the novelty and evaluations behind our work, and for their comments, which we now address one at at time:

“The number of learnable parameters of the standard NN and the asymmetric NN is reported only for the metanetwork task. It should also be reported for the LMC, BMA, and MLI tasks.”

Good point, we have now added the number of learnable parameters for the other experiments to our paper. They are in this table (note that sigma-Asym networks always have the same number of learnable parameters as the corresponding standard network).

We also now report the number of learned parameters for metanetworks in Section 5.3 (not the input data) as follows:

“Comprehensible reason is required for the proposition “the posterior will have less modes” in the hypothesis of the BMA task in order to understand the purpose of the BMA task. No symmetry does not imply less modes… “

“The BMA and MLI tasks are based on the strong assumption, where its evidence is absent, that no symmetry induces the reduced number of modes.”

We gave several citations in that section (5.2) on previous works noting that symmetries induce problematic modes in Bayesian NNs (e.g. [2, 33, 71, 49, 70, 35]).

The argument is simple: if $\tau$ is a parameter symmetry, then $L(\theta) = L(\tau(\theta))$ for loss functions $L$ such as NLL and parameters $\theta$ (because the neural network function is left unchanged by $\tau$, and the loss only depends on the neural network function). Thus, for any mode $\theta^* \in \mathrm{argmin}_{\theta} L(\theta)$, we have that $\tau(\theta^*)$ is also a mode, because it has the same loss value. Removing parameter symmetries $\tau$ also removes these additional modes. We will spell out this argument in the revision.

Also, the empirical results do not rely on these “assumptions”. But rather, previous work shows that we expect these assumptions to hold, so we empirically investigate these hypotheses via Asymmetric Networks.

“What do you mean by “loss” in BNN. Is it an ELBO? Why don’t you report the negative log-likelihood (NLL)?”

We do mean NLL loss. We will make this clearer in the revision.

“Fixing some weights or adding anisotropic activation function implies constraining not only symmetry but also distorted parameter manifold (like equivariant NNs) that also leads to increasing correlation between solutions. The asymmetric NN is limited to directly correspond to the standard networks with a fixed permutation.”

Regarding the comparison to equivariant NNs, coefficients for a steerable basis in an equivariant network is probably more distorting of the parameter manifold. Yes, we do distort the parameter manifold, but we do so in a way that maintains the vector space structure and calculus, so we can optimize Asymmetric Networks with standard gradient-based methods like Adam. In contrast, some previous works restrict parameters to nonlinear manifolds, and require different optimization algorithms such as projected gradient descent or Riemannian optimization methods. See the general comment for more information, as well as our related work section.

“The asymmetric NN is limited to directly correspond to the standard networks with a fixed permutation.” We are not sure what you mean here, but this is an important positive property of asymmetric networks. Since they are like standard networks with a fixed permutation, we do not need to account for parameter symmetries when doing interpolation, metanetwork processing, or Bayesian NN training for asymmetric networks.

“Although the paper partially gives insights on the loss surface, it does not suggest a new practical method utilizing the insights.”

This is not quite the purpose of our paper; we moreso seek to understand various phenomena in deep learning. Some phenomena like Monotonic Linear Interpolation (MLI) have no real applications, and some others like Linear Mode Connectivity (LMC) have related downstream applications in things like model merging and federated model averaging.

Moreover, the BNN results are in some sense a practical method for improving training of BNNs.

That being said, we do believe there are some other potential practical methods inspired by our work, which could be explored in future work. For instance, model merging is extremely powerful for open-weight large language models (see top methods on open LLM leaderboard), and symmetries have not been explored as much there. Our Asymmetry methods cause little overhead (only modifying weights and/or nonlinearities a little), so they could be used for very large models.

We thank the reviewer for appreciating the novelty of our ideas, the effectiveness of our asymmetric parameterizations, the wide range of problems that we consider, and our writing. We think that we have improved our work through your suggested clarifications and ablations.

“Even though I find the idea of studying the behavior of asymmetric neural networks interesting, it is not clear to me how this perspective is beneficial compared to the constrained optimization one…”

This is indeed important, see general comment for our elaboration on this.

“It is not clear from the paper if W-asymmetric and $\sigma$-asymmetric networks are fully asymmetric or just have fewer symmetries in practical setups. A more accurate discussion on which symmetries are removed by the proposed methods in the experimental setups should be added to the paper.”

Good point. We will more clearly explain this. Our theoretical results are summarized in this table:

Scale symmetries are removed by FiGLU, as shown in Proposition 2. Although it is not formally proven that W-Asym networks remove scale symmetries, we believe that they do (intuitively, the fixed weights also fix a scale).

“For example, the paper does not cover the normalization layer symmetries, even though layer and batch normalization are used in the experiments. It seems W-asymmetric networks remove normalization symmetries, while $\sigma$-asymmetric ones do not. It may be one of the reasons why $\sigma$-asymmetric networks show weaker results in linear mode connectivity and linear interpolation sections. The ReLU scale symmetry is also not adequately discussed.“

This is a good point. We agree that the W-Asymmetric networks appear to remove normalization symmetries, whereas the $\sigma$-Asymmetric ones do not in general.

The ReLU scale symmetry can be handled by changing the nonlinearity to e.g. GELU (Godfrey et al. [19] prove that this does not have scale symmetries). Also, our ResNet experiments still use ReLU nonlinearity, yet they achieve good symmetry breaking.

“The analysis of the $\sigma$-asymmetric networks is limited. There is no proof or discussion of the universal approximation for this method…”

We did try to prove universal approximation for $\sigma$-Asymmetric networks before submission, but we could not do it for a few reasons; we will add discussion of this to the paper. Classical universal approximation results with MLPs do not apply, because those generally assume elementwise nonlinearities. We do think there is a potential proof via related constructions to the proof of the W-Asym universality and interesting symmetries of SiLU-type nonlinearities [Martinelli et al. 2023], but we leave this to future work.

[Martinelli et al. 2023] Expand-and-Cluster: Parameter Recovery of Neural Networks. https://arxiv.org/abs/2304.12794

“The analysis of the $\sigma$-asymmetric networks is limited… The paper does not explain poor results on linear mode connectivity and linear interpolation and does not include the results on Bayesian neural networks and metanetworks. A more detailed analysis of the $\sigma$-asymmetric networks, or at least a discussion of its suboptimal results, would benefit the paper.”

Indeed. See our general comment for more on this. We will add more discussion on this interesting point to our paper.

“In the Bayesian neural network experiment, the W-asymmetric networks have fewer parameters than standard ones, which may influence the optimal training hyperparameters. Hence, it may be the case that the difference in the results is due to, e.g., different effective learning rates and not the asymmetric network structure.”

Good point. We have rerun the experiments, with 90% shallower standard ResNets to match the number of parameters of the W-Asymmetric networks, and we find essentially the same performance: W-Asymmetric test accuracy ($49.3\pm .4$) is still substantially better ($46.8 \pm.9$ for standard and $46.5\pm1.1$ for the shallower one). See our 1-page results PDF for more.

“In the metanetwork experiments, standard and W-asymmetric networks have different numbers of parameters. Hence, it may be the case that the difference in the results is due to the different dimensionality of the input space and different optimal training hyperparameters for metanetworks and not the asymmetric network structure.”

Again, good point. We have trained a whole new dataset of smaller standard networks, of the same number of parameters as the W-Asymmetric networks (~ 60,000). There is little to no change in the results of these networks and our previous dataset of standard networks, so the difference in results seems to be from the asymmetric network structure. We will include these results in the revision; see our 1-page results PDF for full results.

“Do I understand correctly that the standard deviation $\kappa$ … is a hyperparameter tuned separately…”

Yes, you are correct. We previously said “standard deviation $\kappa$ > 0 that we tune”, but we will note this more clearly.

We are glad that the reviewer appreciates the writing, topic, and experiments of the paper, and we appreciate the reviewer’s comments. Below, we address them one-by-one.

“A major issue about this paper is that some findings are not new. Especially, some studies have already demonstrated that asymmetric networks are more easily to satisfy LMC. [Cite 1] showed that pruning solutions are within the same basin and pruned networks can be viewed as a special case of W-asymmetric networks …“

We respectfully disagree. The citation refers to methods for pruning a standard neural net, which are very different from our W-Asymmetric nets. The pruning methods for standard neural networks require specialized training algorithms that differ significantly from standard training (e.g. lottery tickets require repeated training and resetting, dynamic sparse training requires updating connectivity during training).

As noted also by reviewers ceUs and tX9W, our methods are novel. Unliked pruned networks, our W-Asymmetric networks have a fixed sparsity. Thus, they can be trained by standard training algorithms like Adam, just like standard neural networks. This is crucial, since we want our Asymmetric networks to be as similar as standard networks as possible, so that we can gain insights into standard networks. See more on this important point (importance of using standard optimization algorithms) in the general comment.

“... in experimental part, the four hypotheses seemingly do not relate to each other. There is not a unified conclusion drawn from the experiments.”

This is true, but we do not see this as a downside. In fact, the point of our work is to provide a tool (analysis of asymmetric networks) for us and future works to study the effects of parameter symmetries in many different phenomena at once.

Also, there are some higher-level conclusions that we will emphasize more in the revision. For instance, asymmetric network loss landscapes are somewhat more well-behaved and similar to convex loss landscapes than standard networks, and we show that understanding aspects of neural network optimization requires consideration of parameter symmetries.

“The most important question about the permutation symmetry remains unresolved. As permutation symmetry holds in most neural network architectures, should we remove permutation symmetry or not? Is the success of deep learning related to the permutation symmetry? This paper seemingly cannot give an insightful answer.”

This is not the point of our work. We probe the effect of parameter symmetries (not just permutation symmetries) in already many different phenomena in deep learning. For instance, such symmetries should be accounted for when merging models, processing models with metanetworks, or training Bayesian neural networks.

“In Sec. 5, despite each experiment are motivated from some simple intuition, there is no rigorous theoretical foundation for each hypothesis, which could potentially lower the value of this study.“

As mentioned in our abstract, “theoretical analysis of the relationship between parameter space symmetries and these phenomena is difficult.” As in most of the study of deep learning theory, it is very difficult and sometimes intractable to theoretically analyze the effects of symmetries in all of these domains. Even when some theory can be done, it is usually done in very restricted settings (e.g. infinite width, one or two-layer networks, optimization assumed to reach a global optima), which are very unrealistic.

We can consider examples from the literature of theoretical analysis of these hypotheses. In [Ferbach et al. 2024], the authors can only theoretically prove linear mode connectivity up to permutations for the (unrealistic) cases of two-layer mean-field MLPs or untrained MLPs. In [Kurle et al. 2022], parameter symmetries in Bayesian learning are theoretically analyzed for a linear model trained on one datapoint.

The point of our paper is to analyze the effects of symmetries in many domains at once (including some that are not covered in our paper) via empirical studies, which have arguably been more impactful in the study of deep learning. Many other works study deep learning phenomena in a purely empirical way, and they give inspiration for future theory; these include seminal works like those on the Lottery Ticket Hypothesis [Frankle & Carbin 2018], permutation matching for merging [Ainsworth et al. 2022], and early empirical investigations into neural networks [Goodfellow et al. 2014].

References
[Ferbach et al. 2024] Proving linear mode connectivity of neural networks via optimal transport.
[Kurle et al. 2022] On the detrimental effect of invariances in the likelihood for variational inference.
[Frankle & Carbin 2018] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.
[Ainsworth et al. 2022] Git Re-Basin: Merging Models modulo Permutation Symmetries
[Goodfellow et al. 2014] Qualitatively characterizing neural network optimization problems.