Understanding Mode Connectivity via Parameter Space Symmetry

We thank the reviewer for their encouraging feedback. We appreciate that they have taken the time to read our proofs. We also appreciate the valuable questions and the many relevant pointers to related work, which we discuss below.

Relation to broader scientific literature

It would be interesting to discuss how the studied symmetries of the general linear group relate to the symmetries discussed by Petzka et al. [1].

Petzka et al. show that permutation and rescaling are the only transformations that leave a 2-layer ReLU network unchanged, excluding degenerate cases. These symmetries are subgroups of the general linear symmetry group studied in our work. They also identify degenerate-case transformations not present in linear networks, when neurons share identical zero hyperplanes.

How do the loss-preserving orbits discussed in this paper relate to the minima manifold in [2]?

Both our paper and Simsek et al. [2] analyze symmetry-induced structures in the loss landscape. Our orbits arise from continuous symmetry groups (e.g., GL(n), rescalings), while their minima manifold arises by embedding smaller networks into overparameterized ones via permutations and neuron duplications. These manifolds can be viewed as unions of affine subspaces, and our orbits form structured subsets within it. The two views are complementary and compatible, especially in linear networks with full-rank weights.

This paper discusses linear reparameterizations. How does it relate to works on non-linear reparameterizations using Riemannian geometry [3]?

Kristiadi et al. [3] study reparameterization as a map from the parameter space to a new space, while we focus on symmetry that maps from the parameter space to itself. Both maps can be nonlinear. Studying symmetry-induced orbits from a Riemannian perspective could be a promising direction for future work.

We will include a brief discussion of these connections in the final version of the paper.

Limitation of the analysis

most of the analysis is for linear neural networks and thus the applicability to modern deep learning is limited. While a detailed empirical analysis of this difference might be out of scope, a discussion of these limitations seems appropriate.

We appreciate the suggestion and agree that a discussion would be valuable. Our results are derived under architectural assumptions that enable precise, mathematically grounded insights into how continuous symmetries shape the loss landscape.

While the findings may not directly transfer to all architectures, our methods are adaptable. For example, Section 5.3 applies to any network with layers involving multiplication of two weight matrices–a pattern that appears, for example, in transformer blocks. In higher-dimensional, non full-rank settings, connectivity depends on how orbits defined by different rank combinations intersect. Extensions to nonlinear networks are possible by identifying approximate symmetries (as in Section 6), or by continuously deforming the minima and studying its behavior in the limit as the network approaches a linear regime.Even partial or approximate symmetry can provide useful structural information about the minima and support new applications.

Other comments and questions

Would, for example the proposed relative flatness [5] be invariant under the reparameterizations discussed in this paper?

Yes, in many cases. The relative flatness measure proposed in Petzka et al. [5] is designed to be invariant under layer-wise and neuron-wise linear reparameterizations, such as rescaling and certain changes of basis. These are precisely the types of continuous architectural symmetries we analyze in our paper. Therefore, for the linear and homogeneous network settings considered in our work, the relative flatness measure would indeed remain invariant under the reparameterizations we describe.

Prop. 5.4: in the equivalence of weights $W_i$ and $W_i'$ at the end of the paragraph, the permutation $P$ seems to be missing.

This is indeed a typo. We will add the permutations. Thank you for the detailed read.

Q1: Cor. 4.2 shows that the number of connected components increases with depth, whereas width plays no role. This is interesting. Can you comment on potential implications for neural network training? It seems related to the fact that deeper networks can be trained easier than wide and shallow ones.

We do not have concrete results on whether more connected components correlates with easier training, but below we share some speculations. A larger number of connected components could mean the minima is distributed more widely in the parameter space, which makes them easier to find during training. However, an exact analysis would require quantifying the area (or volume) and the exact distribution of the minima, as well as the geometry in the surrounding region. Nevertheless, connecting connectedness and optimization seems to be an exciting line of future work.

We thank the reviewer for their comments. Below, we address concerns about the scope and generalizability of our results, and clarify how our approach can be extended.

Scope and generalizability of results

Sometimes the scope of the findings is inaccurate when a high-level picture is given…

We will clarify the intended scope to reduce ambiguity. For higher-dimensional invertible weights in the skip-connection example, the number of components reduces further to 2. We are extending the analysis by relaxing the invertibility condition and will include full proofs in the final version.

The investigation relies on the symmetries of simplified models of neural networks… This limits the scope of finding to toy settings where we know for a fact the conclusions break down when we move to a simple MLP.

While our analysis focuses on simplified models, it does not fundamentally break down for more complex architectures. Many modern architectures retain large spaces of continuous symmetry—for example, our results in Section 5.3 apply broadly to layers involving matrix multiplication, such as transformer blocks. When a homeomorphism exists between the symmetry group and the set of minima, topological properties such as connectedness can be inferred directly. In non-invertible settings, such as with skip connections, connectivity can still be analyzed through how orbits intersect, albeit with more care.

Results rely on strong assumptions, so much so that unfortunately conclusions tell us nothing about realistic neural networks. …one can be sure that conclusions do not transfer to more complicated architectures, since these symmetries do not exist.

We respectfully disagree that our findings are irrelevant to realistic neural networks. We do not claim direct applicability to all architectures; rather, we aim to isolate the role of symmetry in shaping the loss landscape. Our simplified models allow for precise, mathematically grounded insights into how symmetries–especially continuous ones–can yield connected minima and help explain phenomena like linear mode connectivity.

As explained above, our methods provide a framework that can be adapted to a wide range of architectures. Even when we do not know the full set of symmetry, a subset of symmetry or even approximate symmetry can still give useful information on the structures of minima and inform new applications, as demonstrated in section 6.

The conclusions made about the role of symmetry in this paper completely ignores (and eliminates) the effect of overparameterization symmetry…

Overparameterization increases the dimension of the minima set, reflecting redundancy or flatness, but does not induce symmetry groups (e.g., permutations or rescalings) in the group-theoretic sense like the architectural symmetries. Thus, our results neither overlook nor contradict existing theory on overparameterization. Exploring interactions between high-dimensional minima and architectural symmetries remains an interesting direction for future work, and our framework may help analyze such cases.

Could you please explain if you expect any of the findings to “approximately” translate to a realistic/practical setting of an MLP, or even a linear neural network in the overparameterized regime?

Our main multilayer setting (Equation (1)) already allows for overparameterization. While not all MLPs will exhibit the same topology as our simplified models, we believe our approach is generalizable. For higher-dimensional weights, connectivity analysis involves characterizing intersections among multiple orbits defined by distinct rank combinations. Extending our analysis to include nonlinearity could involve approximate symmetries, as in Section 6, or by continuously deforming the minima and studying its behavior in the limit as the network approaches a linearity.

Writing and presentation

The paper reads like a collection of loosely related results; it includes different toy models used to study different phenomena.

The paper's central theme is that parameter space symmetries, especially continuous ones, can help explain why and how minima are connected. Each model is chosen to isolate a specific aspect of this symmetry-connectivity relationship. Collectively, they build a coherent picture of how architectural symmetries shape the loss landscape and empirical behaviors observed in deep learning.

I got lost at the beginning of section 6... It would be nice to clarify the material of that section…

We will revise this section to clarify and better motivate the use of one-parameter subgroups. The transformation in Equation 6 is not arbitrary–it is specifically constructed to follow a one-parameter subgroup and defines a smooth curve within the minima. The section analyzes the curvature of such symmetry-induced curves to understand when linear mode connectivity approximately holds. Proposition 6.1 formalizes this by relating interpolation loss to curvature.

We thank the reviewer for their encouraging feedback and insights on our work’s relation to broader literature. We will incorporate their suggestions into the final version of the paper.

Some well-known simple things are presented as Propositions (for ex. Propositions 3.4 and 3.5). These are not particular contributions of this paper hence should not be presented as Propositions.

Section 3 is intended as a background section, and it is not our intention to present these propositions as novel results. We chose to include them in proposition form primarily for completeness and ease of referencing them in proofs. We agree they should not be positioned as contributions of this paper, and we will revise the paper to clarify this.

ResNet 1D section should not be called that as that architecture has no non-linearity.

Thank you for pointing this out. We will change the terminology to skip-connection to avoid ambiguity.

Comment: Proposition 5.2 uses permutations to connect components for a deep ---linear--- network. However, the relevant symmetry group here is $GL_n$ as those matrices can be multiplied by any invertible matrix without changing the loss. In other words, symmetries of deep linear networks have more ---volume--- compared to the deep non-linear networks.

Please add a comment along these lines so that the readers are aware of this distinction between linear and non-linear networks.

We appreciate this observation and will include a comment along these lines. While permutations are sufficient to connect components in deep linear networks, the full symmetry group in this case is indeed $GL_n$. Also, it is indeed true that deep linear networks have larger symmetry groups than deep non-linear networks.

We thank the reviewer for their constructive feedback and insightful questions. We are encouraged by the recognition of our paper’s novelty, rigor, and intuition. Below we expand on the practical takeaways and respond to the questions.

Practical takeaways

Our work shows that parameter space symmetries, especially continuous ones, explain why and how minima are connected. For practitioners, these results motivate concrete strategies – and cautions – for tasks that navigate the loss landscape, including model merging, ensembling, and fine-tuning:

• One can build low-loss curves explicitly using known parameter symmetries. This gives a principled and efficient way to obtain new minima from old ones, potentially useful in (1) generating model ensembles with low cost; (2) improving model alignment by allowing a much larger group of transformations than permutation; and (3) mitigating catastrophic forgetting in fine-tuning by constraining updates to remain on the symmetry-induced manifold of the pretraining minimum.

• The connectedness of minima supports the practice of model merging and ensembling, even when models are trained separately. In addition to permutation, many other symmetry transformations can connect solutions that would otherwise appear very different.

• Linear interpolation between minima is not guaranteed to lead to better models, despite its widespread use. This highlights the need to evaluate whether the minima found by specific learning algorithms are approximately connected before averaging models directly.

Finally, for theorists working in this area, shifting focus from viewing loss landscapes as chaotic to structured, symmetry-rich spaces invites new mathematical tools to explain empirical observations in deep learning. This has led to new intuitions ranging from why residual connections lead to more connected minima, why linear model connectivity can fail, to why model averaging often works when the models are not too far from each other. We also hope that the broader approach – inferring properties of unknown solution sets from known symmetry groups – could inspire new work beyond this field.

Response to questions

Is it clear that having a lower number of connected components makes for easier connectivity?

We agree with the reviewer that connectedness alone does not imply easy connectivity in the sense of short or simple paths between solutions. Being in the same connected components is a necessary condition for connectivity, but a single component may still contain complex geometry necessitating complicated connecting paths.

Defining the ease of connectivity is subtle, and we agree that a discussion would be valuable. One natural measure is the parametric complexity of the connecting curves, quantifiable by their degree if polynomial, or number of segments if piece-wise. Another possible definition for easy connectivity would be low curvature of the minimum manifold or short geodesic distance between two points on it. As we discussed in Section 6.2, low curvature implies that linear interpolation stays near the manifold. Other potential definitions include whether the connecting curve has an analytical expression, or how many points are needed to approximate it within a certain error. It would indeed be interesting to examine these properties for symmetry-induced connecting curves.

How do you reconcile the failure case of multi-layers with the empirically observed largely well-connected paths for deep networks?

The empirical observation of mode connectivity and linear mode connectivity is likely due to the fact that certain family of optimizers, especially stochastic gradient descent (SGD), tend to explore only a subset of the minimum, a phenomenon often referred to as implicit bias. Regularization techniques and weight decay may further encourage SGD to favor certain regions of the minimum. As a result, the subset of minima that is likely to be reached by SGD can have very different structures and perhaps more connected than the full set of minima.

I am curious what are the nature of theoretical bottlenecks that impede a general analysis for ResNets, besides the general case.

Extending our proof to a general analysis is challenging because the number of orbits grows rapidly with the dimension of the weight matrices. In the 1D case, the minimum consists of only two orbits, so the proof only requires analyzing the connected components of each and checking how they intersect. For higher-dimensional weights, especially when not full-rank, each distinct rank combination defines multiple orbits, and characterizing the intersections among these many orbits, while possible, becomes combinatorially complex. We therefore chose to include only the 1D ResNet example to demonstrate the proof technique. It is possible that future work with more refined tools could extend this analysis to higher-dimensional settings more efficiently.