Understanding Mode Connectivity via Parameter Space Symmetry

Response to questions

1 - The networks analyzed are invertible up to the output layer, meaning the output dimension matches the input dimension. How strictly is this condition required? Does switching to a one-dimensional output immediately yield negative results, as suggested in Section 5.2?

The invertibility condition is required when we want to establish a homeomorphism between the minimum and the symmetry group. When there is one, we can easily infer topological properties of the minimum from the symmetry group. When the network is not invertible, as in the example with skip connections, we are still able to analyze the connectedness of the minimum, but this requires more careful handling of analyzing multiple orbits. Switching to a one-dimensional output may change the number of connected components of the minimum, although the direction of change may depend on the exact loss function (Proposition A.8). There does not seem to be a connection between this change and the failure cases of linear mode connectivity in Section 5.2, which is primarily caused by the non-compact symmetry group.

2 - According to [3], layer-wise mode connectivity is achievable. Does Proposition 5.3 contradict this result, or is there a possible connection?

[3] Adilova, Linara, Asja Fischer, and Martin Jaggi. "Layer-wise linear mode connectivity."

Proposition 5.3 does not contradict with the layer-wise connectivity result in [3]. In the proof, we construct the two minima $W, W’$ by rescaling two layers. As a result, both layers are different between $W$ and $W’$. This is different from the setting in Theorem 4.1 in [3], where only one layer is different between the two sets of parameters. The empirical observations of the connectivity of certain groups of layers in [3] may reflect the implicit bias of SGD, which means it is possible that the minima reachable by SGD are approximately linearly connected, even though the complete set of minima may have more complex structures. We appreciate the pointer to this relevant work and have added a brief discussion in the updated paper.

3 - Is proposition about residual connections restricted to 1 dimension?

Yes, as mentioned in the proposition ($n=1$) and subsection header. When the weight matrices are higher-dimensional invertible matrices, the number of connected components is further reduced to 2. We are working on relaxing the invertibility condition and will include full proofs in the final version of the paper.

Thank you for your comments! We are encouraged that you consider our framework powerful for analyzing the loss surface. We address your comments and questions below.

However, in its current form, the paper does not fully clarify the connection between topological concepts and the loss landscape of deep neural networks. While it opens with a detailed and precise introduction to topological concepts, it then directly applies these concepts to loss surfaces and networks without discussing necessary assumptions (such as the invertibility of all networks considered). Additionally, there is little exploration of how the topological concept of connected components relates to the depth of the network or which elements correspond to orbits or groups, potentially with examples. Although all this does not detract from the technical accomplishments, it may reduce the paper’s impact by making it difficult for an unprepared reader to link the framework to neural network applications.

We appreciate your suggestions on clarifying the connection to topological concepts. We have attempted to make the assumptions clear by stating them before each proposition as well as the subsection headers. Since the topological concepts from the preliminary section are mostly used in proofs, we did not reference them often in the main text. However, we have made an effort to explain the topological intuitions in the proof sketch and explanation of the theorems. We also hope the last two corollaries in Section 3 provide some correspondence, or at least a hint of connection, between the topological concepts and elements in neural networks. We will expand Section 3 by including more connection to neural networks if space permits.

In the final section, the concept of curvature is used without clearly defining it in this context. Further, this section could connect more directly to practical applications by demonstrating empirical curves that connect modes and align with derived formulas (e.g., regarding loss growth). A similar issue is in Section 5 concerning symmetries.

Thank you for these suggestions. In the final section, we have added a formal definition of curvature, as well as experiments showing that the loss on the curves induced by approximate symmetry is consistently low, as predicted by Proposition 6.1 (Figure 3b,c). For Section 5, we added a visualization showing that the loss barrier on a linear interpolation between two minima in a homogeneous network can become unbounded, as predicted by Proposition 5.3 (Figure 4 in Appendix C).

Minor Issues.

Thank you for pointing these out. We have added the reference, corrected the notation for the input space of $L$ in Section 3.3, and replaced \citet with \citep where appropriate.

Thank you for your comments and positive feedback!

Lack of limitations: Since the homeomorphism is specifically tailored to linear regression models, it is necessary to clearly state this limitation in the introduction.

Thank you for the suggestion. We have made the limitations explicit in the introduction. However, we would like to point out that while our examples of homeomorphism between the minimum and the symmetry group are limited to linear regression models, our framework can be applied to networks without a homeomorphism and different loss functions. For example, when the minimum comprises more than one orbits, we can still obtain the number of components by analyzing the connectedness of each orbit. Our method can also be generalized to loss functions other than the mean square loss.

Response to Questions

1. Is it possible to experimentally validate the results of Sec. 6? Can we confirm that the valleys of the loss lines predicted by Eq. 7 correspond to the valleys in the two-dimensional heatmap of the loss landscape?

Yes, we have added experiments showing that the inequality in Proposition 6.1 holds empirically (Figure 3a), and the loss on the curves induced by approximate symmetry is consistently low as predicted by Proposition 6.1 (Figure 3b,c). Since these curves live in a high dimensional space, it is not straightforward to produce a two-dimensional heatmap. Nevertheless, we hope it suffices to show that the loss at every point of the curve is low compared to the loss on linear interpolations between two minima.

2. Is it possible to extend the analysis to more realistic models such as ResNet, which has a softmax function, using parameter symmetry and homeomorphic mapping?

We believe it is possible to extend our results to a network with a softmax function. Softmax is known to have a translational symmetry, which means that points on the minimum can have different network outputs before applying the softmax while giving the same final output after the softmax. For each possible network output, the connectedness of the set of minima corresponding to that output can be analyzed by methods from our paper. The connectedness of the union of these sets, or the entire minimum, can then be obtained by analyzing the connectedness of the set of outputs that map to the same value after softmax. We will include a precise formulation and full proofs in the final version of the paper.

Thank you for your feedback and comments. You are right that this paper is held together by group actions - our main message is precisely that we can infer topological properties of the minimum from topological properties of symmetry groups. This connection, while obvious in hindsight, has been overlooked by mode connectivity researchers for years. We hope that our insights will help future research on loss landscapes. We also hope that the idea of inferring properties of an unknown object from a known one could inspire new work beyond this field.

The results are not very strong or interesting. Many standard basic results of math are presented like big theorems. Overall, this article does not make things simpler or clearer.

We appreciate your perspective, although we believe our results are novel and, according to other reviewers, will be of interest to the field. It is not our intention to make our theorems look like big results - we value simplicity over complexity. Our goal is to introduce new intuitions behind why and when mode connectivity holds and would appreciate concrete suggestions on how to make the presentation clearer.

Is there any reason to believe that there is a natural group action like the ones you explain beyond the cases where it is intuitively clear that there is one?

Yes, parameter space symmetry is prevalent in common architectures, and there exists complicated and possibly data-dependent symmetry group actions [1]. The high-dimensional nature of the minimum [2] also suggests possible group actions with nontrivial orbits. The existence and number of symmetries in neural network architectures is an active field. Recent work has also found symmetry groups and actions with an automated framework [3].

Can simulations reveal the presence of group actions or the relevance of your derivations beyond the obvious cases?

It is not clear whether simulation could reveal the presence of group actions, but other approaches, such as the learning-based symmetry discovery method in [3], have shown that there exist non-obvious parameter symmetries. Our paper complements these works by providing an application for the discovered symmetries.

References:

[1] Zhao, Ganev, Walters, Yu, Dehmamy. Symmetries, flat minima, and the conserved quantities of gradient flow. arXiv preprint arXiv:2210.17216, 2022.

[2] Cooper. The loss landscape of overparameterized neural networks. arXiv preprint arXiv:1804.10200, 2018.

[3] Zhao, Dehmamy, Walters, Yu. Symmetry Discovery in Neural Network Parameter Spaces. UniReps 2024.

Thank you for your comments and positive feedback!

I see that there is no dependence on width of the network when considering the number of connected components, however permutation symmetries grow exponentially with the width of the network. This appears to be due to the fact we are studying a very simplified setting of linear networks. It’ll be useful for the readers to include a discussion on this.

The lack of dependence of the number of connected components on width is a result of the fact that the set of $n$ by $n$ invertible matrices ($GL_n(\mathbb{R})$) has two connected components independent of $n$. Therefore, although wider networks have a larger symmetry group and a larger set of minima, the number of connected components remains unchanged. This is one example where connecting the minimum to symmetry groups brings out simple yet otherwise non-obvious results. We appreciate this question and have added a short discussion in Section 4.

It’ll improve the paper further to improve an example / figure for section 5.1 where permutations leads to mode connectivity.

Thank you for the suggestion. We have added an example in Appendix C.

It is well known that scale symmetries lead to a failure of linear mode connectivity however it is interesting that controlling the weight norms and control over the curvature leads to approximate linear connectivity.

How does this relate to empirical solutions explored by SGD? Specially because it appears that weight decay is necessary for lmc mod permutations.

This is an interesting observation, and we have included a short discussion at the end of Section 5.2. The empirical observation of mode connectivity and linear mode connectivity is likely due to the fact SGD typically only explores certain parts of the minimum, often referred to as implicit bias. Weight decay may further encourage SGD to favor certain minimizers. The subset of minima that is likely to be reached by SGD can therefore have very different structures than the entire set of minima.

Thank you for your detailed comments and positive feedback!

1. What does it mean intuitively and geometrically if two network parameters are in the same connected component? As the authors point out, this does not imply being path-connected, so while it sounds convincing at first, it is actually not clear to me how this notion ties back to the usual geometric understanding of connectivity used in deep learning. I would appreciate if the authors could clarify things. I.e. how pathological are counter-examples of “connected but not path-connected”?

Intuitively, imagine the minimum of a loss function as a manifold or a high dimensional surface. Then two network parameters are in the same connected component if they reside on the same piece of this manifold. Connectedness ensures there is no separation of the space into disjoint non-empty open subsets, while path-connectedness allows one to construct continuous paths between points.

While it is theoretically possible for two points in the same connected component to lack a path between them, such counterexamples are often specifically constructed and unlikely to be encountered in the context of deep learning. A classic example is the topologist’s sine curve $T = T_0 \cup T_+$, where $T_0=\{(x, y): x=0 \text{ and } y \in [-1,1]\}$ and $T_+ = \{(x, y): x \in (0, 2/\pi] \text{ and } y = \sin(1/x)\}$. This space is connected but not path-connected since the infinitely oscillating waves prevent any continuous path from linking $T_+$ to $T_0$.

2. One weakness in the “large barrier” type of results is their global nature, i.e. there is no notion of what types of minima SGD actually finds. The counter-example (as far as I understood) for which a path is constructed for Prop 5.3 and 5.4 starts from a set of parameters and then constructs a new one using the rescaling symmetry. It is not clear to me how “degenerate” these solutions are in the sense that SGD might never choose them due to its implicit bias. I believe there are actually results that show that SGD prefers certain parameters out of the re-scale orbit. In general I think it would be important to better highlight that this work deals with the loss landscape in a global sense, and is not restricted to the minimizers discovered by SGD.

We agree that our work studies the entire set of minima instead of the ones discovered by SGD, and have added clarifications regarding this aspect in the introduction. We do not believe this is necessarily a weakness though. While SGD is known to explore only a small portion of the minimum, it is less clear whether or to what extent other optimizers behave in similar ways. Additionally, a characterization of the complete set of minima might be useful beyond the context of optimization, such as in studying model complexity. Hence, although our paper does not specialize on minimizers discovred by SGD, the results could still be useful in understanding the loss landscape.

3. The word “connected” has several meanings in this work and I sometimes was confused which one is currently used in a given part of the text. E.g. when two points are in the same connected component (e.g. when mapping with permutations), this is not the same thing as when two points cannot be connected linearly etc. I feel like the manuscript could do a better job at distinguishing these things.

“Connected” indeed has multiple possible meanings. To distinguish different definitions, we have included clarifications in the paper and checked for consistency of the use of this concept. In the first part of the paper (Section 3 and 4), connectedness assumes its mathematical definition given in Section 3. From Section 5 onwards, when discussing mode connectivity, we use the term “mode connectivity” when points can be connected by arbitrary curves and always specify “linear mode connectivity” when only linear interpolation is considered.