The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions

Thank you for your substantive review, to which we have replied below.

F. Comparing Dissimilar Tasks and Scales

Experiments are performed mostly on relatively small vision models with relatively simple tasks. [...] To compare different domains, one would have to compare models in the same regime varying the domain,

We agree with this assessment of our experimental scale, and analogous to figures 4-6, have added an experiment that fine-tunes ViTs from google/vit-base-patch16-224 on CIFAR-100: https://imgur.com/a/AculLLH.

We also note that the claims below (sections 4.3 and 5) are negative statements, for which a counterexample is sufficient:

1. More pre-training does not always improve fine-tuning stability (figures 4-6).
2. Barriers do not generally scale with $L^2$ divergence (figures 7-8).
3. Barriers and $L^2$ do not diverge exponentially proportional to perturbation scale (figure 9).

In these cases, we do not strictly need to isolate the effects of task vs architecture or scaling.

G. Fine-tuning Task Difficulty

the claim that going from hard to simple improves stability is evaluated only on CIFAR100 and CIFAR10, which are very entangled beyond ‘hard’ and ‘simple’.

We agree that the claim “Pre-training on harder tasks improves stability” is weak. We attempted to make this claim more precise by relating stability to fine-tuning performance, but preliminary evidence in this direction is inconclusive. We have therefore replaced this claim with our new findings on CKA and ensembling (below).

A., B. Functional Similarity And Model Diversity

Why not evaluate behavioral similarlity (CKA, SVCCA, agreement, etc)? As the authors note, the same/different basins may be relevant for merging / ensembling models, for which it is essential to know whether or not models actually learn different behavior.

Reviewers sufv and tDxT have also raised this excellent point. Based on your feedback, we have run additional experiments that find:

1. Training instability (as evidenced by larger barriers) indeed correlates with functional dissimilarity (Angular CKA).
2. Perturbations to training can be used to improve ensemble performance, with more functional dissimilarity leading to better ensembles. This shows that instability increases model diversity.

Please see our full rebuttal to Reviewer sufv for details and figures.

Finally, to understand why we chose to focus on barriers as a measure of functional dissimilarity, please also see our rebuttal to Reviewer HaCY under Non-Linear Connectivity.

Other Questions

There are several alignment methods that improve over git re-basin, e.g. Sinkhorn based [1] or learned [2].

We note that weight matching (WM) is already able to greatly reduce barriers between independently trained networks for our standard ResNet20-32 architecture [Ainsworth et al. 2023], whereas WM has little effect on the identically spawned networks of our experiments. The ineffectiveness of WM in our case suggests that other methods may not fare much better. Practically speaking, Peña et al. [2023] and Navon et al. [2024] are also quite costly (scaling poorly to larger models), as they must learn their permutations.

Figure 2 and 3 titles and caption could be improved. Left and middle figure have the same title. Why are Gaussian permutation barriers evaluated without accounting for permutations? Figure 3 - same as which of Figures 2?

Thank you for the detailed corrections. We will revise the titles/captions, and add figures for permutation-aligned barriers for Gaussian perturbations into the appendix. Note that just like in batch perturbations, permutation does not reduce Gaussian-perturbed barriers.

I would be curious to learn why the authors chose their experimental setup as they have. Are these mostly computational constraints, or based on previous work?

Indeed, our experimental setup is motivated by that of previous works [Frankle et al. 2020, Vlaar & Frankle 2022, Juneja et al. 2023, Ainsworth et al. 2023]. Having said that, evaluating stability for training (as opposed to fine-tuning) would require full runs which is costly when considering many replicates, perturbation times/magnitudes, and hyperparameter settings.

Also, the submission motivates the work via two streams: optimization and work on early training phases. To me, that duality is lost in the course of the submission, and it is more of a generalization of existing work in the latter. I am curious what the authors see as the optimization perspective on their work.

We apologize for this confusion. Our intent was to disambiguate between our notion of instability (divergence of two trajectories that each independently converge to a solution) and that of work such as Wu et al. [2018] (tendency of a trajectory to fail to converge to certain solutions). We will de-emphasize the optimization perspective in the introduction to reflect its relevance to our work.

Thank you for the detailed review, which we address by topic below.

E. $L^2$ Divergence

Also the $L^2$ distance is an intuitive metric to quantify the divergence between two runs. [...] Can the authors therefore provide similar Figures as the ones in Figure 2 or Figure 3, where the error barrier is measured in terms of divergence? It would particular interesting for me to see, how much the permutation of neurons $P$ can reduce the $L^2$ distance between two networks.

Apologies for the omission. We have plotted $L^2$ divergence in place of barriers here and will add them to the appendix for all figures: https://imgur.com/a/BKMQHQk.

We do not prioritize $L^2$ divergence as it is not as indicative of functional similarity as barriers. $L^2$ is sensitive to optimization choices (e.g. weight decay), and network scale symmetries. Indeed, we observe that:

• The relationship between $L^2$ and barriers differs depending on hyperparameter settings for our vision experiments (section 5, figure 7 colors).
• $L^2$ and barriers are unrelated for our language experiments (section 5, figure 8).

Regarding weight matching (WM), Ito et al. [2024] finds:

permutations found by WM do not significantly reduce the distance between the two models [...] WM satisfies LMC by aligning the directions of singular vectors with large singular values in each layer’s weights.

This highlights another lack of correspondence between $L^2$ and barriers.

• Ito, A., et al. (2024). Analysis of linear mode connectivity via permutation-based weight matching

C. Non-Linear Connectivity

Given the fact that there is some work, e.g. Draxler et al. [2018] or Garipov et al. [2018], which showed that the optima in neural networks from different initializations can be connected through (simple) non-linear curves, I wonder how much it makes sense to look only at the barriers along the linear path between the weights. [..] Both work show that minima reached from different initializations can be connected through simple, non-linear curves. Can the authors discuss how this paper relates to these works?

This is an important point also raised by Reviewer HaCY. Please see our rebuttal to Reviewer HaCY under C. Non-Linear Connectivity, where we address this issue.

A., B. Functional Similarity And Model Diversity

I think the main weakness of this paper is that it is still unclear to me in what way the diverged solutions are actually "different" from each other, [...] For instance, it could be interesting to understand whether all pertubations still converge to the same manifold and how different pertubations for different points in time and magnitude of pertubations are distributed to each other (in terms ofdistance).

The authors also mentioned that it is desired in ensembling methods to combine diverse solutions, [...] I think my main concern with this work is that I am not sure what a reasonable way is to connect divergence to diversity of solutions. Can the authors perhaps comment on this?

These are important observations echoed by Reviewers sufv and mmJd. Based on your feedback, we have conducted additional experiments using CKA and ensembling to measure model diversity. Please see our full rebuttal to Reviewer sufv for details.

Regarding the first quoted passage, could you clarify what you mean by “converge to the same manifold”? This may be related to the section C. Non-Linear Connectivity.

Other Questions

It would be helpful to show Figures 2-6 and Figure 9 with the y-axis in log-scale to distinguish the lines even better.

We have updated these figures accordingly and will add them to the text:

• Figures 2 and 3 https://imgur.com/a/fig2-3-jND4s6E.
• Figures 4, 5, and 6 https://imgur.com/a/hsJyGhZ. We provide tablbes in place of Figure 4 as barriers are 0 for $\sigma<0.01$.
• Figure 9 https://imgur.com/a/jUXmcZQ.

What are the conclusions on network training that you draw from the results of this paper?

Our main conclusion is that neural network training is unstable near initialization even without randomness, meaning that interventions to reduce SGD noise (larger batch size, reduced learning rates, longer warmup periods) cannot eliminate this instability. Similar work has found that the trainability of neural networks can be unstable to small hyperparameter changes [Sohl-Dickstein 2024].

Conversely, pre-trained networks can still be fine-tuned into different modes (and thus, diverse solutions) with sufficiently large perturbations. Our findings suggest ways to manage model diversity among multiple training runs, such as by adjusting pre-training length, changing SGD noise magnitude via hyperparameters, or by directly perturbing networks.

We will add this conclusion into the text.

• Sohl-Dickstein, J. (2024). The boundary of neural network trainability is fractal.

Thank you for your thoughtful review. We have addressed the quoted comments and questions point-by-point below.

C. Non-Linear Connectivity

Due to the continuous symmetries, there are equivalent local minima that should be considered the same basin and are not linearly connected. [...] However, these solutions are not considered to be in the same basin according to the authors. [...] I wonder if the authors know how to tell whether $\theta_T$ and $\theta’_T$ are connected by continuous symmetry or not.

Thank you for raising this crucial and subtle distinction, which is also suggested by Reviewer tDxT.

Our definition of “loss basin” (which we refer to here as “LMC basin”) in section 2 (Linear mode connectivity) reflects when the loss landscape is locally convex, and is drawn from Neyshabur et al. [2020] and Frankle et al. [2020]. Although this perspective is more restrictive than a general notion of “mode connectivity” [Draxler et al. 2018, Garipov et al. 2018], ours is both theoretically and practically relevant:

1. Despite being a non-convex problem, neural network training has many connections with convex optimization. LMC basins describe approximately convex regions of the loss landscape.
2. Classical methods for determining the stability of dynamical systems take a linear or quadratic approximation, which holds precisely in LMC basins.
3. Model merging applications require networks to share a LMC basin, as shown by permutation alignment research [Singh & Jaggi 2020, Entezari et al. 2022, Ainsworth et al. 2023].

Prior work, both theoretical [Simsek et al. 2021, Lin et al. 2024] and empirical [Draxler et al. 2018, Garipov et al. 2018, Sonthalia et al. 2024], also points to non-linear connectivity being a trivial property of neural network minima in general. If this is the case, then knowing if $\theta_T$ and $\theta’_T$ are continuously connected would be rather uninformative.

We apologize for not making our definition of “loss basin” and our justifications for the choice clear, and will update the above points in the text.

• Simsek, B., et al. (2021). Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances.
• Lin, Z.,et al. (2024). Exploring neural network landscapes: Star-shaped and geodesic connectivity.
• Sonthalia, A., et al. (2024). Do Deep Neural Network Solutions Form a Star Domain?.

D. Correspondence With SGD Noise

In this work, the authors freeze the noise in SGD, and the injected noise is the main object of study for stability analysis. In reality, the noise of SGD may be much larger than the perturbations studied in this paper and overshadow them. Thus, I don't think the stability the authors study is very relevant. [...] I wonder if the authors could quantify how important the perturbation studied in this paper is compared to the SGD noise.

We agree that we have not sufficiently established the relative magnitude of our perturbations vs SGD noise. To address this, we replicate the parent-child spawning experiment of Frankle et al. [2020] (Frankle baseline) and show that our batch perturbations are a lower bound on the Frankle baseline’s barriers https://imgur.com/a/frankle-ThKay8T. In this comparison, we scale batch perturbations to the expected magnitude of SGD noise at the perturbation time $t$: making batch perturbation equivalent to taking only one step at time $t$ with different SGD noise, as opposed to using different SGD noise from $t$ onwards in the Frankle baseline.

We also tested even smaller perturbations by perturbing a fraction of weights at the smallest perturb scale we used in our main experiments ($10^{-4}$). We find that as little as a single (!) perturbed weight (at a fraction of $10^{-6}$) causes barriers at initialization https://imgur.com/a/AvfP8Mh, which is well below the scale of noise from sources such as hardware indeterminacy.

The significance of our approach is that, as per section 3.1, the exponential convergence or divergence of a deterministic dynamical system dominates the diffusion effects of SGD noise [Wu et al. 2018]. This means that, in the dynamical systems model, training trajectories will diverge as long as they are unstable to noise once, whereas in the diffusion model, divergence depends on noise persisting throughout training.

Other Questions

In the numerical experiments, I have the impression that the amplitude of the perturbation can be much larger than rounding error. Where could these perturbations possibly come from?

Sources of perturbation in regular training could include SGD noise (batch order, data augmentation, hardware indeterminism) as well as model pruning or quantization. We do not target a specific source so as to keep our experimental findings general.

Thank you for your insightful review. Please find our replies below for the quoted points.

A. Other Similarity Measures

I'm wondering how other similarity measures differ from the perspective presented by the authors, e.g., CCA or CKA index, which could be an interesting add-on to the current work.

This is an excellent idea and was also recommended by Reviewer mmJd. We prioritized barriers in our original analysis in order to determine when networks are in the same locally convex loss basin. For our reasoning about why this is useful, see the discussion of the advantages of linear mode connectivity in our rebuttal to Reviewer HaCY, under C. Non-Linear Connectivity.

Representational similarity measures (e.g. CKA) are a more generic way to measure functional similarity that is not sensitive to mechanistic differences between neural network weights, such as barriers that prevent effective model merging. Nevertheless, we agree that measuring similarity in a more general sense will better illuminate how perturbed training trajectories differ.

Accordingly, we have now conducted the following experiment on our standard setting (figure 2):

1. We compute output dissimilarity (in $L^2$ between logits, or % of disagreeing classifications) and Angular CKA [Williams et al. 2021] between pairs of networks in our experiments: https://imgur.com/a/zOBlg5X.
2. As expected, measures of functional dissimilarity increase as barriers increase.
3. Functional similarity becomes more sensitive to batch perturbation later in training.

Note that Angular CKA ranges from 0 (perfectly similar) to $\pi$ (perfectly dissimilar), with $\pi/2$ indicating no correlation. We plot the largest (most dissimilar) CKA value over all residual block outputs, which are computed over 10000 examples using software from [Lange et al. 2023].

We will replicate the above experiment for our fine-tuning settings for inclusion in the text.

• Williams, A. H.,et al. (2021). Generalized shape metrics on neural representations.
• Lange, R. D., et al. (2023). Deep networks as paths on the manifold of neural representations.

B. Model Diversity

How can optimally perturb a model (and at which point) increase the final model's performance of the averaged model? Would that perturbation be universal across different merging strategies or should each strategy have its own method of perturbation?

This is a very interesting and novel application which we did not previously consider. We have interpreted your point as the hypothesis that deliberately perturbing models increases model diversity, leading to improved ensembling performance. We test this hypothesis as follows:

1. We take our measurements from above, and plot Angular CKA (x-axis) against ensemble performance: https://imgur.com/a/ah7kCfg.
2. We find that the ensembles indeed perform better on more dissimilar model pairs, which is in line with the proposed hypothesis.
3. Note that, as shown by barriers increasing with Angular CKA, model averaging performs worse for more dissimilar model pairs.

By ensembling, we mean that in each of our spawn-and-perturb experiments, we evaluate the average of the output logits for the unchanged and perturbed networks after training. While Gaussian perturbations appear to have a more consistent relationship between functional dissimilarity and ensemble performance, we leave a full exploration of which perturbations are best suited for different merging strategies to future work.