Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods

Thank you for your valuable feedback.

• A key weakness is the lack of significant novelty in the paper's primary insights. Similar observations have been made in previous works, such as [1] and [2]. These papers also explore Hessian-based barrier approximations and spectral analysis to explain LMC. Although this does not undermine the findings of the current paper, these existing works should be cited and discussed.

We apologize for the lack of discussion of these papers. We have added a discussion of these to the Appendix A in the manuscript.

The existence of these works does not significantly diminish the novelty of our study. Specifically:

1. [1] demonstrated that the barrier value can be approximated using a second-order Taylor approximation, but this approach is only validated for the case of spawning and has not been validated for permutations. Our work highlights the distinctiveness of these scenarios. Unlike [1], we do not merely show that a Taylor approximation can be used to derive an equation to predict the barrier value. Instead, we reveal that the model distance under permutations cannot be reduced to a level where the barrier can be well-estimated by such an approximation. This distinction underscores the unique contribution of our findings.

2. [2] explored the reduction of rank during SGD-based training and the orientation of top singular vectors via SVD on the weights. While this is relevant to our paper, [2] does not examine the effects of permutations in details. Moreover, it does not discuss the interplay between hidden layer inputs and top singular vectors under permutations. Our study addresses these gaps, thus providing complementary insights that are absent in [2].

We believe these points demonstrate that our contributions remain novel and valuable, even in light of these prior works.

• Another concern is the transition to per-layer analysis in the discussion of WM. The authors claim that eigenvector alignment results in aligned representations, but this can be challenged. Firstly, the use of the Lipschitz constant in the analysis limits the result to essentially one activation only. When multiple layers are involved, the Lipschitz constant becomes excessively large.

Indeed, the Lipschitz constant becomes very large when considering multiple layers, making it challenging to directly extend the discussion in this paper to such cases. However, as demonstrated in the paper, the analysis of the singular vectors and the singular values appear to be crucial factors in establishing LMC, as suggested by experimental results. For example, in Figure 14, we examine the relationship between the number of large singular values, the test loss of the original ResNet20 and VGG11 models, and the test loss of the merged model using WM. As shown in this figure, when the number of large singular values is relatively small compared to the model width and the loss of the original models is low, the loss of the merged model is also low. In this paper, we have demonstrated that singular vectors corresponding to large singular values play a crucial role based on single-layer analysis. Moreover, Figure 14 suggests that our single-layer analysis yields consistent insights for multi-layer models. In this regard, we believe that singular vectors play a significant role in the analysis of multiple layers. A more detailed and specific analysis remains a topic for future research.

• Secondly, [3] show that there are often no barriers at the layer level (even when barriers exist at the full network level). Although this could still be explained by eigenvectors, it weakens the connection between similar activations on the level of one layer and the lack of barriers at the full-model level.

Although further investigation is required to understand why LMC holds at the layer level as shown in [3] when permutations are not used, Figure 2 in [3] demonstrates that the barrier monotonically increases when multiple layers are averaged between the two models. We expect that closing the activations of each layer could reduce the barrier that arises during this averaging process. Otherwise, it would be counterintuitive for LMC to be established by optimizing permutations to minimize the objective functions used in activation matching and weight matching. In our study, we focused on single-layer analysis as a first step, given the difficulty of addressing multiple layers. However, we recognize the importance of extending the analysis to multiple layers in future work.

Thank you very much for your response. Below, we have provided answers to your additional questions.

• Can you please clarify these sentences in your comment: "We expect that closing the activations of each layer could reduce the barrier that arises during this averaging process. Otherwise, it would be counterintuitive for LMC to be established by optimizing permutations to minimize the objective functions used in activation matching and weight matching."?

We apologize for any confusion or lack of clarity in these sentences. The objective functions of AM and WM are designed to find a permutation that makes the hidden layer outputs of the two models close. The fact that the permutations found by these methods reduce the barrier suggests that bringing the outputs of the hidden layers closer is effective in reducing the barrier. What we intended to convey in these sentences is that bringing the outputs of the hidden layers close would also reduce the barrier when merging some layers (but not all layers) between the two models.

• I believe that it should be emphasized that the theorem holds only for one layer, because otherwise it is misleading in a sense of possible interpretation that alignment of eigenvectors is the condition for full LMC.

You are correct. In this paper, we did not emphasize the fact that the results were based on an analysis focusing on only a single layer of the model. However, when discussing LMC in a rigorous manner, all layers of the model should be taken into account. One of the contributions of this paper was to empirically show that the results obtained from an analysis of just one layer can provide valuable insight into the merging of entire models, including all layers.

We have revised Sections 1 and 7 to emphasize this point in the manuscript, with the changes highlighted in blue. In the Conclusion (Section 7), we added the following statement: "In this paper, we primarily analyzed WM from the perspective of individual layers (e.g., Theorem 4.2). However, since the actual network is multi-layered, it remains unclear why our analysis can explain the phenomenon so effectively. In the future, a more comprehensive analysis that accounts for the multi-layered structure of the network will be necessary."

• In my view results in Table4 are less significant not because of the difference of the barrier size between STE and WM, but rather because of the relative raise of the barrier - the initial barriers for STE in case of CIFAR10 with ResNET are much more significant than the barriers obtained with WM, so I guess the barrier between two permuted models is also to be expected to be high.

This is certainly true for ResNet20. However, as shown in Table 5, the results for VGG11 reveal a different trend. Specifically, the accuracy barrier between $\theta_a$ and $\pi_b(\theta_b)$ and the accuracy barrier between $\theta_a$ and $\pi_c(\theta_c)$ are smaller for STE compared to WM. In contrast, the barrier between $\pi_b(\theta_b)$ and $\pi_c(\theta_c)$ is larger for STE than for WM.

These observations suggest that, at least for VGG11, the large barrier between $\pi_b(\theta_b)$ and $\pi_c(\theta_c)$ in STE is not merely due to the permutations found by STE being of lower quality than those found by WM (i.e., failing to effectively reduce the barriers). Instead, we believe that this behavior is driven by a more fundamental underlying cause.

• But we do assume that ReLU is differentiable when we compute gradients and Hessians, why do you think it is a problem?

The ReLU function is non-differentiable only at the point where the input is 0. In other words, it is differentiable at all other points. When calculating the test loss, we use the outputs corresponding to the test data inputs fed into the model. If the distribution of the test data is continuous, the probability of the model encountering non-differentiable points is 0, provided that the model's weights do not take special values. Thus, we can compute the gradient for the given weights and data.

However, a point along the linear interpolation between the parameters $\theta_a$ and $\pi_b(\theta_b)$ may pass through a non-differentiable region. Specifically, if the distance between the two models is large enough to cross the non-differentiable region, the barrier estimation using Eq.(2) could fail. On the other hand, if the interpolation avoids the non-differentiable region, Eq.(2) remains valid. In such cases, the approximation accuracy depends on the order of the Taylor approximation.

In this experiment, it is unclear whether the discrepancy between the barrier estimates based on the Taylor approximation and the actual barrier values is due to the interpolation passing through a non-differentiable region or the low order of the Taylor approximation. What can be stated with confidence, however, is that a large distance between the two models increases the likelihood of such issues arising.

Thank you very much for reading our paper so carefully. We especially appreciate that you read and commented on the details of the paper, such as the proofs.

• In line 70, the sentence "... this intuition is inaccurate ..." raises a concern. I think the intuition presented above this sentence is correct, i.e., if $\theta_a \approx \pi(\theta_b)$ then it is reasonable to think that the interpolated model will produce similar outputs. I believe the main narrative of the paper appears to be that even when , where $\theta_a \not\approx \pi(\theta_b)$ is found by WM, the interpolated model can produce approximately equal outputs.

You are absolutely right; the phrase "this intuition is inaccurate" was misleading. The intuition that $\theta_a \approx \pi(\theta_b)$ implies the establishment of LMC is correct. More precisely, what we intended to convey here is that this intuition is not the underlying reason why LMC holds. Even when $\theta_a \not\approx \pi(\theta_b)$, LMC can still hold by using WM. We have revised and updated Section 1 of our manuscript accordingly.

• In line 211, the phrase "... algorithm similar to the backpropagation method to efficiently compute $\mu^\top H$ ..." would benefit from further clarification in the appendix. I couldn’t locate a section elaborating on this point. What specifically do the authors mean by "similar algorithm to backdrop"?

We apologize for not including the details. We used the vhp function provided in PyTorch to compute $\boldsymbol{\mu}^\top \boldsymbol{H}$. The explanation for this computation is detailed in Chapter 5.4.6 of Bishop (2006). It is known that $\boldsymbol{\mu}^\top \boldsymbol{H}$ can be efficiently computed by applying the differential operator $\boldsymbol{\mu}^\top \nabla$ to the equations of forward and backward propagation. Consequently, the equations for evaluating $\boldsymbol{\mu}^\top \boldsymbol{H}$ closely resemble those of standard forward and backward propagation. PyTorch offers the torch.autograd.functional.vhp function for this purpose, which we used for our calculations. We have modified the sentence in our manuscript to clarify that we used this function to calculate $\boldsymbol{\mu}^\top \boldsymbol{H}$.

https://pytorch.org/docs/stable/generated/torch.autograd.functional.vhp.html

• Regarding the sentence in line 261 "... we can find permutation matrices such that the left and right singular vectors of the two models match ...": it seems unlikely that all singular vectors can be matched purely through permutation in all scenarios.

As you pointed out, the explanation may be misleading, so we will rewrite "If the singular values of both models are almost equal (i.e., for all $\ell$ and $i$, $s_{\ell, i}^{(a)} = s_{\ell,i}^{(b)}$) and we can find permutation matrices such that the left and right singular vectors of the two models match, then two models $\theta_a$ and $\theta_b$ will be exactly the same and LMC clearly holds." as “The $L_2$ distance between the models is expressed by the difference in singular values and singular vectors between the two models, as indicated by Equation (3).”

• In line 286 "distributions of the singular values of the weights ... almost equal": This seems like a strong claim to me. The paper presents results with common datasets like MNIST and CIFAR10 using MLP, VGG, and ResNet. I think the conclusion requires further evidence or a more theoretical foundation. The appendix H.1 discusses this in a more detailed manner, and Figure 7 indicates that output layers' singular value distribution have variability. For this case, the paper suggests that the singular values are approximately a constant multiple across the two models. Even though the output layers' singular value distributions seem correlated, the "constant multiple" assumption seems like a strong one to me as there are contradicting examples in Figure 7 (e.g. Figure 7(a) light blue curve and red curve).

Indeed, this might seem like a strong argument at first glance. In our experiments, we observed that the distribution of singular values is nearly identical across independently trained models, at least within the hidden layers shown in Figure 7. However, it remains unclear whether this observation generalizes to other datasets or broader tasks. Providing theoretical support for this would require a detailed analysis of the weights trained by SGD, which is likely to be highly challenging. This remains an open question for future research.

Finally, we have added an additional figure (Figure 8) to the paper, demonstrating that correcting the output layer by a constant factor can account for the differences in the distribution. While some discrepancies persist, the figure illustrates that these differences can largely be attributed to constant factors in all cases.

Thank you for your valuable comments.

• Table 1 (and others) could be improved through highlighting sections to support the conclusion
• Figures are small and difficult to read. For the distribution plots (Fig. 1, 4),
• "the barrier between $\pi_b(\theta_b)$ and $\pi_c(\theta_c)$ is smaller with WM than with STE. This means that there is a significant difference between the principles of permutations obtained by WM and STE." (line 519) is difficult to evaluate by eye in Figure 6 and not particularly convincing.

We are sorry about the difficulty in reading the tables and figures. For Table 1, a column labeled "Diff." was added to display the difference between the Taylor approximation and the actual barrier values, making the differences easier to read. Additionally, a t-test was performed, and values in the "Diff." column were bolded if the differences were statistically significant.

Next, regarding the suggestion to use a logarithmic scale for the vertical axes of Figures 1 and 4 for better readability, we tested it but found no significant improvement in visibility. Therefore, we have kept the current format. However, we increased the font size in Figure 4 as it was previously too small.

Finally, regarding the comment that the difference between STE and WM is difficult to see from the figures, we have moved the accuracy landscape figures to the appendix and introduced a new Table 3. This table represents the values of the loss and accuracy barriers between $\pi_b(\theta_b)$ and $\pi_c(\theta_c)$, making the differences clearer.

• The statement "suggesting that the reason AM achieves LMC is likely the same as for WM" (line 458) seems overly confident given the evidence presented. Are there other explanations for the similar results?

When using WM and AM, the $R$ values and distance reduction rates (Figures 2 and 15, Tables 1 and 4) between the two models are very similar. Therefore, we concluded that the principles of WM and AM are fundamentally the same. However, some degree of uncertainty remains, as they are not entirely identical. Meanwhile, providing additional evidence would be challenging. As a result, we have opted to moderate some of the more emphatic claims in the text.

• Connecting it back to the idea in the intro about SGD and the loss landscape, do the authors have any thoughts on how their findings relate to the efficacy of SGD?

This paper is inspired by Ainsworth et al. (2023). Section 1 in their paper states that if the solutions of SGD form a basin when considering permutation symmetries of NNs and the basin is furthermore a convex set, three key questions can be addressed:

1. Why does SGD work well?
2. What are all the local minima?
3. Why do independently trained models perform similarly?

The paper demonstrated that LMC can be established by identifying an appropriate permutation between any two trained models. This partially supports the hypothesis that the solutions of SGD form a convex basin. Conversely, the existence of such permutations might also stem from a certain degree of similarity between independently trained models. In this study, we analyzed this similarity through singular vectors and singular values. We believe that these findings contribute to answering the three questions regarding the efficiency of SGD mentioned earlier.

• How well does this analysis scale beyond 2 or 3 models? "Therefore, WM is likely to be more advantageous, especially for merging three or more models" (line 522), is there additional proof of this?

As the number of models increases, the barriers are expected to grow; however, this analysis should scale reasonably well. The distance between models reduced by WM is small, but this reduction allows us to align a small number of singular vectors with large singular values. In other words, the rate of distance reduction is proportional to the degree of alignment of singular vectors with large singular values between the two models.

For the three models $\theta_a$, $\theta_b$, and $\theta_c$, if $\theta_b$ and $\theta_c$ are brought closer to $\theta_a$, the distance between $\theta_b$ and $\theta_c$ also decreases. This process indirectly aligns the singular vectors with large singular values in $\theta_b$ and $\theta_c$, thereby reducing the barrier. In principle, the same trend should hold even as the number of models increases.

In Section 6, we conducted experiments using three models to demonstrate the differences between WM and STE visually. A detailed investigation into how well this approach scales with four or more models, as well as exploring more appropriate merging methods for such cases, is left as a topic for future work.

• What explains that permutations found by activation matching are much different from weight matching? Can we use SVD to analyze the qualitative/quantitative differences between the two methods that ostensibly minimize a similar objectives, i.e. matching the singular values.

At least in our paper, we did not conclude that the permutation matrices identified by WM and AM were entirely different. In our experimental results, applying the permutations found by WM and AM to the model produced very similar results, suggesting that these permutations may be closely related.

We experimentally measured the distance between the permutation matrices found by these search methods when matching ResNet20 models. In ResNet20, there are 12 permutation matrices. Note that the number of permutation matrices is smaller than the total number of fully connected and convolutional layers due to the presence of skip connections. For the permutation matrices found by WM and AM at each layer, we calculated the number of matching rows and the ratio of matching rows to the matrix size. The results are summarized in the following table. For reference, the table also includes the experimental results for the permutation matrices found by STE.

The table presents the average and standard deviation of the results from five runs conducted with different seeds. The table shows that the permutation matrices found by WM and STE are entirely different, whereas those found by WM and AM are more similar compared to the results of STE. In fact, as shown in Tables 1 and 4, and Figures 2 and 15, both WM and AM reduce the distance between the two models and align singular vectors with large singular values.

Finally, it is important to note that in models with smaller widths, the results may vary between AM and WM, although previous studies have focused on wider models to facilitate easier achievement of LMC. As described in the paper, WM aligns singular vectors with large singular values across the two models. In contrast, AM aims to make the outputs of the hidden layers close, prioritizing not only singular vectors with large singular values but also right singular vectors with large inner product values with the hidden layer inputs, as discussed in Section 5.

In Subsection 4.4, we demonstrated that singular vectors with large singular values have large inner product values with hidden layer inputs, and therefore, the results of WM are very similar to those of AM. However, this observation applies primarily to models with large widths. When the width is small, right singular vectors with large singular values do not necessarily have large inner product values with the hidden layer inputs (see Figures 11 and 12). Consequently, for models with smaller widths, the results of WM and AM differ, and AM may yield superior performance.

• In [2], authors also analyzed linear connectivity of more than 3 models for checking strong linear connectivity where it is found that as width of the model increases, weight matching allows strong linear connectivity. Are there any insights we can gain from SVD when it studying strong linear connectivity.

The method used to evaluate strong linear connectivity in [2] is similar to the approach in our study. Specifically, three independently trained models, $\theta_a$, $\theta_b$, and $\theta_c$, are prepared, and permutations $\pi_a$ and $\pi_b$ are searched to make $\theta_a$ and $\theta_b$ close to $\theta_c$. The barrier between $\pi_a(\theta_a)$ and $\pi_b(\theta_b)$ is then calculated.

By making $\theta_a$ and $\theta_b$ closer to $\theta_c$, the singular vectors between $\theta_a$ and $\theta_b$ will be aligned indirectly. At this time, increasing the model width facilitates aligning the singular vectors between models because the ratio of the number of larger singular values to the model width decreases with increasing model width (see Figure 10). In other words, widening the model width is expected to reduce the barrier between $\pi_a(\theta_a)$ and $\pi_b(\theta_b)$. Therefore, even in the case of strong linear connectivity, increasing the model width should further reduce this barrier.

Thank you for reading our paper carefully and for your constructive comments.

• I would suggest authors to keep the vectorized notation instead of summations to make the paper an easier read.

Thank you for your suggestion, and we apologize for the difficulty in reading the equations. While many of the equations involve summations, most are already expressed in vector form. Additionally, consolidating these expressions into matrix form would make it more challenging to highlight our core idea, namely, the inner products between left singular vectors or between right singular vectors, as shown in Equation (4). For this reason, we prefer to retain the current formulation.

Nonetheless, your suggestion is highly valuable, and we will carefully consider whether other parts of the paper could be simplified or made more readable.

• Observation that minimizing the l2 distance is equivalent to maximizing the inner product is pretty well know. [1] also leverages this observation to implement the weight matching algorithm. I found those contributions incremental.

Indeed, [1] demonstrates that minimizing the $L_2$ distance is equivalent to maximizing the Frobenius inner product (i.e., the inner product between the weight matrices of two models). However, [1] does not discuss how this relates to the inner product of singular vectors, as we have analyzed in our paper. Our findings reveal that WM maximizes the inner product of singular vectors with large singular values, which plays a key role in the formation of LMC. We believe this is an important contribution, as it has not been emphasized in existing research.