Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries

Thank you very much for your thoughtful comments and for taking the time to carefully read our paper.

The contribution seems important for the larger literature on linear mode connectivity, but this isn't particularly well motivated in the paper itself. The STE-MM algorithm seems novel, but because the methods either lower test accuracy or provide a very small increase, my understanding is that these algorithms are not intended to improve metrics for their own sake but to give more evidence that linear mode connectivity of independently trained networks is empirically true. I think this requires a bit more discussion in the paper. In particular, the fact that the convex basin of merging multiple models becomes less sharp is intuitive, but the consequences of this fact aren’t elaborated on much.

Thank you for your thoughtful feedback. As you pointed out, we believe that our work contributes to the broader understanding of linear mode connectivity (LMC), particularly in the context of merging multiple independently trained models. While we touch on this connection in Section 1 of the paper, it may not have been sufficiently emphasized. To better reflect this perspective, we are considering revising the introduction to explicitly highlight the relevance to LMC and potentially modifying the title to more directly reflect this connection, for example, “Linear Mode Connectivity for Multiple Models Modulo Permutation Symmetries.”

We use average-case sharpness as a metric because, as discussed in Andriushchenko et al. (2023), it quantifies how much the loss increases under Gaussian perturbations of model parameters. If multiple models can be transferred via permutations into the same convex basin, we expect the loss around their midpoint to remain relatively flat—reflected in low average-case sharpness.

Indeed, our experiments show that STE-MM consistently reduces average-case sharpness around merged models. While this does not provide direct evidence of a single convex basin, it supports the possibility that such a basin exists. A more fine-grained analysis of this phenomenon remains an important direction for future research, which we intend to pursue.

We are grateful for your valuable comments and for your close reading of the paper.

I recommend that the authors extend their approach to more challenging datasets, such as CIFAR-100 and ImageNet, to further validate its effectiveness.

Thank you for your suggestion. To test our method on more complex tasks, we performed an experiment on model merging using ResNet-50 models trained on the ImageNet dataset. The following table presents the test losses of merged models using various permutation search methods, including STE-MM. Results are shown for up to six models due to GPU memory limitations. The ResNet-50 models were trained using a training script published on GitHub by the FFCV library (i.e., https://github.com/libffcv/ffcv-imagenet). The table reports the average and standard deviation of the test loss across three model merging trials. As seen in the table, the test loss of the merged model using STE-MM decreases monotonically as the number of merged models increases, while that of other permutation search methods increases, demonstrating the effectiveness of our method. Note that the test loss of the merged model could potentially be improved by increasing the model width, even when multiple models are merged. Ainsworth et al. (2023), for instance, have shown that the test accuracy of the merged model improves with increased model width when two ResNet-50 models are merged.

Test loss of the merged model when ResNet-50 models trained on ImageNet are combined

Typo "mdoel" on line 060, second column Please revise lines 115-134 could be shown in a dedicated plot.

We apologize for any typos, unclear figures, or ambiguous expressions in the text. We will carefully revise the manuscript to address all of these issues in the camera-ready version.

I suggest to move Fig. 7 and Fig. 8 to the main paper.

Thank you for your suggestion. As you mentioned, these are important results, so we will include Figures 7 and 8 in the main body of the paper.

could it be used to analyze potential biases or inefficiencies in other, more lightweight, permutation search techniques?

This is a valuable perspective. Since methods based on $L^2$ distance, such as MergeMany and CCMM, do not require training, they can search for permutation matrices with low computational cost. However, our experimental results show that the performance of the merged model deteriorates as the number of models being merged increases when using these methods. This suggests a fundamental difference between the permutation matrices discovered by STE-MM and those identified by lightweight approaches. A more in-depth comparison of these permutation matrices may help uncover the causes of inefficiency in the latter and guide improvements in their design.

Could you comment on how your paper relates to the concurrent work https://arxiv.org/abs/2403.07968?

Thank you for the reference. We were not previously aware of this paper. It proposes the star domain conjecture and introduces the Starlight algorithm to verify it by identifying a parameter $\theta^\ast$ from multiple SGD solutions. The authors show empirically that such a $\theta^\ast$ exists even for models with smaller widths. Entezari et al.'s conjecture (i.e., multiple SGD solutions can be transferred into a single approximately convex basin via permutations) makes a stronger claim than the star domain conjecture, as shown in the figure on page 2 of their paper.

This result is particularly interesting, as the method performs well in the small-width regime. In contrast, our STE-MM approach requires sufficiently wide models to transfer them into a shared approximately convex basin; otherwise, a significant barrier remains.

It is unclear whether this limitation arises from the difficulty of searching the large permutation space—due to the discrete nature of the optimization—or from a more fundamental property of the loss landscape.

If the latter is the case, it may suggest that wider (i.e., overparameterized) models tend to induce simpler loss landscapes. In fact, a recent theoretical study (https://openreview.net/forum?id=4xWQS2z77v) on two-layer neural networks has shown that increasing the width leads to a simpler structure in the loss landscape. It is plausible that a similar phenomenon occurs in deeper networks as well. We believe that further investigation into this aspect is a promising direction for future research.

We appreciate your insightful feedback and the effort you put into reviewing our work.

The algorithm explanation could be improved. I suggest the authors expand and clarify the reason for dummy variables and how the algorithm works in general by following the explanation of the seminal paper (Ainsworth et al 2023), section 3.3.

Thank you for your suggestion on improving the explanation. We will revise the description of STE-MM to make it clearer, drawing on the explanation provided in Ainsworth et al. (2023), Section 3.3.

Experiments are only performed on academic benchmarks such as CIFAR10 and MNIST. I’m sorry about this comment, I hope it does not trivialize your analysis, but I recommend testing the scalability of the method on more complex tasks such as ImageNet.

Thank you for your suggestion. To evaluate the scalability of our method, we conducted additional experiments of model merging with ResNet-50 models trained on the ImageNet dataset. Due to space limitations in the rebuttal, the detailed experimental results are included in our response to Reviewer SJMK’s first comment. The results demonstrate that our method scales effectively to more complex tasks such as ImageNet.

typos: 190: Permutaion Search for Multiple Models 403 right - experimetnal results

We are sorry for the typos. We will carefully review the entire paper and make the necessary corrections.

can the authors provide an analysis of the permutation statistics, for example, what is the percentage of the weights that are being permuted?

Yes, certainly. The following table provides an analysis of how closely the permutation matrices discovered by STE-MM resemble the identity matrix. Specifically, we measured this by counting the number of entries equal to 1 in the diagonal of each permutation matrix and dividing this count by the size of the matrix. This matching rate was calculated for each layer, then averaged across all layers in a model, and finally averaged across all models being merged. The table reports the mean and standard deviation over three model merging trials. As shown, the matching ratio remains close to zero regardless of the number of models, indicating that the found permutation matrices are significantly different from the identity matrix.

Percentage of matches between the permutation matrix and the identity matrix [%]

another thing is, can the authors provide a baseline where no permutation is applied? I think it is important to establish that there are actual permutations in the considered setting.

The following table shows the test accuracy of the merged model when the models are combined without applying any permutation matrices. As shown, the accuracy drops significantly as the number of models being merged increases. This result highlights the importance of searching for appropriate permutation matrices when merging multiple models.

Test accuracy of the merged model without permutation

Thank you for reading our paper carefully and for your constructive comments.

However, other merging methods, such as TIES merging and DARE, are also widely used and have been shown to be effective. Additionally, due to computational cost considerations, simple averaging remains a common approach in multiple-model merging.

Thank you for your comment. While those methods are indeed important prior works, their assumptions differ significantly from ours. Specifically, TIES and DARE propose methods for merging multiple fine-tuned models based on a shared pre-trained model. In contrast, we focus on merging models that have been trained from scratch using different random seeds. Previous studies, such as model soups (Wortsman et al. 2022), have shown that simply averaging the weights of fine-tuned models originating from a common base model can improve performance. However, when merging models trained from scratch with different seeds, it has been observed that naive weight averaging can substantially degrade performance. Our method addresses the more challenging scenario of merging such models effectively, which we consider one of our key contributions.

The clarity of Figure 5 could be improved.

We apologize for Figure 5's poor visibility. As you pointed out, we will revise it in the camera-ready version to improve its readability.

Typos: 059 (right column) model; 061 (right column) denotes.

Thank you for pointing out the typos. We will thoroughly proofread the entire manuscript to ensure that all typos are corrected.

Could STE-MM also assist in determining the optimal hyperparameters for different models? For instance, in model merging, where different models have distinct weights, can STE-MM help in assigning appropriate values?

Thank you for your question. If we understand correctly, you may be referring to whether STE-MM could be extended to assist with hyperparameter tuning when merging models with distinct weights. In its current form, STE-MM does not explicitly address hyperparameter optimization—its objective is to discover effective permutations for merging models. The algorithm itself only includes standard training hyperparameters such as learning rate and batch size. For example, in Equation 5, the values of $\lambda_1, \ldots, \lambda_n$ are drawn from a uniform distribution and are not considered hyperparameters. Therefore, applying STE-MM to hyperparameter selection would require substantial modification, as this lies outside its intended scope.