Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion

Thanks for your constructive feedback. We will include additional results and discussions in the revision. We believe that our paper will be much stronger thanks to your efforts.

Response to claims: We thank the reviewer for the suggestion, and we will adjust the wording in the next version of our manuscript. Additionally, we would like to clarify that we have included [1] as a concurrent work in the discussion.

Response to W1: We thank the reviewer for this suggestion. Our backbone model selection follows previous studies on model fusion [2,3]. Specifically, we deliberately selected models representing different transformer architectures.

While experiments on larger models would indeed be valuable, they would require substantially more computing resources. Due to computing resource constraints, we focused on models that could be effectively studied within our available infrastructure.

Response to W2: For clarity, the three model fusion methods (Fisher, Regmean, and OT) are not baselines in our evaluation, but rather backbone methods that we enhance with our parameter matching algorithm. To the best of our knowledge, our work represents the first parameter matching algorithm specifically designed to enhance model fusion performance We specifically selected these methods because they achieve desirable performance and provide a diverse set of merging strategies to demonstrate the general applicability of our matching algorithm.

Response to W3: We first would like to clarify that we follow previous studies [2] to leave classification heads unmerged. Unlike attention layers that capture generalizable patterns, classifier layer parameters are highly task-specific in nature. Even minor modifications to downstream tasks (e.g., altering label orders in classification tasks or targeting different output distributions) result in entirely different optimal parameter values for classifier heads. This task-specificity makes merging classifier heads conceptually unsound without task alignment information.

Response to W4: Please refer to our response to reviewer ydmT.

Response to W5: Thank you for this insightful question that helps clarify component contributions. We conducted an ablation study isolating the effects of matching different components:

For Fisher and RegMean, attention matching provides more contributions to model fusion. For OTFusion, FFN matching provides more contributions. These differences highlight how the underlying fusion method interacts with component matching. We will include the results in our revised manuscript.

Response to references: We are committed to adding the suggested references to Related Work of our next revision.

Response to Q1: We thank the reviewer for this thoughtful question. You raise an important theoretical point about joint optimization of rotation and scaling. It's true that our sequential approach (first optimizing rotation $R$, then scaling $\alpha$) cannot guarantee global optimality for the joint ($R$, $\alpha$) optimization problem. Proving that sequential optimization yields the global optimum would be non-trivial, as calculating the general solution of Eq. (12) without knowing the specific values of matrices is complex. We can view our approach (sequentially optimizing $R$ and $\alpha$) as a practical approximation to the joint optimization problem. Developing an algorithm that jointly optimizes for both rotation and scaling symmetries would be an interesting extension to our method, though it would likely come with increased computational complexity. Our current method balances theoretical soundness with computational efficiency.

Response to Q2: We believe there might be no symmetries for Softmax and LayerNorm that require matching since these modules do not contain parameter matrices to be rotated.

Response to Q3: We thank the reviewer for this thoughtful question. While our work focuses on rotation symmetries, we acknowledge that transformers may exhibit additional symmetries beyond the permutation, rotation, and scaling transformations we address. The transformer architecture, with its complex interplay of components, likely possesses a rich symmetry structure that extends beyond what we have explored. Characterizing the complete symmetry group of transformers remains an open question. We identify this as an important direction for future research.

[1] Tran, Viet-Hoang, et al. "Equivariant Neural Functional Networks for Transformers." arXiv 2024.

[2] Jin, Xisen, et al. "Dataless knowledge fusion by merging weights of language models." ICLR 2023

[3] Imfeld, Moritz, et al. "Transformer fusion with optimal transport." ICLR 2024

Thank you very much for your thoughtful feedback. We are honored to have this valuable chance to address your raised concerns and questions. We believe that our paper will be much stronger thanks to your efforts.

Response to comments and Q1: We thank the reviewer for this important question. To clarify, our choice of end ViT models strictly follows previous works [1], resulting in similar performance degradation when merging models trained on different tasks (Table 1 in [1]). We attribute these low metric values to two key factors:

1. Task divergence. The end ViT models were fine-tuned on substantially different tasks, leading to specialized parameters that conflict when naively merged.
2. Non-convexity. ViTs exhibit highly non-convex loss landscapes, making naive interpolation between models challenging compared to simpler models such as MLPs.

Actually, merging large models on different tasks remains challenging. This also motivates our parameter matching strategy which improves model merging without access to any training data. We are committed to including the metric values of the end models in the next version of our paper.

Response to Q2: We thank the reviewer for providing this insightful comment. We find that all orthogonal matrices in the rotation symmetry can be directly replaced by invertible matrices without any barrier. However, invertible matrices are not applicable for parameter matching. In our parameter matching algorithm, the orthogonal matrix is an important promise for Theorem 1. If using invertible matrices, Eq. (9) can only be solved by gradient descent, losing the guarantee of global optimum. In addition, parameter matching requires computation of the inverse matrix of every $R$, which is impractical for invertible matrices. In contrast, the inverse matrix of the orthogonal matrix is just its transpose. In summary, if we only consider the rotation symmetry for transformers, then yes, the orthogonal matrices in the rotation symmetry can be replaced by invertible matrices without barrier; while if we consider the parameter matching based on rotation symmetry, then no, using invertible matrices results in suboptimal parameter matching and large complexity.

We extend our best gratitude for your efforts in the rebuttal phase. We highly value the opportunity to improve our paper. We will include the additional results and discussions in the next version of our paper. We sincerely appreciate your time and consideration.

[1] Imfeld, Moritz, et al. "Transformer fusion with optimal transport." ICLR 2024

Thank you very much for your thoughtful feedback. We are honored to have this valuable chance to address your raised concerns and questions. We believe that our paper will be much stronger thanks to your efforts.

Response to theoretical claims: Thank you for this thoughtful point. We would like to clarify that the perfect orthogonality isn't an assumption but rather a property that holds for any SVD decomposition. Even in noisy parameter spaces, SVD always provides orthogonal singular vectors. Any parameter matrix obtained through practical training, e.g., standard SGD, differentially private SGD, and other optimization methods, can be decomposed this way.

Response to questions: Yes, the correlation between reduced Euclidean distance and improved model fusion performance is backed by previous studies [1]. Their theoretical results show that strong convexity and closer end models can boost the utility of direct model fusion. Additionally, experimental results in [1] and our paper support the correlation from an empirical perspective.

We extend our best gratitude for your efforts in the rebuttal phase. We highly value the opportunity to improve our paper. We will include the additional results and discussions in the next version of our paper. We sincerely appreciate your time and consideration.

[1] Wortsman, Mitchell, et al. "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time." ICML 2022.

Thank you very much for your thoughtful feedback. We are honored to have this valuable chance to address your raised concerns and questions. We believe that our paper will be much stronger thanks to your efforts.

Response to claims and W2: We thank the reviewer for this insightful suggestion. In the binary case, the optimality of parameter matching does not rely on the choice of anchor models. However, this might not hold for multiple models. When merging multiple end models, a naive extension of our method is to select a single model as an anchor and align all others to it via rotation symmetry, as done in pairwise merging. However, the optimality of this approach depends on how the overall distance measure is defined. An alternative approach could involve iterative pairwise merging, but this introduces path dependency issues where the final result depends on the order of merging. We agree that finding optimal alignments across multiple models is an interesting and important direction and plan to explore it in the future work. We will make corresponding adjustments to the claims in the revision of our paper.

Response to experiments 1: Thank you for this valuable suggestion. We've computed the LMC curves of the ViT model with and without our proposed matching technique on the CIFAR-10 dataset. Our analysis reveals two key observations:

1. For most interpolation coefficients $\lambda\in[0,1]$, models merged with our matching approach exhibit lower loss values than unmatched models, indicating improved connectivity.

2. We observe a loss barrier from the LMC curve between the end models used in our experiments. We explain this because ViT exhibits a strong non-convexity [1]. Similar results are recorded in previous works [2].

We will include detailed LMC results in our revised manuscript to fully illustrate these findings.

Response to experiments 2 and W1: We thank the reviewer for this insightful question. We've analyzed the LMC curves between end models fine-tuned on different NLP tasks and observed that all model pairs (in our settings) exhibit positive loss barriers, confirming that fine-tuning often does move models into different loss basins. However, the magnitude of these barriers varies across tasks.

Importantly, we observe positive loss barriers between the non-anchor original model and its matched counterpart. This provides evidence that our matching technique achieves model rebasin through rotation symmetry.

Regarding why matching only the first few layers is often sufficient: this suggests that rotational symmetry divergence primarily occurs in early layers during fine-tuning, while later layers maintain more consistent representations. This aligns with observations that early layers capture more task-specific features [3].

We acknowledge that precisely characterizing loss basin distributions remains an open problem due to the highly non-convex nature of transformer loss landscapes. Even for simpler architectures like MLPs, loss barrier analysis remains challenging. We believe this is a valuable and insightful direction for future research.

Response to comments: We believe there might be misunderstanding regarding this sentence. In our manuscript, we mentioned that the linear assignment problem can be solved precisely by Hungarian Algorithm, rather than weight matching or activation matching problems.

Response to questions: We thank the reviewer for this important point. To our knowledge, our experimental setting differs from the model soup paper [4]. The model soup paper primarily studies merging models trained on the same dataset with different hyperparameters, where models likely remain in the same loss basin as they optimize for the same objective. In contrast, our experiments merge models fine-tuned on different datasets. Our LMC analysis confirms these models occupy different loss basins, as evidenced by positive loss barriers.

We extend our best gratitude for your efforts in the rebuttal phase. We highly value the opportunity to improve our paper. We will include the additional results and discussions in the next version of our paper. We sincerely appreciate your time and consideration.

[1] Park, Namuk, and Songkuk Kim. "How Do Vision Transformers Work?." ICLR 2022.

[2] Imfeld, Moritz, et al. "Transformer fusion with optimal transport." ICLR 2024

[3] Tenney, Ian, Dipanjan Das, and Ellie Pavlick. "BERT Rediscovers the Classical NLP Pipeline." ACL 2019.

[4] Wortsman, Mitchell, et al. "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time." ICML 2022.