Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks

We are grateful to the reviewer for their valuable feedback. To address some of the reviewer's concerns, we have added new experiments to our manuscript (Appendix F.3, Appendix F.4), which are discussed in bullets 3 and 4 of this response.

1. “The target scenario… is not common in practice”

We believe the current merging literature focuses on a narrow scenario—merging models fine-tuned from the same initialization—and that it is important to extend model merging to broader scenarios, as demonstrated for CNNs in [1,2,3]. For example, merging weights of two unrelated models from an online repository (e.g., Hugging Face or GitHub) is a practical use case where models are often not fine-tuned from the same initial model, making most existing merging techniques unsuitable.

2. “I suggest the authors address this point when discussing the connections with ZipIt”

We thank the reviewer for this suggestion. Our manuscript already details the similarities and differences with ZipIt (Section 2, Appendix A.2) and the effect of data augmentation (Appendix F.7, Appendix G). We will ensure these are referenced in the main text. Please let us know if further clarification is needed.

3. “...discussion about the next steps in the paper… merging truly similar architectures but trained entirely from scratch on different datasets?”

To address this concern, We have added new experiments to our work (Appendix F.3, Tables 22 and 23) merging fine-tuned ViTs from different pre-training strategies (CLIP contrastive approach and supervised learning with ImageNet-1K). Again, FS-Merge consistently outperformed the baselines across all pre-training strategies.

4. “I am a little curious about the potential effects if the method were applied to the more common setting”

In response to the reviewer's request, we evaluated FS-Merge in the scenario commonly studied in the literature, which involves merging models fine-tuned on different tasks from the same pre-trained initialization (Appendix F.4, Table 24). While FS-Merge outperforms most baselines in this setting, its margin over methods like distillation and RegMean is small, with RegMean achieving comparable performance using fewer resources. Thus, despite its success in this scenario, we conclude that FS-Merge's true strength lies in more challenging scenarios involving models with different initializations.

[1] Stoica et al., "ZipIt! Merging Models from Different Tasks without Training"

[2] Ainsworth et al., “Git Re-Basin: Merging Models modulo Permutation Symmetries”

[3] Jordan et al., “REPAIR: REnormalizing Permuted Activations for Interpolation Repair”

Thanks for the response. I am willing to keep my score.

We would like to thank the reviewer for the time and effort, as well as the constructive feedback. To address some of the reviewer's concerns, we have added new experiments to our manuscript (Appendix F.5), which are discussed in the response to question 3.

Weakness 1. "“FS-Merge… can be costly compared to other model merging techniques”

As noted in the article (e.g., Table 1), traditional efficient merging methods fail catastrophically in our challenging setting of merging transformers with different initializations, leaving models performing at chance level. Consequently, only resource-intensive non-local methods are effective in this scenario.

Weakness 2. “The figures, such as Figure 3, are low resolution”

We thank the reviewer for this helpful feedback. The figures have been replaced with higher-resolution versions.

“the overall writing quality of the paper is not very professional, requiring significant improvement and refinement”.

We apologize for any difficulties the reviewer experienced with the language of our article. If the reviewer could kindly provide specific examples of unclear sections we will address these in the final manuscript.

Weakness 3. “Performance improvements appear marginal”

In most vision experiments, FS-Merge outperforms the strongest baseline, KD, by a margin of over 4%, and by more than 10% in some cases. To the best of our knowledge, such improvements are typically considered significant in image classification problems.

“The experimental setup seems unique to this paper and not aligned with standard practices in prior literature”

Indeed, this work seeks to broaden the scope of model merging by addressing the challenging scenario of merging transformers from different initializations, an important and practical direction as demonstrated for CNNs [1,2,3]. For instance, merging unrelated models from online repositories like Hugging Face or GitHub often involves models not fine-tuned from the same initial model, rendering most existing techniques unsuitable. Thus, we view our unique setup as a strength, not a weakness.

Weakness 4. “The paper is largely empirical, lacking an in-depth discussion on why the proposed method effectively combines models”

We thank the reviewer for this important note. Most model merging works are indeed empirically focused. As discussed in the article, we believe FS-Merge succeeds where traditional methods fail due to its use of a global objective rather than a local one, along with its ability to employ more complex merging rules. Additionally, FS-Merge outperforms KD by leveraging the knowledge encoded in the original model weights. We discuss some of those issues in detail in Appendix A.2, Appendix G and Appendix H.

Question 1. “What is the training cost associated with different datasets, such as FLOPs?”

To the best of our knowledge, the resources that limit the GPU and the demands of an algorithm are runtime and memory usage. In the case of merging four ViT-B-16 models with 100 original images and 1,000 augmented images per dataset, RegMean takes about 3 minutes and 4,056MB; distillation 1.9 hours and 15,740MB; FS-Merge 3.6 hours and 29,550M; and FS-Merge seq. around 2.2 hours and 19,088MB.

Question 2. “...is similar to adapter-based methods. Could the authors discuss this similarity?”

We thank the reviewer for this suggestion. We have added a section discussing adapter-based methods (Appendix A.1). Both methods inject and train additional weights, but FS-Merge merges knowledge from multiple models, unlike adapter-based methods, which fine-tune a single model. After merging, FS-Merge folds the trainable parameters into the original weights, while many adapter methods retain additional complexity by remaining in the fine-tuned models.

Question 3. “It would be beneficial to include results on larger-scale datasets”

To mitigate this concern, we conducted new experiments, merging models finetuned on SUN397, Food101, and CIFAR100. Combined, these three datasets contain over 290,000 images and 598 classes. This represents the most challenging merging attempt known to us in the transformers merging literature under our setting. FS-Merge outperforms the baselines in this scenario as well (and see Appendix F.5, Table 25). We also wish to emphasize that some of our existing experiments were performed on large or challenging datasets, such as MNLI (392,000 data points, Table 26), QQP (363,000 data points, Table 26), or merging groups of 5 ViT (Table 19).

[1] Stoica et al., "ZipIt! Merging Models from Different Tasks without Training"

[2] Ainsworth et al., “Git Re-Basin: Merging Models modulo Permutation Symmetries”

[3] Jordan et al., “REPAIR: REnormalizing Permuted Activations for Interpolation Repair”

We thank the reviewer for their insightful comments and suggestions. We appreciate the opportunity to improve our manuscript. To address some of the reviewer's concerns, we have added new experiments to our manuscript (Appendix F.3), which are discussed in bullet 2 of this response.

1. “The novelty of the proposed method is still limited…”

We wish to emphasize that a naive version of FS-Merge, which simply trains M and U layers with KD, is insufficient for this challenging setting. Our success stems from key innovations: shifting to a global reconstruction problem, parametrizing M and U matrices as diagonal plus low-rank matrices, leveraging “first” initializations, and using data augmentations. These contributions are discussed in Section 2, Appendix H, and ablation studies (Appendix G). We also proposed a more efficient version (FS-Merge seq.) and evaluated alternative approaches, such as using inner features (Appendix H.3), which did not improve accuracy. Additionally, our work offers the most detailed evaluation to date of merging transformers from different initializations, covering both vision and text domains.

2. “The experiments are performed only with the Vision Transformer with a few downstream tasks…”

Our work is not limited to Vision Transformers; it also includes merging fine-tuned BERT models pre-trained with Masked Language Modeling and Next Sentence Prediction (see Appendices F.5 and F.7, mentioned on lines 313-314 in the main text).

“It should be tested with other initializations rather than just ImageNet-1K pre-trained ones”

To address this concern, we have added new experiments to our work (Appendix F.3, Tables 22 and 23) merging fine-tuned ViTs from different pre-training strategies (CLIP contrastive approach and supervised learning with ImageNet-1K). Again, FS-Merge consistently outperformed the baselines across all pre-training strategies.

3. “Distillation… now has plenty of existing approaches ([2,3,4,5])… the paper should discuss more on the relationship to such works”.

We evaluated many KD approaches (Appendix G.2), including different initializations (random, average, RegMean), inner feature usage (Appendix H.3), and augmentations. The KD method used in our article is the strongest we identified. Regarding those references, [2] is specifically designed for NER, and [5] assumes no access to the original teacher's data, making them irrelevant to our setting. [4] employs inner features, which we found unhelpful (see Appendix H.3). [3] uses the teacher models in a progressive manner, similarly to FS-Merge seq., which has shown a somewhat lower performance compared to the original FS-Merge. Therefore, we believe it is unlikely to outperform our KD baseline. We thank the reviewer for those references, they have been added to our article.

4. “There is a concern about the computational/memory inefficiency…” +
5. “in Table 6… the proposed method seems more than 2x inefficient…”

As explained in lines 466-475 in section 3 and Appendix C:

a) When merging two models, our method is comparable in efficiency to KD.

b) Our method becomes less efficient when more models are merged (e.g. the mentioned result in Table 6 in the appendix merges four models).

c) We provide ways to improve efficiency, such as reducing the rank of M, U matrices and using FS-Merge seq. (Appendix B.4). FS-Merge seq. outperforms KD in accuracy while maintaining comparable efficiency.

Thus, our method involves an efficiency-performance tradeoff but consistently outperforms KD. Moreover, the costly merging phase occurs only once; afterward, all methods, including FS-Merge, produce a model of the original size with the same resource requirements. Additionally, our implementation of FS-Merge could likely be optimized for greater efficiency.

5. “The number of optimized parameters… may be (possibly intentionally) misleading”

We thank the reviewer for this remark. We did not intend to obscure efficiency limitations, as we discussed them in detail in Section 3 and Appendix C, as mentioned above. To avoid confusion, we modified the column from “number of learnable parameters” to “Does the method use learnable parameters” (see tables 3, 4, 5).

We hope these revisions address all the reviewer’s concerns and clarify the intentions and contributions of our work. We thank the reviewer again for their careful review and helpful feedback.

[2] Li et al., "An Unsupervised Multiple-Task and Multiple-Teacher Model for Cross-lingual Named Entity Recognition"

[3] Nguyen-Meidine et al., "Unsupervised Multi-Target Domain Adaptation Through Knowledge Distillation"

[4] Park and Kwak, "Feature-level Ensemble Knowledge Distillation for Aggregating Knowledge from Multiple Networks"

[5] Shi et al., "Data-free Model Fusion with Generator Assistants"

We appreciate the reviewer's insightful comments and suggestions. To address some of the reviewer's concerns, we have added new experiments to our manuscript (Appendix F.4), which are discussed in bullet 1 of this response.

1. “to investigate its performance when applied to pretrained models that are fine-tuned on different sources”

In response to the reviewer's request, we evaluated FS-Merge in the more common scenario found in the literature, where models fine-tuned on different tasks originate from the same pre-trained initialization (Appendix F.4, Table 24). While FS-Merge outperforms most baselines in this setting, its margin over methods like distillation and RegMean is small, with RegMean achieving comparable performance using fewer resources. Thus, despite its success in this scenario, we conclude that FS-Merge's true strength lies in more challenging scenarios involving models with different initializations.

2. “It would be helpful to compare this method to Mixture of Experts (MoE) approaches, such as 'Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM'”

In contrast to the MoE approach, which retains multiple MLP blocks in memory, model merging methods such as FS-Merge fuse these blocks, offering memory and computational advantages. We added “Branch-Train-MiX” and “Robust fine-tuning of zero-shot models” to our references.

3. “Why does distillation perform poorly in the last task (C, M, C100, E) as shown in Table 4, while it performs well in the earlier tasks?”

We thank the reviewer for this important question. Discussion regarding which groups of tasks or models are easier to merge is indeed crucial, and we plan to investigate it in future work. Cars, MNIST, CIFAR100, and EuroSAT involve very different domains, but there appear to be additional complex factors influencing the ability to merge (such as the performances of the models, the tasks' difficulty, and other factors).

4. “Given the use of the ‘first’ initialization, how sensitive is the method to changes in the order of models?”

Based on our limited experiments on the subject, changing the order of tasks does affect the resulting accuracy of the merged model in both FS-Merge and KD. Selecting the order of tasks is a significant and challenging question, which remains an active topic in the fields of continual and curriculum learning [1, 2]. For example, in [1] it is shown that sometimes it is better to start with the easiest tasks, while sometimes it is better to begin with the hardest tasks. Also, in [2] it is shown that a complex interaction between Task similarity and overparameterization can affect the final accuracy. Thus, we believe the complexity of this question warrants a great direction for future work.

[1] Saglietti et al., “An Analytical Theory of Curriculum Learning in Teacher-Student Networks”

[2] Goldfarb et al., “The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model”