Wolf2Pack: The AutoFusion Framework for Dynamic Parameter Fusion

Question 2: Scaling to Larger Models like Vision Transformers

We acknowledge the importance of scaling to larger models. Due to the limitation of computational resources, we currently introduce only a portion of more complex experimental setups to validate the generalization ability of our method, and in Appendix A we provide test results using Resnet as a baseline network on the relatively complex CIFAR100 dataset:

It can be clearly seen that our method still stably outperforms the baseline method, and a more specific explanation of the experimental setup and results can be found in Appendix E.7. The aforementioned mentioned that we tested the fusion effect on a object detection model, which also belongs to the type of task that pushes the AutoFusion method to more complex tasks, and also showed good results, detailed results are shown in Appendix E.8.

Dear Reviewer aNrj，

We sincerely appreciate the time you took to review our paper and provide valuable feedback. We have carefully considered your comments and made revisions to address each issue raised. Please find our responses to your specific points below:

Weakness 1: Number of Optimized Parameters

Thank you for pointing this out. We have added a table summarizing the number of optimized parameters for the AutoFusion model and the ensemble model. This addition is included in the revised manuscript in Appendix E.9, providing a clearer comparison of computational complexity across methods. And the results are as follows:

You can get information about the setup and interpretation of the form in Appendix E.9.

Weakness 2: Claim on High Similarity Assumption (Lines 215-218)

We appreciate your observation and agree that further evidence strengthens this claim. To address this: We have computed similarity metrics across layers of fusion models, demonstrating the divergence in parameter distributions. These results substantiate our statement and are presented in Appendix F.1 of the revised manuscript.

Furthermore, In Appendix E.8 our newly added experiments for object detection model fusion(First time in the model fusion field) on VOC dataset highlight that retaining task-specific parameters improves multi-task performance, validating the necessity of aligning similar parameters while preserving distinct ones.

And the results are as follows, too, you can get more information from Appendix E.9:

Weakness 3: Absence of Related Work Section

While we appreciate your suggestion to include a separate related work section, we respectfully assert that the current structure integrates related work into the introduction and throughout the paper. This decision was made to focus on highlighting the unique challenges addressed by AutoFusion, such as merging models without shared pre-trained initializations.

The introduction discusses prior works like Git Re-Basin and ZipIt, situating AutoFusion within the broader research context.

Section 2 contrasts traditional model fusion methods with our approach, explaining how AutoFusion handles the absence of pre-trained initializations—a critical and less explored challenge in the field.

Given the page constraints, we chose this integrated approach to avoid redundancy while providing sufficient context for our contributions.

We hope this justifies our structural decision while ensuring the manuscript remains concise and focused.

Dear Reviewer ACFL，

We would like to express our sincere gratitude for taking the time to review our paper and providing valuable feedback. We have carefully considered your comments and have made revisions to address each of the issues raised. Please find our responses to your specific points below:

Weakness 1: Experiments Could Be Improved

1.1 Analysis of Model Similarity：

Thank you for this suggestion. We agree that analyzing model similarity is crucial to understanding the challenges of parameter fusion.

To address this, we have computed similarity metrics via cosine similarity between models trained on different tasks. These results will be included in the revised paper in Appendix E.9.

1.2 Baselines of Fine-tuning Models Jointly on Multi-tasks

Thank you for your suggestion, we also believe that the addition of an ensemble model is necessary, and we have included the results of the ensemble model in our comparison experiments in Section 4.1 - Table 1 - 'Ensemble Model' by red words.

1.3 Comparison to LoRA Fine-tuning

While we acknowledge its relevance as a fine-tuning method, our approach, AutoFusion, fundamentally differs in purpose and methodology. LoRA primarily focuses on parameter-efficient fine-tuning by introducing additional low-rank matrices to existing parameters, while AutoFusion aims at parameter fusion across models trained on disjoint tasks, without introducing additional trainable parameters.

Moreover, AutoFusion does not inherently rely on the concept of low-rank decomposition or optimization for specific parameter subsets. Instead, it employs a differentiable permutation matrix to align and merge parameters dynamically. This makes LoRA's low-rank perspective less applicable as a direct comparison.

Finally, while LoRA is a robust fine-tuning approach, its inclusion as a baseline might lead to confusion regarding the scope of our work, which is centered on unsupervised parameter fusion rather than fine-tuning. For these reasons, we have opted not to include LoRA as a baseline in this study. We hope this explanation clarifies our rationale and aligns with the focus of the paper.

1.4 Comparisons to Git Re-Basin and Zipit in Section 4.3

We have extended the experiments in Section 4.3 to include comparisons to Git Re-Basin and Zipit. The updated results demonstrate that AutoFusion consistently outperforms these baselines on different distributions. Please refer to the revised Section 4.3 Table 3 for details. And the additional results are as follows, too:

1.5 Experiments on Larger Datasets

Limited by computational equipment, we supplemented the fusion results with the ResNet family of models while taking the more complex CIFAR100 dataset into account. Additional experimental results are shown below:

More detailed results can be found in Appendix E.7.

Dear Reviewer Tk42，

We would like to express our sincere gratitude for taking the time to review our paper and providing valuable feedback. We have carefully considered your comments and have made revisions to address each of the issues raised. Please find our responses to your specific points below:

Weakness 1: Incomplete Sentence (Line 225)

Thank you for pointing out the issue in Line 225. We acknowledge that the sentence is incomplete. The complete intended statement is as follows:

"We attempted to utilize neural functional functions from neural functional analysis to predict network parameters from network parameters \cite{navon2023equivariant} \cite{zhou2024neural} \cite{zhou2024permutation}. "

Weakness 2: Evaluation on More Complex Tasks and Datasets

We appreciate the reviewer’s suggestion to evaluate our method on more complex tasks and datasets. To address this, we have conducted additional experiments on object detection tasks using the VOC dataset(Constrained by computing resources). The results are as follows:

We divided the 20 target categories in the VOC2007 dataset into two parts, each containing 10 categories, where Model A and Model B represent the object detection models trained on these two parts, respectively, and the Ensemble model represents the results of training on the full training set. We uniformly use the pre-trained Feature part of VGG16 as the feature extraction network, and the object detection head is constructed using random initialization, and our fusion also only considers the fusion of the object detection head, as can be seen from the table, our method not only generalizes well on the object detection task, but also exceeds the known baseline methods

Additionally, we extended the AutoFusion method to the CIFAR100 dataset, as well as to the more complex Resnet network, obtaining the following results:

These results further demonstrate the robustness and generalization of the AutoFusion framework for complex tasks. For detailed analyses, please refer to Appendix E.7 and E.8 of the revised manuscript, where we have provided additional experimental details and results. We also provide an analysis of AutoFusion's computational efficiency in Appendix E.9. Limited by computational resources, we will be completing richer supplemental experiments next, which may be available in Camera Ready.

Thank you again for the valuable feedback. We hope these additional experiments address your concerns and demonstrate the broader applicability of our proposed method. I also sincerely ask if you have any other doubts and concerns that we can solve to make it possible to improve the score for our paper.

Dear Reviewer,

Thank you very much for your thoughtful review and for the valuable feedback on our revised article. We appreciate the opportunity to address your comments regarding the CIFAR100 dataset and the performance of our fusion model.

We understand your observation that the fusion model exhibits weaker performance on the Joint metric compared to Model A and Model B. We would like to clarify our approach in the context of model parameter fusion. Our method involves directly merging the parameters of two models trained on separate tasks to create a single model that aims to maintain a certain level of competence across both tasks. It is not uncommon for such fused models to experience a reduction in joint performance, as highlighted in several foundational studies in this field [1].

While Models A and B may show superior Joint metrics, it’s important to note that they excel in only one task each, demonstrating limited capability on the other. In contrast, our fusion model seeks to balance performance across both tasks, achieving a more uniform capability that is significantly above the baseline for similar tasks. This characteristic indicates that our approach effectively enhances overall performance, even if the Joint metric does not surpass that of the individual models. Furthermore, we are pleased to report that our ResNet18-based fusion model indeed outperforms both Model A and Model B on the Joint metric, reinforcing the validity of our methodology.

Additionally, in response to your insightful suggestion, we have extended our method to the object detection task using the VOC dataset, and we provide detailed results in Appendix E.8. The outcomes demonstrate the strong generalization performance of our proposed approach, further validating its effectiveness.

Thank you again for your detailed feedback. We believe that these updates will strengthen our paper and enhance its clarity, and we look forward to addressing any further questions you may have.

References:

[1]arxiv.org/abs/2305.03053[accepted by ICLR2024]