FusionBench: A Comprehensive Benchmark of Deep Model Fusion

W1: This work appears to be an engineering effort to generalise existing models and methods within a unified pipeline. However, it lacks scientific novelty and significant technological advancement. Therefore, it may be better suited for applied venues, such as workshops, rather than a top-tier fundamental research conference.

Thank you for your thoughtful review and for highlighting your concerns. While our work does involve engineering efforts to unify existing models and methods, we believe it goes beyond mere integration and introduces significant scientific advancements.

• Our work enables the systematic investigation of model fusion techniques across a wide range of tasks and domains.
• Our work enables reproducible research through standardized benchmarking. This facilitates comparative studies across different fusion approaches.
• Research Impact: There are already several studies that are based on the codebase of FusionBench, including research on more advanced model fusion techniques and investigations into trustworthy model fusion methods.

W2: Besides the above issue, it still seems unclear how the model mixing happens in the background, though they state a list of existing methods they use in Table 1. and W3: Figure 1 is not very illustrative as Fig-1(b) and Fig-1(c) are almost identical. Further demonstration might be required in this regard.

We appreciate the feedback. The complexity of model mixing techniques presents significant challenges for clear visual representation, primarily due to the architecture variability - model mixing methods fundamentally alter the model architecture in distinct ways. Specific Technical Challenges are as follows:

• Depth Upscaling modifies the number of layers in the network resulting in variable depth structures.
• Model Expansion methods alter the dimensional properties of hidden states. This changes the internal representation spaces and introduces new parameter matrices of different sizes.
• Mixture-of-Experts (MoE) Upscaling transforms standard dense layers into complex MoE structures. This introduces routing mechanisms and expert networks. Creates branching architectures that are inherently more complex.

Therefore, creating a unified visual representation would require multiple different diagrams. The readers can refer to the online documentation for more detailed explanations of each model mixing technique in this anonymous link.

W4: In terms of evaluation, it would be more beneficial to see how the proposed system works on large-scale data, such as the ImageNet-1k involved.

We appreciate the suggestion. In fact, it is convenient to conduct evaluations on custom image classification datasets using FusionBench. We provide additional examples of how to configure and run the benchmark on custom datasets in the codebase. Anonymous links to the example configurations can be found here.

In this benchmark, we are currently focusing on evaluating the performance of multi-task model fusion in the in-domain multi-task setting.

W5: I could not find any evidence of the claimed deliverable in the submitted work as stated in the abstract. "...This includes detailed documentation, code examples, and tutorials, making FusionBench a user-friendly and accessible platform for both beginners and experienced researchers."

Due to the double-blind review process, we can not provide a direct link to the online documentation in the paper. Here is the anonymous link to the online documentation for the FusionBench project: https://anonymous.4open.science/w/fusion_bench-gh-page-A360/

We appreciate the suggestions. In fact, we have implemented some fusion algorithms for large language models (LLM) in FusionBench, such as SLERP, DARE-Task Arithmetic, AdaMerging for Llama, Depth Upscaling, and ExPO, etc. Anonymous links to the code are listed as follows:

• SLERP
• DARE-Task Arithmetic
• AdaMerging for Llama
• Depth Upscaling
• ExPO

But the implementation of these methods is not yet fully tested and documented. We will continue to improve the documentation and testing of these methods.

W1: While the authors seem to evaluate against 26 tasks as claimed in the paper, I found the actual number of tasks quite limited. For example, there are 8 different datasets used for image classification, but the authors deem them as 8 tasks. Fairly speaking, they can only be regarded as 8 benchmarks on the image classification task. In that sense, the wordings in such as Table 5 should be seen/unseen domains instead of tasks.

We appreciate the feedback. We have made the necessary changes in Table 2 to the terminology used in the tables to avoid confusion.

• In the context of image classification, we acknowledge that we have used "task" and "domain" interchangeably, which may cause some confusion.
• For text generation tasks, we maintain a more precise distinction where "task" specifically refers to fundamentally different operations - such as question answering, text summarization, sentiment classification, etc.

The term "domain" would indeed be more appropriate when discussing variations within a single type of task, such as open-vocabulary image classification and different applications in mathematics or coding challenges.

W2: For the scene understanding part, the tasks are all limited to the dense per-pixel prediction type (depth, normal, and segmentation), and are only conducted against the NYUv2 dataset. I would recommend adding additional tasks that are common in this area, such as object detection.

Thank you for the suggestions. We are consistently working on expanding the FusionBench to include more tasks and datasets.

Q1: In Table 2, the term tasks seems abused. Technically, there are 8 datasets instead of tasks for the image classification task. The same applies to the text classification and generation task, with 7 and 8 datasets, respectively. The scene understanding does contain 3 tasks (segmentation, depth, and normal). I recommend the authors edit the terminologies used and make the tables clearer.

We appreciate the feedback. We have made the necessary changes in Table 2 to the terminology used in the tables to avoid confusion.

Q2: All the scene understanding tasks are dense (per-pixel) prediction tasks. It would be interesting to consider tasks such as object detection. Also, using the NYUv2 dataset only covers the in-domain multi-task setting and is limited. How about doing a cross-domain multi-task setting? For example, a segmentation model on ADE20K, plus a depth model on KITTI.

We appreciate the suggestions. In fact, many implemented algorithms are model-agnostic and task-agnostic, and can be applyed to object detection tasks with minimal modifications.

Of course, we will also consider adding object detection tasks and cross-domain multi-task settings in the future.

W1: The absence of direct comparisons with baseline models.

We include the performance of task-specific CLIP vision models, GPT-2 models and Flan-T5 models in the supplementary material. It can also be found in our online documentation in the anonymous links.

For example, for CLIP-ViT-B/32 models, here are the results:

W2: A deeper explanation for the categorization of model ensembles, merging, and mixing would help clarify their task-specific benefits.

Model ensemble aggregates the predictions of multiple models to improve the overall performance, while all models in the ensemble are kept separate. Model merging combines the weights of multiple models into a single model, keeping the model architecture the same. Model mixing combines the weights of multiple models into a single model, changing the model architecture. For instance, with depth upscaling, a model mixing technique, the number of layers (depth) in the model differ from that of the original models.

The pros and cons of each techniques:

• Model ensemble: simple to implement, but expensive in terms of memory and computation resources for inference.
• Model merging: no additional cost for inference, but the performance may not be as good as model ensemble.
• Model mixing: can archive better performance than model merging, but it is more complex to implement. Typically requires additional training for performance recovery after fusion.

W3: More insights on scalability to additional tasks or modalities would enhance FusionBench’s applicability to diverse research needs.

That is a great suggestion. The benchmarking suite is designed to be easily extensible to new tasks and modalities, and we will include more tasks and modalities in the future.

W4: FusionBench’s effectiveness depends on high-quality pre-trained models, which might limit performance in under-resourced domains.

We agree that the performance of pre-trained models is crucial for the effectiveness of model fusion. In fact, the idea of model merging is widely used in machine learning to improve the performance of models with limited resources. For example, in each round of federated learning, the weights of the models from different clients are merged and distributed back to the clients. This can also be considered as a form of model merging. Other techniques for optimization such as exponential moving average (EMA), stochastic weight averaging (SWA), and latest weight averaging (LAWA) also use the idea of model merging to improve the performance and generalization of the models.

Dear Reviewers,

We sincerely appreciate you taking the time and effort to review our manuscript. Your insights and feedback are precious to us.

As the discussion phase nears its end, we want to confirm that we have adequately addressed all of your comments and concerns. We have carefully considered each point raised and believe we have provided comprehensive responses. However, if any questions remain unanswered or if there are areas where you feel further clarification or discussion is needed, please do not hesitate to let us know. We are happy to provide additional information or engage in further discussion.

Thank you again for your review and constructive feedback, which has significantly contributed to improving our work.

Best regards,

The Authors