Realistic Evaluation of Model Merging for Compositional Generalization

Thanks for the detailed review, your questions have highlighted a lot of interesting future research directions and some places where additional ablations and explanations would be helpful.

We agree that a main goal of the paper is to help a user understand when they should reach for model merging as a solution for their problem, however, we disagree that the restricted scope of our benchmark is harmful to this goal. Rather, we think that our decisions, which were made in an attempt to meet the users where they are, makes our study more useful. In this work, we aim to compare merging methods in realistic settings while the majority of your questions and concerns are about quantifying the differences when merging in different settings.

For example, your suggestion of using more than one architecture per modality is a good one from the perspective of having more data points helps draw more robust conclusions and the differences in “mergeability” of different model types is an interesting question, but the models and architectures we selected are the ones that are currently being used in each setting. Transformers are the de facto standard for NLP tasks, diffusion models are the architecture of choice for image generation, and many vision transformers are the state-of-the-art architecture for image classification.

Additionally, the specific models we used are ones used in previous works on model merging. Those results may address the issues you raise. For example, you ask if the “Is t5 just poor at merging or is it actually the difficulty of the task?” Previous work, such as TIES [1], merge monolingual T5 models and achieve strong multitask performance. This supports our suggestion that the difficulty is in the setting itself as opposed to issues with the model itself. Similarly, you mention the same thing again when you say. “The authors attribute the negative slope in in- and out-of-domain correlation to modality in line 70 and "cross-domain" vs. "cross-lingual" generalization in lines 306-319, but this seems to ignore a potential dependence on model architecture / pretrained parameters.” Previous work having success with merging T5 models suggests the new “cross-lingual” differences are where the difficulty comes from. As a follow on point, you note that “I'm not sure "cross-domain" and "cross-lingual" are quite the right characterizations here; surely there are some domain generalization settings where merging will also have weak generalization performance”. We agree that there could be cross-domain pairs that could be more challenging for merging. Our conjecture that cross-lingual seems harder on average doesn’t preclude the existence of hard cross-domain transfers in vision settings, but we conjecture that cross-lingual generalization could be considered an especially challenging variant of "cross-domain" generalization (though settling the debate as to whether a shared multilingual representation is possible and attainable is a longstanding debate that we can't hope to settle in our work). You also suggest that difficulties could be a quirk of the “specific cross-lingual benchmark the authors set up.” We would argue that the size of this NLP benchmark provides credence to our claims: We use 5 different datasets representing 5 different tasks with an average of 4 languages per datasets. This is much larger than previous work such as [2] and [3] which use just 1 and 3 multi-lingual datasets respectively. Overall, we agree with your point that it would be valuable to confirm that within-language cross-task generalization can be attained by merging and will add an ablation experiment of merging t5 models across domains, but within the same language, to address this.

Your concerns about the size of the NLP dataset and your suggestion of “testing more than one dataset per modality” dovetails nicely with the second main point of the paper—namely, that previous work focuses on held-in performance, but going forward merging methods should explicitly consider compositional generalization. Prior to our work, there has been a lack of benchmarks for testing compositional generalization abilities, and we hope that our work spurs the development of even more.

This second point of the paper is closely related to your question, “Can you better characterize when we expect merging to enable generalization vs. not?” It is still an open question and it currently isn’t clear when merged models will be effective at compositional generalization. That’s why we argue that merging methods should be explicitly evaluated on their compositional generalization going forward.

Thanks for the review. Your questions helped find places where we can lay out our intentions and takeaways more explicitly. We’ve answered your questions directly here and have updated our paper accordingly.

The messages this paper aims to deliver to the community are not super clear to me. Do we already believe model merging is the proper way toward a multitasking model and do the authors suggest it is a promising approach or not?

Our findings confirm that model merging is a promising approach towards building a multitask model. We find that model merging performance can approach the performance of multitask models on held-in tasks in some settings. Additionally and importantly, we find that model merging can achieve better compositional generalization than multitask training. However, the absolute performance of compositional generalization is still quite poor compared to the best-possible performance. These trends indicate that model merging is a promising direction for developing performant multitask models but that more work is required to realize merging's full potential.

Title: I'm not super convinced that "compositional generalization" is the "realistic" goal for model merging. Many times, model merging might not be for the emerging capability of several different tasks, but just to improve on the same held-in tasks such as the original motivation of Model Soup etc.

Thanks for pointing out an unintended reading of our title. We don't mean to imply that compositional generalization is the only realistic application of merging. Instead, we aim to highlight that our evaluation setup provides a realistic evaluation of model merging for compositional generalization, which was underexplored in past work compared to single-task (ensemble, as in model soups) or multitask performance. We do agree that held-in task performance is an important goal for many users of model merging, and indeed we therefore included the held-in task performance of each merging method in all of our setups. However, given that merging shows promise for compositional generalization and can even outperform explicit multitask learning, we highlight compositional generalization as an important focus for evaluation. Please let us know if you think we could amend our title to make this more clear.

As for a conference-level study paper, I would expect it to reveal more surprising conclusions, insights, or theoretical motivations. Otherwise, it feels like this paper leans towards a workshop paper or a survey-style report.

A primary contribution of our paper is to provide a much-needed rigorous evaluation of merging methods for the promising and important application of compositional generalization. Our findings include the surprising demonstration that held-in performance and compositional generalization can sometimes be inversely correlated, reinforcing the need to explicitly test compositional generalization abilities. Thus, when developing well-performant merging methods, the community should not just optimize for held-in performance, but for compositional generalization as well. In addition, we highlight major gaps and differences in merging methods that past papers have glossed over or omitted, such as practical requirements for applying merging and computational costs. To the best of our knowledge, all of these aforementioned insights are novel and we are optimistic that our paper will significantly shift the field towards more realistic and reliable evaluations.

Thanks for the review! Your questions helped highlight places where our paper is unclear. We’ve answered your questions directly here and have updated our paper to provide more clarity for future readers.

The paper would be stronger if it benchmarked different base models on each task in order to make sure the core results generalize (especially different architectures, such as Transformers and CNNs on the vision tasks or Transformers and SSMs on language tasks).

We agree that it would be ideal to test multiple models in each setting, but we chose to design our benchmark around a single model for a few reasons: In modern practice, state-of-the-art models for each setting are remarkably and increasingly uniform—i.e., it is widespread to use transformers for image classification and NLP and to use a diffusion model for image generation. The specific models we used in each setting are the same as those used in past merging evaluations: the CLIP vision encoder is the same model as the 8-task vision benchmark from [1] and the mT5 model used for the language experiments is from the same model family as the 7 and 8 task NLP benchmark used in [2]. The trends we notice for the held-in setting matches the trends in the current evaluation benchmark landscape as reported in [3]. Including additional models in each setting increases the computational cost of the benchmark, making it less likely to be adopted. We consider it likely that the insights gained on standard Transformer architectures will be similar on less popular architectures However, if there is a specific architecture you think would lead to different insights and/or would like to see results on, please let us know and we will do our best to evaluate it and include it in an updated draft.

It feels like a baseline is missing. To evaluate generalization error on held-out tasks, one could also evaluate a single fine-tuned model (one of the models being merged) on the held out task. For instance, maybe a model fine-tuned on English Question-Answering would do better on, say, Arabic Question-Answering than some of the merged models (or better at least than the “pretrained” baseline). This would amount to a comparison between merging methods vs. fine-tuning a single model on a task that is closely related to the target held-out task. While this is not necessarily the most important baseline out there (I’m sure the top merging methods would surpass it), it would nevertheless be nice to know how this performs, if the authors have to bandwidth to add it. If the baseline always performs worse than the simple “pretrained” baseline, it could simply be left out of the results figures.

Thank you for pointing out this possible baseline. We chose not to include the "individual model generalization performance" for a few reasons: As you suspect, in the settings we consider, a single-task fine-tuned model typically generalizes worse to unseen tasks than the underlying pre-trained model. This is due in part to the familiar problem of catastrophic forgetting and the fact that the base pre-trained models we consider are already reasonably competent at the target tasks. While there have been some works showing beneficial cross-task generalization of single-task models (e.g. [4] showed that a language model fine-tuned on CosmosQA generalized well to many unseen tasks), this rarely happens when there is a substantial shift between the held-in and generalization tasks (as in the English to Arabic QA example you proposed; see e.g. [5], where it was shown that same-task-different-language can be especially harmful). This baseline either could be reported as the "average performance of held-in task models on generalization tasks" (which, as discussed previously, would result in poor performance) or "best-case performance among held-in task models on generalization tasks". This latter option requires access to generalization task data for evaluation, which makes it an unattainable "oracle" baseline in practice. Since the held-out task models themselves are likely a stronger baseline, we consider them more important to include. We hope this clarifies why we didn't include this baseline. We will add some discussion of it to the paper to clear up further questions.