Response to Reviewer LqLU

Q1. Lack of technical details on the router (Model architecture and training configurations ), validation set for router training. Does the validation set consist of the integration of downstream tasks? or general text corpus?

Thanks for your suggestion, we will revise our paper in the next version. Our router is implemented as a three-layer linear network with Leaky ReLU activations and batch normalization. We train the router on the validation dataset with a learning rate of 5e-4 for 10 epochs.

The validation set consists of the integration of in-domain downstream tasks, not the general text corpus. The validation set is taken from a split of the training set, and we use at most 1,000 items for router training for each task.

Q2. how the construction of the validation set affects the final merged model performance.

As for how the construction of the validation set affects the final merged model performance, if the dataset is too small or imbalanced, it can affect the router's precision and degrade the merging performance. Therefore, we typically ensure the sample numbers from different tasks are the same. Practically, using 1,000 items (which is about 2% of the discriminative task test sets) is sufficient to achieve comparable performances.

Q3. They only validate it on the parameter-efficient fine-tuning regime for the generative tasks. They should show the applicability of their method on fully fine-tuned model merging.

We primarily merge using LoRA for the Qwen-14B models because fully finetuning them to obtain task-specific experts would require a huge amount of resources and computation (finetuning the full 14B model requires at least 8 A100 GPUs). However, to further demonstrate, we provide experiments on fully fine-tuned LLaMA 7B models for generative tasks (gsm8k and truthfulqa), where our approach still exhibits superior performance.

We have also verified the effectiveness of our method on fully fine-tuned RoBERTa (as shown in Table 2) and ViT-B/32 models (in the global rebuttal section).

Q4. I speculate that the amount of computation for the merging operation is still a huge burden. The input-dependent dynamic merging requires computations for merging operation per every sample. I wonder about the runtime and computation cost during the inference phase when the fully fine-tuned models are the target of validation rather than the parameter-efficient fine-tuning regime in Table 7.

Please refer to the analysis in the "Inference Efficiency" section of the global rebuttal. Our approach introduces negligible cost in typical generation scenarios and can be easily optimized by group-wise merging.

As for runtime and computation costs for fully fine-tuned models, please refer to the LLaMA 7B time cost in the response to Q3 (186s->198s) and the ViT-B/32 results in the global rebuttal (47m22s vs 215m01s).
From these tables, we can see that our method introduces only an extra 0.01 seconds per sample for LLaMA7B and is faster than methods like AdaMerging and Surgery, which also improve performance through additional training and modules.

I appreciate the authors' kind response! Some of my concerns are addressed.

• I still think that TwinMerging's inference time overhead compared with the vanilla task arithmetic in Table 2 of global response is significant (about two times), given that group-wise extension is not included in the reviewed manuscript.
• Moreover, as from your answer, TwinMerging requires labeled samples from ALL test domains in advance (even though it is validation splits), which is an optimistic assumption compared with AdaMerging, which only requires unlabeled test domain data.

Thanks for your detailed rebuttal.

Authors' claims are convincing and I will raise my score 5 -> 6

I want to express my severe gratitude for the authors' kind response and my regret for my unprofessionalizm, such as the score-reversing behavior.

Thanks to the discussion with the authors, I could also extend my sight, not only gain a clearer understanding of the paper.

I will raise my rating accordingly.

Response to Reviewer 2gMR

Q1. Why this is a viable alternative to just keeping the base model and LoRA low-rank vectors? Why we would use this method over keeping separate task low-rank adapters and just using these at test time ?

Table 2 has proven that our knowledge modularization technique outperforms directly using task-specific LoRA at test time in generative tasks, exhibiting an average score of 102%. Compared to several common strategies to utilize task-specific adapters, Twin Merging is still a promising method to improve performance:

• Directly Route to Top-1 Expert, which is equivalent to the fine-tuned baseline in Table 2: This is the simplest but highly impractical approach, as it relies heavily on the router. It has the worst expected performance for out-of-domain data when the router cannot properly predict. Twin Merging outperforms this method (102.38 > 100.0) and has superior performance on unseen tasks (Table 5), as it can benefit from complementary knowledge from different exclusive sources. Additionally, routing to the top-1 expert requires storing all experts, which can be a large storage requirement when used with fully fine-tuned models.
• Combining Multiple Experts: Without the isolation of shared and exclusive knowledge, combining multiple experts suffers from interference problems, as the redundancy of shared knowledge may obscure the key knowledge required for tasks, and conflicts between the exclusive knowledge are also unavoidable (analyzed in Section 3.2).
  • Statically Combining: This is equivalent to "Task Arithmetic" when using LoRA. Twin Merging outperforms static combination (102.38 > 96.61).
  • Dynamically Combining: This is equivalent to the method (A) in the following table. Twin Merging(B) outperforms dynamic combination (102.38 > 97.03).

Q2. Baselines like DARE and Task Arithmetic don’t enjoy the expressivity that test time adaptation + extra memory gives them.

We add baselines like AdaMerging and Representation-Surgery that have extra time cost and memory consumption in the "More Baseline" section of the global rebuttal. While consuming less time cost and memory consumption, our approach superior in performance (95.33 > 94.04 > 88.50). As analyzed in the "Inference Efficiency" section of the global rebuttal, our method actually introduces neglectable cost compared to the total generation (0.039s per sample in Table 7), and can be further optimized by the group-wise merging.

Q3. Twin Merging Performance in Table 2 is much more significant than in Table 5 with unseen generalization.

This is because we present unnormalized scores in Table 5. We cannot directly normalize them as we do not have the corresponding expert on unseen datasets to get upper-bound performances (as noted in line L242). This leads to relatively lower scores due to the narrower score ranges for tasks like RTE (max 66.43 vs max 91.71 for QNLI) and MMLU (max 68.03).

If we use the maximum score from Table 2 as the upper bound for normalization, we observe more significant improvement for Table 5, e.g. 91.16 -> 96.98 for MMLU (We present the "unstrictly-normalized" scores in parentheses):

The scores are lower than in Table 2 because the in-domain knowledge may be unrelated or even harmful to the test data, leading to a performance downgrade. However, our approach helps mitigate this effect by dynamically adjusting the merging weight.

Q4. How are the $v_t$ represented/stored? Are the $v_t$ stored as the low rank matrices? It seems size of $v_t$ is equal to the size of the full parameter space.

We do not directly store $v_t$ since it is the same size as the original parameter. As detailed in Appendix D5, we further compress $v_t$: Given the size-$m$ decomposition $v_t = \mathbf{U}_t \mathbf{\Sigma}_t \mathbf{V}_t^T$, we select the top-$r$ singular values to form $\mathbf{U}_t(r) \mathbf{\Sigma}_t(r) \mathbf{V}_t(r)$. We store only $\mathbf{U}_t(r)$, $\mathbf{\Sigma}_t(r)$, and $\mathbf{V}_t(r)$.

During merging, we decompress these matrices by extending $\mathbf{U}_t(r)$, $\mathbf{\Sigma}_t(r)$, and $\mathbf{V}_t(r)$ to size-$m$ by filling with zeros, allowing us to recover $v_t$ via their product. This operation is only at the matrix level; once we obtain the merged matrix, we discard the decompressed matrices, ensuring efficient storage.

Q5. For Figure 4, is the fine-tuned storage size much larger because you are doing full fine-tuning ? If you are doing LoRA fine-tuning is the gap because the rank of the lora matrices are larger than the rank of the final $v_t$ ?

Yes, we mainly show the full fine-tuning results for RoBERTa in Figure 4. As analyzed in Appendix E, for LoRA fine-tuning, the typical rank is 32, but in our experiments, we can use a rank-1 for the best storage efficiency and still obtain good results (refer to Table 9), which is much smaller in storage.

Q6. Where is the "Best Storage" option

The "Best Storage" option refers to the rank-1 compression via SVD, as illustrated in Table 8 and Table 9. The detailed storage analysis is provided in Appendix E.

Q7. Showing the additive results for the Ablation Study

Thanks for your suggestion. We show the additive style ablation as follows:

Response to Reviewer Q3uE

Q1. The design does not support batching

While the router process supports batching, the dynamic merging process currently handles inputs sequentially. However, it is straightforward to extend the merging process to support batching. As detailed in the "Inference Efficiency" section of our global rebuttal, we can achieve this by clustering and rearranging data items based on the router logits into groups. This approach significantly reduces the number of merging operations required, with minimal impact on performance. By doing so, we maintain the benefits of our approach while efficiently processing inputs in batches.

Q2. Does every input requires two forward passes (through the fuser and through the merged model) ?

For the fuser, yes, it requires one forward pass to induce the merging weights. However, the merging model typically requires hundreds of forward passes for generation (e.g., 300 tokens for summarization). Therefore, the additional cost is typically negligible, referring to analysis in the "Inference Efficiency" section of the global rebuttal.

Q3. The basic task arithmetic (A) already performs some form of modularization where the pretrained model is the shared knowledge and task finetuned models are task-specific. Knowledge modularization(B) redefined the task vectors relative to the merged model and compressed them via SVD. Comparing A and B would be a better ablation experiment than the one in Table 6.

Thank you for your suggestion. We have revised the ablation study table as shown below:

We observe that A performs worse than B (85.90 vs 96.14), which can be attributed to two main reasons:

• The pretrained model may contain relatively sparse shared knowledge that benefits the input tasks. In contrast, the shared expert, constructed by merging task-specific experts, contains more abundant and diverse shared knowledge.
• Modulizing knowledge by subtracting the pretrained model does not effectively mitigate interference, as it does not consider exclusive knowledge specific to each task. This explains the performance gap between the task vectors and the fine-tuned experts, as analyzed in Section 3.2.

Q4. Compare to AdaMerging which assumes extra data is available.

Thank you for your suggestion. We add AdaMerging which need extra validation data in the "More Baseline" Section of the global rebuttal. Our approach still outperforms them with less time costs (95.33>88.50).

Q5. They do not really make assumptions on whether these parameters were trained jointly/with overlap. therefore I'm not sure how the LoRA experiments contradicts this insight: Even if the LoRA modules are trained separately, they may still have redundant statistics at the end of training ?

To clarify, our study focuses on the "parameter interference" phenomenon as defined by the Ties-Merging paper[1]. This refers to the conflict of parameters at the same position across task experts, e.g., the sign disagreements of the up-projection layer in different task models, which is the main focus of the Ties-Merging method. Our Section 3.1 experiment demonstrates that even when task-specific modules are trained without overlap thus merging without overlap, interference still occurs. This indicates that Ties-Merging approach does not fully resolve parameter interference.

Q6. The notion of similar tasks/task interference is hard to define in general. So it is not clear to me how significant the results of the XSUM/DailyMail experiment are.

We use "task interference" from MTL literature [2] to describe the distinct nature of different task types. For instance, summarization, math reasoning, and code generation each require different forms of responses. Conversely, XSUM and DailyMail are both summarization tasks, handled similarly by the model. Our experiments showed that even similar task types experience interference, prompting us to explore finer-grained relationships between tasks, such as the knowledge types (Sec 3.2).

Q7. It seems that most of the time the samples are routed to respective task-specific experts. If that insight is true, it is not too surprising that TwinMerging performs on-par with the finetuned baseline.

To clarify, we do not directly route samples to task-specific experts (equivalent to the fine-tuned baseline in Table 2), which is highly impractical when facing unknown test distributions. Instead, we combine shared and exclusive knowledge, which leads to even better performance sometimes, e.g., averaging 102% over the fine-tuned baseline on generative tasks (Table 2) and 101% on COLA (Table 8), and better unseen generalization (Table 5). By isolating different types of knowledge and composing them dynamically, we avoid redundancy and leverage complementary information from both shared and exclusive sources, resulting in improved performance.

Q8. Shouldn't Model Merging and Twin Merging also integrates the cost of training the initial task vectors in training cost ? Otherwise it seems unfair to the MTL baseline.

We did not include training time in Table 7 because merging methods can directly download task experts from Hugging Face/PyTorch Hub without post-training. However, to demonstrate, the training time and cost for custom fine-tuning from scratch are as follows:

Our approach shows an 8.5% performance improvement over MTL with only a 0.1% increase in training tokens and a 0.4% increase in training time.

[1] TIES-Merging: Resolving Interference When Merging Models [NIPS23]

[2] Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitives [CVPR23]

Response to Reviewer vkK5

Q1. The proposed method dynamically merges the models with the varying inputs, which can be extremely time-consuming in practice.

Please refer to the "Inference Efficiency" section of our global rebuttal. Our method achieves superior performance (95.33 vs 94.04) with less time and storage cost compared to AdaMerging and Surgery baselines (47m22s vs 215m01s, 5.0GB vs 32.4GB). As shown in Table 7, our approach adds only 0.039 seconds per sample while improving performance by 28.34% for discriminative tasks. Additionally, our method supports common inference speedup techniques and offers efficient group-wise merging variants.

Q2. In Figure 3, Why 8-model merging is used for comparisons? Why not 2-models merging is compared?

We chose to highlight the 8-model merging scenario to illustrate the degradation gap that can occur in practical scenarios, as typically merging multiple models is more common. We actually have demonstrated the performance for 2 to 8 model merging in the left figure of Figure 4.

Q3. More works (MuDSC and REPAIR) in relative works

Thank you for your suggestion. MuDSC and REPAIR can be categorized as Linear Mode Connectivity (LMC) based methods, which we have discussed in the relative work section (L79-L82). MuDSC addresses the issue of inconsistent similarity calculations in activation and weight spaces by designing a merging framework with dual-space constraints to ensure high similarity in both spaces between models. REPAIR addresses the problem of collapsed variance in activations during interpolation by rescaling pre-activations, thereby mitigating performance degradation.