Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent

Thanks for your review and detailed comments. We hope the following discussion can address your concerns!

Q1: Based on Tables 1, 2, and 3, the model merging proposal does not maintain the same level of performance as the individually trained ones, and is even worse than multi-task learning in many cases.

A1: Transfer learning has driven the proliferation of fine-tuned models, but deploying separate models for each task creates significant storage and computational burdens. While multi-task learning could address this, it involves costly training and simultaneous access to all tasks. Additionally, determining the optimal data mixture for effective multi-task training can be complex and resource-intensive. Model merging addresses these challenges by compressing task-specific models without requiring access to training data (privacy or copyright). The performance gap between merged model and individual models or multi-task learning is an inherent constraint, as merging multiple trained models into a single model occurs without the benefit of costly computations.

While previous methods focus on alleviating conflicts between tasks, our approach takes a more direct path by establishing the minimization of the gap between the merged model and individual models as our explicit optimization objective (Line 24). By formulating and effectively solving this as a data-free constrained optimization problem, we achieve significant performance improvements. On ViT-L/14, our method reaches 92.6% performance, approaching the 93.5% achieved by multi-task learning—a substantial narrowing of the gap. We have revised the description of model merging requirements in Line 18, and greatly appreciate your suggestion.

Q2: Theoretically, it is also unclear why "only take gradient steps in the direction orthogonal to the shared space" (Line 68) can help us achieve this ambitious goal. The argument is not convincing.

A2: The optimization objective in Eq. (5) promotes orthogonality between task vectors to mitigate conflicts, while multi-task learning similarly emphasizes shared representations. Parameters between similar tasks can be shared (e.g., applying the MNIST task vector improves accuracy on SVHN). Therefore, we propose constructing a shared subspace $S_{share}$ to preserve common representations. By constraining task vector optimization to reduce updates along $S_{share}$, we maintain shared knowledge while minimizing the gap for each task as defined in Eq. (5).

Ablation studies demonstrate a 3.5% improvement with $S_{share}$. Table 7 presents a comparison of different gradient directions, revealing dataset-specific performance variations. Our method achieves significant improvements on the DTD dataset while showing decreased performance on SVHN. This pattern stems from DTD's reliance on rich textural features that are preserved in $S_{share}$. In contrast, SVHN's visual representations differ substantially from other tasks, making the primary components in $S_{share}$ less suitable. This observation is further validated by examining the performance gap between pre-trained and fine-tuned models: SVHN exhibits the lowest pre-trained performance (31.4%) but achieves remarkable results after fine-tuning (97.5%), indicating its strong dependence on task-specific features. In summary, our approach effectively preserves shared knowledge across tasks while achieving optimal overall performance.

Q3: The connection between the motivation (keeping task-specific information) and the method is not very clear.

A3: To isolate task-specific information, the task vector is defined as $\tau_i = \theta_i - \theta_0$ to capture unique characteristics for each task. While preserving task-specific information through simple vector addition is straightforward, the challenge in model merging lies in managing conflicts between multiple tasks. This challenge becomes evident in Figure 1, which demonstrates how performance consistently declines across all merging methods as the number of tasks increases, directly reflecting increased task conflicts.

As shown in Eq. (3), we measure the gap between the merged model and individual models in terms of task-specific losses. To alleviate conflicts, we introduce a modification vector $\Delta$ for each task vector. This leads to our optimization objective in Eq. (4), which aims to achieve optimal cross-task performance by optimizing $\Delta$. Through this optimization process, the merged model approximates the behavior of task-specific models while effectively resolving conflicts. In short, by minimizing our proposed loss function, we ensure the merged model preserves essential task-specific information.

Q1: Taylor expansion may need to point this out and justify.

A1: During fine-tuning, parameter evolution in pre-trained models is frequently minimal, indicating that training remains within the tangent space where Taylor expansion closely approximates network behavior. This aligns with MAP, which examines task vector magnitudes and employs 2nd Taylor expansion to approximate metrics. It provides a formal proof regarding the negligibility of the remainder in Taylor series, and interestingly proposes using linear regression to estimate Hessian. Thanks for suggesting this related work! It strengthens our theoretical foundation, and we include this reference to further substantiate the rationale behind our approach.

We examined the difference between the 1st order Taylor expansion and the original loss, finding them to be within the same order of magnitude, confirming the accuracy of the estimation. Since calculating the gradient $\nabla_{\theta}\mathcal{L}_j(\theta_0)$ requires specific data $\mathcal{D}_j$, we used task vector $\tau_j$ as an approximation. Interestingly, when we attempted to optimize using actual gradients computed from specific data, we observed performance degradation. We attribute this to highly unstable gradients at the initialization, which complicated the optimization process. Thus, approximating the original loss using task vectors appears to be the superior way.

Q2: Whether $\Delta$ is input or how it is initialized.

A2: $\Delta$ is initialized as a zero tensor with the same shape as the task vector.

Q3: Strong baseline methods such as EMR merging, Twin merging.

A3: As discussed in Twin merging [2], they all belong to dynamic merging, which requires additional storage for task-specific modules. Such methods face parallelization challenges during inference, necessitating either dynamic I/O loading of task-specific modules or storing all modules in GPU memory. EMR merging requires priors during inference to load corresponding modules, while Twin merging trains a router using validation datasets to select modules.

Both EMR merging and Twin merging can be viewed as lightweight WEMoE [3], yet they still impose storage demands (2.25× our approach). For instance, EMR merging's proposed mask implementation still uses 8-bit Bool types, and Twin merging's module reconstruction $U\Sigma V$ requires matrix operations that may not reduce peak GPU consumption. Notably, these approaches avoid direct comparison with WEMoE, which is unsurprising. According to the no free lunch theorem, performance increases with the number of retained parameters, with complete task-specific models representing the upper performance bound.

Our approach, by contrast, is a static merging plug-and-play method (like TA and Ties merging) that maintains standard model size and enables parallelized inference. We compare our method with SOTA static merging approaches such as AdaMerging and PCB-Merging. We believe methods should first be classified before conducting fair comparisons within each category. Otherwise, MoE methods will always outperform others simply due to larger parameter count.

Q4: Present the time/space that DOGE requires.

A4: We have reported training time and memory usage in Table 10 of the Appendix, demonstrating remarkably efficient performance with only 121 seconds total training time and a memory usage of 729MB. We will relocate this information to the main text.

Q5: Generalization performance on SUN397, DTD, and Cars.

A5: Based on your request, we conducted experiments evaluating generalization on three unseen tasks when merging five other tasks. The results reveal that SUN397, DTD, and Cars datasets pose challenges for ViT models, while MNIST/EuroSAT show limited generalization to these complex tasks. Despite this, our method consistently outperformed other model merging approaches by a significant margin.

Q6: Is $\Delta$ a task-specific vector?

A6: Thanks for the suggestion. $\Delta$ is a universal modification vector to each task vector. In our experiments, using a universal modification vector yields performance nearly identical to that of task-specific modification vectors, as they are mathematically equivalent when optimizing Eq.(5).

[1] EMR-Merging: Tuning-Free High-Performance Model Merging. NeurIPS 2024.
[2] Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging. NeurIPS 2024.
[3] Merging Multi-Task Models via Weight-Ensembling Mixture of Experts. ICML 2024.

Q1: The omission appears intentional, since the prior work demonstrates superior performance.

A1: Our approach differs fundamentally from [1,2] in both setting and methodology. Our objective is to close the performance gap between model merging and multi-task learning without introducing additional computation and memory requirements—a core and previously unresolved challenge in model merging research.

• Parameter

  MoE architecture preserves MLP layers from each fine-tuned task-specific model and the pre-trained model, while additionally training a router module. In contrast, our merged model maintains a standard model size. The parameter comparison for ViT-B/32 (8 tasks) is as follows: ||Total Parameters| |-|:-:| |Individual|113.45M| |Ours|113.45M| |WEMoE|573.96M|

  The primary motivation for model merging is parameter reduction. If performance were the sole consideration, retaining each task-specific model would be the trivial solution. Our method aims to compress multiple models (whether 8 or even 20) into a single standard-sized model, which aligns with the typical settings in model merging and multi-task learning. As MoE methods (89.4%) exceed the performance upper bound of multi-task learning (88.9%), comparing our approach directly with MoE would be inappropriate.

• Data Requirements

  MoE approaches employ unlabelled test datasets to train the router module, whereas our optimization of task vectors is data-free. The performance benefits from test-time adaptation are self-evident. Merging based solely on model parameters is more practical and represents the focus of most model merging methods.

• Computational Overhead

  Static merging maintains inference costs equivalent to standard models, while MoE dynamic merging consumes more memory and computational resources (router + activated experts $k$). The inference phase memory usage comparison is as follows: ||ViT-B/32 (8 tasks)|ViT-B/32 (20 tasks)|ViT-L/14 (8 tasks)| |:-:|:-:|:-:|:-:| |Ours|963.42MB|963.42MB|3772.63MB| |WEMoE|2750.65MB|5346.00MB|10063.64MB|

  Similarly, test-time adaptation incurs additional training costs, while our method requires only lightweight training overhead (as shown in Table 10 of our paper):

  	ViT-B/32 (8 tasks)	ViT-L/14 (8 tasks)	ViT-B/32 (8 tasks)	ViT-L/14 (8 tasks)
  Ours	729MB	2448MB	2.02min	5.18min
  WEMoE	3744.19MB	24535.53MB	7.07min	56.84min

  Notably, our method can be trained layer by layer, enabling model merging for large models with minimal memory requirements.

• Regarding Figure 3

  Figure 3 visualizes task vector magnitudes, highlighting a phenomenon inherently observable across domain benchmarks. E-WEMoE and DOGE propose different approaches to address this phenomenon. Figure 3 was drawn with assistance from E-WEMoE authors to create a new version. Associating performance gaps with non-citation introduces a conceptual misunderstanding, as fair comparison is impossible due to differing settings. Meanwhile, we compare our approach with state-of-the-art methods in both data-free and test-time adaptation scenarios (described in lines 314-328). We appreciate your feedback and will introduce MoE-like methods and the clear differences.

Q2: Each dataset should have a description. Most of the included datasets focus on classification. Can this approach be applied to heterogeneous MTL?

A2: We will add descriptions for each dataset. Model merging in CV indeed focus primarily on classification tasks, following common experimental settings (as acknowledged by Reviewer 97J9). Research on heterogeneous model merging remains limited, with existing work mainly centered on VGG and ResNet architectures using CIFAR datasets. We would welcome suggestions for appropriate benchmarks to explore this direction.

Q3: SVD is computationally expensive. Can this approach be applied to Llama 2 or Llama 3?

A3: SVD computation only needs to be performed once at the beginning. As shown in Table 10, which details the computation overhead, our approach requires minimal memory and time. We conducted experiments following standard LLM settings, completing the merging in 58 min on a single A100 GPU. We report normalized scores on merging WizardLM-13B (Instruction-Following), WizardMath-13B (Math), and llama-2-13b-code-alpaca (Code). Our method achieves optimal average performance across tasks.

[1] Merging Multi-Task Models via Weight-Ensembling Mixture of Experts. ICML 2024.
[2] Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging. ArXiv 2024.

Thanks for your detailed comments. We hope the following discussion can address your concerns!

Q1: Some relevant work that tackle model merging on subspace need to be discussed.

A1: Thanks for suggesting additional relevant work. We will discuss them in related work: TSV [1] aggregates task vectors within their subspaces via low-rank approximation and whitens matrices to minimize interference. KnOTS [2] aligns representation spaces between LoRA models using SVD, enabling the application of merging methods. Both these methods and ours recognize parameter low-rankness and implement merging within subspaces.

Q2: Theoretical Rigor: The rationale behind using $−τ_j$ as a proxy for the gradient, could benefit from a more rigorous treatment.

A2: Under the Neural Tangent Kernel assumption (i.e., fine-tuning often occurs in a linear regime), which has been validated in prior work [3,4], $\nabla_ {\theta}\mathcal{L}_ j(\theta_ 0)$ can be estimated as ${k\tau_ i}$ where $k < 0$. Here, $\tau_ j = θ_ T - θ_ 0 = -\sum_ {t=1}^T\alpha_ t\nabla_ {\theta_ t}\mathcal{L}_ j(\theta_t)$, where $\alpha_t$ represents the learning rate and $T$ denotes the update iteration. Given the linearity of parameters $θ_0$ in the vicinity, we have $\nabla_ {\theta_t}\mathcal{L}_ j(\theta_t) = \nabla_ {\theta_0}\mathcal{L}_ j(\theta_0)$. Therefore, we derive $\nabla_ {\theta}\mathcal{L}_ j(\theta_0)=-\frac{τ_j}{\sum_{t=1}^T\alpha_t}$.

Q3: Hyperparameter Sensitivity: The method involves several hyperparameters (e.g., the subspace basis size $k$ and global scaling factor $\eta$) whose selection may critically affect performance.

A3: We have conducted experiments on the subspace basis size $k$ in Figure 4, which displays performance with varying rank ratios alongside the explained standard deviation. We also investigated the relationship between different projection directions and basis sizes. Additional sensitivity analysis for the global scaling factor $\eta$ is supplemented as follows:

The evaluation of across values from 0.01 to 0.09 demonstrates that performance remains stable and even achieves higher results. (We did not conduct a specialized grid search, this setting was chosen because the calculated $\lambda$ was close to 0.3). This consistency across different $\eta$ values verifies the robustness of our approach and highlights the practicality of applying task-aware coefficients.

Q4: Computational Overhead: A deeper analysis of the additional computational costs (e.g., due to SVD and projection operations) would enhance understanding of the method’s scalability.

A4: We have reported training time and memory usage in Table 10 of the Appendix, showing an efficient total training time of only 121 seconds and memory usage of 729MB. The SVD operation only needs to be executed once at the beginning, with a computational complexity of $O(\min(mn^2, m^2n))$. We appreciate your reminder and will relocate this to the main text. Moreover, we supplement the final version with convergence loss curves for 8 and 20 tasks, showing that convergence is typically achieved within 100 to 200 iterations.

Q5: Clarity in Derivations: A step-by-step explanation with more intuition would improve readability. For example, ∥θ∗ − θi∥_Gap apprears in Eq. (2) without introduction.

A5: We apologize for any confusion caused. Eq. (2) is a brief mathematical summary presented before the detailed methodology. We have revised it and explained each symbol's meaning. Combined with the proof presented above, this will enhance the overall clarity of the derivation.

Q6: It would be beneficial for the authors to include a discussion on potential limitations and directions for future research.

A6: Thanks for your suggestion. A potential limitation is the lack of consideration for heterogeneous model merging, which requires transformation when task vectors have inconsistent shapes or layer numbers. Regarding future research, we are extending our work to LLMs by merging WizardLM-13B, WizardMath-13B, and llama-2-13b-code-alpaca, achieving SOTA performance. For detailed table results, please refer to our response to Reviewer QBR6.

Q7: Should I expect gradient-based MTL methods can outperform task arithmetic.

A7: Yes. Task arithmetic implements MTL in a training-free manner and can be viewed as a post-transfer approach for existing models, while MTL methods typically serve as performance upper bounds that we aim to approach.

[1] Task Singular Vectors: Reducing Task Interference in Model Merging. CVPR 2025.
[2] Model Merging with SVD to Tie the KnOTS. ICLR 2025.
[3] Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. NeurIPS 2023.
[4] A Linearized Framework and A New Benchmark for Model Selection for Fine-Tuning. ArXiv 2021.