Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts

Q: Orthogonal components may introduce task-irrelevant noise and harm to model performance.

A: Thank you for your insightful question. Intuitively, parameter changes in directions orthogonal to the gradient directions are expected to introduce less task-irrelevant noise than changes in the directions parallel to the gradient.

We empirically validate if TATR reduces interference between tasks by comparing TATR with TA when a new task vector is introduced to the mix of task vectors. Specifically, we quantify the test accuracy changes for a given task between incorporating and excluding a task vector. Each table entry in the following (corresponding to column task $j$ and row task $i$) displays the difference in task $j$ accuracy when including all task vectors versus excluding the vector of the task $i$., i.e., $ACC_j( \theta_{MTL}(\Delta_1, \dots, \Delta_K) ) - ACC_j( \theta_{MTL}(\Delta_1, \dots, \Delta_{i-1}, \Delta_{i+1}, \dots, \Delta_K))$.

TATR has a notably smaller interference between tasks compared to TA. This confirms that TATR introduces limited task-irrelevant noise.

Q: Why does TATR perform poorly on the Cars dataset?

A: We notice that the Cars dataset is challenging for all TA-based methods. For instance, under the ViT-B/32 architecture, even AdaMerging++, a test-time training-based approach, struggles to outperform simpler non-TA-based methods like Fisher Merging.

A possible explanation is that the Cars dataset has larger knowledge conflicts than other datasets. As highlighted in Figure 4 of the manuscript, the knowledge conflict of Cars is markedly higher than that of other datasets. This heightened knowledge conflict guides TATR to mask a substantial knowledge of the Cars task vector, inadvertently discarding critical task-specific information. As a result, the merged model struggles to perform well on the Cars dataset.

Q: The authors should highlight the best and second-best performance values, and indicate the number of datasets on which the proposed method achieved the best.

A: Thanks for your suggestion. We have uploaded a revised manuscript where these changes are made.

Q: What are the differences and new insights of knowledge conflict between existing research [1-5]?

A: Thank you for the important question. Works such as [1-4] address negative transfer focusing on the dynamic gradient interference during multi-task training. Our work addresses model merging from static task vectors. For example, the large step sizes in task vectors introduce a unique set of problems not present during multi-task training. We have added the citation of these papers in the discussion of the second paragraph of the Introduction of our manuscript.

[5] is contemporaneous with our work. It will be published at NeurIPS 2024 on 10 Dec 2024, which is after the ICLR submission deadline (01 Oct 2024). In addition, [5] is substantially different with our work in terms of methodology and objectives. [5] eliminates components that have small products between two task vectors but TATR masks components with big absolute products between task vectors and gradients. In terms of objectives, [5] tends to make task vectors aligned, whereas we aim to make them orthogonal. Beyond the difference in objectives, our approach is a lightweight, plug-and-play module, distinguishing it from the method in [5], which contains multiple techniques for test-time adaptation.

We hope this response clarifies the unique contributions of this work, and will incorporate these comparisons in the revised version. Thank you again for bringing this to our attention.

[1] Yang, Chenxiao, et al. "Cross-task knowledge distillation in multi-task recommendation." Proceedings of the AAAI conference on artificial intelligence. Vol. 36. No. 4. 2022.

[2] Meng, Ze, Xin Yao, and Lifeng Sun. "Multi-task distillation: Towards mitigating the negative transfer in multi-task learning." IEEE International Conference on Image Processing (ICIP). 2021.

[3] Liu, Bo, et al. "Conflict-averse gradient descent for multi-task learning." Advances in Neural Information Processing Systems. 2021.

[4] Wang J, Ren Y, Song Z, et al. "Hacking task confounder in meta-learning". IJCAI. 2024.

[5] Guodong, D. U., et al. "Parameter Competition Balancing for Model Merging." Advances in Neural Information Processing Systems. 2024.

Thank you for your response, which has addressed most of my concerns, especially regarding the differences about the knowledge transfer concept and methods with prior works (W1 for motivation and concept overlap). Please ensure that the additional experimental results and explanations are incorporated into the work. I am happy to raise my score accordingly.

Q: Robustness analysis for different sample sets.

A: To evaluate the robustness of exemplar selection, we repeated the TATR experiment five times using randomly selected exemplar sets. The table below presents the mean and standard deviation over five runs with varying numbers of examples, evaluated on both the ViT-B/32 and ViT-L/14 architectures. As can be seen, TATR demonstrates consistently robust performance. Notably, the model achieves both strong performance and robustness with just 16 samples.

Q: Why is there a difference in performance between ViT-B/32 and ViT-L/4?

A: The differences in performance between ViT-B/32 and ViT-L/14 can be attributed to the higher capacity of ViT-L/14. ViT-L/14 features a larger number of parameters and a smaller patch size (14x14 vs. 32x32), enabling finer-grained feature extraction and richer task-specific representations. These properties mean that different task vectors for ViT-L/14 may not conflict as easily.

This inherent strength of ViT-L/14 limits the room for improvement from model merging methods. For instance, methods like Ties-Merging see a gain of 3.3% on ViT-B/32 (from 69.1% to 72.4%) compared with TA, but only 1.5% on ViT-L/14 (from 84.5% to 86.0%). Similarly, Surgery achieves 11.8% improvement on ViT-B/32 (from 69.1% to 80.9%) but only 4.5% on ViT-L/14 (from 84.5% to 89.0%).

The higher capacity of ViT-L/14 also mitigates the knowledge conflicts, which reduces the relative effectiveness of TATR.

Q: In line 414, "Table 5 and Table 2" should be "Table 1 and Table 2".

A: Thanks for your suggestion. We have uploaded a revised manuscript where this typo has been corrected.

Q: Does TATR directly remove components with a high magnitude of the task vector? Is TATR aligned with the motivation that selecting components of task vectors orthogonal to other task gradients?

A: TATR does not simply “select components with smaller values (or norms) for merging.” Instead, we select components based on smaller absolute products between the task vector and the gradient (Definition 2). Selecting vector components with small absolute products with the gradient vector is a sufficient condition for the update vector to be orthogonal to the gradient vector. Therefore, selecting components with smaller absolute products aligns the motivation of selecting components of task vectors orthogonal to other task gradients

To demonstrate that our smaller absolute products do not rely on selecting components with smaller values, we conducted a straightforward experiment: we removed high-magnitude components from task vectors based on TA in a removal ratio τ, denoted as RM(τ). The table below shows that the new method is substantially worse than TATR.

Q: The claim about the zero-shot version of TATR is not accurate since TATR does not yield performance gains in all cases.

A: Thanks for your suggestion. Considering that the zero-shot version of TATR shows improvements in 7 out of 9 scenarios (including ViT-B/16), we revise the statement to "Although there may be estimation errors, this approach still offers performance improvements for TA-based methods in most scenarios."

Thank you very much for your reply, which solved most of my questions. The results look very good.

But I still have one more doubt that needs your answer:

In lines 259-265 of the article, you use the element-wise product as the inner product, which is confusing as it seems to conflict with the definition of the inner product, where the output is required to be a scalar. This also makes the definition of 'orthogonal' prone to misunderstanding. Furthermore, I am concerned about whether such a definition is appropriate. So I think maybe there is a more appropriate name to describe it.

Thank you again for your reply. I am very happy to increase the score.

Q: The experiments convincingly demonstrate the effectiveness of the proposed data augmentation method. However, the performance improvements are minimal in most cases.

A: Thank you for your feedback and support for our work.

Our method demonstrates good performance gains in training-free cases. For instance, on the ViT-B/32 model, our method provides an average improvement of 3.7% for TA and 0.9% for Tiesmerging. On ViT-L/14, the improvements are 0.8% and 0.5%, respectively. Similarly, on ViT-B/16, our method achieves a notable 3.2% improvement for TA.

AdaMerging and Surgery show high performance gains over TA. However, our method is better in the following ways.

1. TATR can be applied on top of AdaMerging and Surgery and improve these methods further.
2. AdaMerging and Surgery require expensive training with substantial data, whereas TATR is a lightweight, plug-and-play module that is completely training-free.

Q: Is it always possible to find orthogonal components for task vectors?

A: Yes with high probability. The orthogonal components defined in our work refer to the vector which contains elements of task vector with near-zero product to other task gradients, denoted as $\Delta_i^{\perp} = \Delta_i \odot 1_{\{\nabla_\theta L_j(\theta_{\text{pre}})\odot \Delta_i\approx 0\}}$. These components are not difficult to find, since the gradient $\nabla_\theta L_j(\theta_{\text{pre}})$ tends to be sparse, especially in neural networks such as transformers that leverage the attention mechanism or ReLU activation [1].

Additionally, the table below reports the distribution of the absolute products between task vector elements to gradients across all task pairs, where the value of $n$-th element is calculated as $\sum_i \sum_{j, i\neq j} \big| \nabla_\theta L_j( \theta_{\textnormal{pre}})[n] \cdot \Delta_i [n] \big|$. As observed, the majority of values falls within the range of (0, 0.005]. In other words, most vector components are orthogonal components.

[1] Li, Chunyuan, et al. "Measuring the Intrinsic Dimension of Objective Landscapes." International Conference on Learning Representations. 2018.

Q: Evaluation of heterogeneous task.

A: The experiment is currently in progress, and we are striving to finalize and release the results at the earliest opportunity. I appreciate your understanding.

Q: If a large task vector along the gradient descent direction causes overshooting, how about scaling down the magnitude of the task vector?

A: Thank you for the suggestion. We agree that other important criteria, such as vector magnitudes, could play a critical role in knowledge conflicts. However, finding an appropriate scaling factor for various negative components remains a challenging task. We conducted a straightforward experiment: if a task vector component has the same sign with the corresponding component of the gradient descent vector, we follow your advice to scale it down using a multiplier $\tau$. Otherwise, we mask that component of the task vector.

$m[n] = \tau \ \ \text{if } \sum_{i \neq j} \nabla_\theta L_j( \theta_{\textnormal{pre}})[n] \cdot \Delta_i [n] < 0 \ \ \text{else} \ \ 0$

$\theta_{MTL} = \theta_{pre} + \lambda m \odot \sum_i \Delta_i$

We tried different multipliers and the results are in the following table. The new method is substantially worse than TATR.

Q: The paper computes the trust region by summing over all tasks. Why not compute a trust region for each pair of tasks?

A: Our algorithm aims to create a network that can handle all tasks of a multi-task problem. Hence, we must update $\theta_{pre}$ with all task vectors and use a trust region from all task vectors. We do not build trust regions for each task pair, because we do not create separate networks for each task pair.

To further illustrate the benefits of the TATR trust region, we do the following experiment. We first compute a TATR-style trust region for each possible pair of tasks, which selects some components of the task vectors. After that, we compute the intersection of the selected components.

Formally, the TATR-style trust region for every task pair $(i, j)$ is defined as $TR_{j|i}$, which contains selecting components of the task vectors satisfying $\forall n\in TR_{j|i} ,\ \{ \big| \nabla_\theta L_j( \theta_{pre})[n] \cdot \Delta_i [n] \big| < \epsilon \}$, and then we merge the network weights as $\theta_{pre} + \lambda \sum_k \Delta_k \odot 1_{\{n \in \mathcal{ \hat{TR} }\}}$ where $\hat{TR} =\bigcap_{i, j, i \neq j} TR_{j|i}$.

The following table shows that the new method degrades performance, proving the effectiveness of maintaining a single trust region.

Q: Evaluation of heterogeneous task.

A: Thank you for your patience. We followed your suggestion to perform experiments on PASCAL-Context. Specifically, we utilized the encoder of SAM with a ViT-B architecture as the pre-trained backbone, and fine-tuned it independently on four diverse tasks: semantic segmentation, human part segmentation, saliency estimation, and surface normal estimation.

The following table exhibits the results, where TATR consistently improves the performance of TA-based methods.

Q: Evaluation of heterogeneous task.

A: We also conducted experiments on vision-language tasks. We obtained the task vectors by finetuning the BLIP model [1] independently on three diverse vision-language datasets: TextVQA [2], ScienceQA [3], and SVR [4]. Among these, TextVQA focuses on answering questions by reading the text on the image. ScienceQA is about answering visual questions in the scientific domain, which involves knowledge recall and reasoning. SVR is about classifying textual descriptions regarding spatial relations in the image into true or false statements.

More specifically, we finetuned the VQA version of BLIP for 6000 steps per task. The model architecture contains an image encoder, a text encoder, and a text decoder. All model weights were finetuned.

The table below presents the results, demonstrating that TATR enhances the overall performance of TA-based methods.

[1] Li, Junnan, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." International conference on machine learning. 2022.

[2] Singh, Amanpreet, et al. "Towards VQA models that can read." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[3] Lu, Pan, et al. "Learn to explain: Multimodal reasoning via thought chains for science question answering." Advances in Neural Information Processing Systems. 2022.

[4] Liu, Fangyu, Guy Emerson, and Nigel Collier. "Visual spatial reasoning." Transactions of the Association for Computational Linguistics. 2023.