Data Attribution for Multitask Learning

More Complex Models:

Thank you for suggesting the exploration of more advanced models, such as CLIP and LoRA. While this paper does not include experiments on such models, we would like to emphasize the following points:

1. Complex models also exhibit parameter-sharing structures, and our derivations remain relevant and insightful in these contexts. STL-based influence functions have already been generalized to complex models [3]. Therefore, we believe extending our method to these architectures is both feasible and an exciting direction for future work.
2. In many real-world applications that have strict requirements for serving time latency, simpler MTL models are still widely used due to their computational efficiency. Examples include recommender systems [4] and autonomous driving [5].
3. Certain scenarios, particularly those requiring interpretability, favor simpler models over complex architectures. This is especially important in domains like healthcare and finance [6,7], where understanding model predictions is crucial. As a result, simpler MTL models continue to hold significant academic interest (e.g., the linear models used in our experiments come from Duan and Wang (2023) published in Annals of Statistics [8]).

Single-Task Method Baselines:

1. The primary contribution of our work is adapting STL-based IF to the MTL setting, which necessitates a new framework. This adaptation involves addressing the unique challenges posed by task-specific and shared parameter structures in MTL. That's the focus of our paper.
2. For the experimental baselines, we have additionally included gradient-based task-relatedness measures from the MTL literature to provide further points of comparison and enhance the rigor of our evaluation.

Relationship to Gradient-Based Optimization Techniques:

Our method and gradient-based optimization techniques are orthogonal approaches. While gradient-based techniques adjust conflicting gradients to preserve beneficial components during training, our data selection method focuses on identifying and mitigating the influence of negative samples via data selection. Notably, our data selection method can be combined with gradient-based techniques to achieve improved performance, as indicated by our experimental results.

We sincerely appreciate your insightful feedback, which has allowed us to improve both the clarity and robustness of our work. Thank you again for your thoughtful comments!

[1] Fifty, Chris, et al. "Efficiently identifying task groupings for multi-task learning." Advances in Neural Information Processing Systems 34 (2021): 27503-27516.

[2] Azorin, Raphaël, et al. "" It's a Match!"--A Benchmark of Task Affinity Scores for Joint Learning." arXiv preprint arXiv:2301.02873 (2023).

[3] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023. URL https://arxiv.org/abs/2308.03296.

[4] Zhao, Zhe, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi. "Recommending what video to watch next: a multitask ranking system." In Proceedings of the 13th ACM conference on recommender systems, pp. 43-51. 2019.

[5] Liang, Xiwen, Yangxin Wu, Jianhua Han, Hang Xu, Chunjing Xu, and Xiaodan Liang. "Effective adaptation in multi-task co-training for unified autonomous driving." Advances in Neural Information Processing Systems 35 (2022): 19645-19658.

[6] Parker Knight and Rui Duan. Multi-task learning with summary statistics. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 54020–54031. Curran Associates, Inc., 2023.

[7] Adel Javanmard, Jingwei Ji, and Renyuan Xu. Multi-task dynamic pricing in credit market with contextual information, 2024. URL https://arxiv.org/abs/2410.14839.

[8] Yaqi Duan and Kaizheng Wang. Adaptive and robust multi-task learning. The Annals of Statistics, 51(5):2015 – 2039, 2023. doi: 10.1214/23-AOS2319. URL https://doi.org/10.1214/23-AOS2319.

(continued) Results for synthetic dataset:

Results for HAR dataset:

Results for CelebA dataset:

We thank Reviewer MsCP for taking the time to review our paper and for their constructive feedback. Please find below our point-to-point response:

Contribution:

We acknowledge that applying influence functions (IF) to multitask learning (MTL) may appear conceptually similar with IF for single-task learning (STL). However, our contribution lies in two significant aspects:

1. Adapting single-task IF to the MTL setting requires a new framework due to the unique parameter structure in MTL. Specifically, MTL involves both shared and task-specific parameters, and test data predictions in MTL are tied to only a submodel within the overall framework. Addressing these complexities necessitated rethinking the application of influence functions in this context.
2. Our method introduces a natural way to estimate task-relatedness and address negative transfer, two critical challenges in MTL. This contribution is particularly relevant to the MTL literature, as understanding and mitigating negative transfer has significant implications for improving MTL performance.

Furthermore, motivated by the reviewer’s question, we further investigated the potential adaptation of efficient approximate IF methods developed for STL to MTL, and showed that it is non-trivial. Specifically, we examined two common Hessian inverse approximation techniques used in STL IF settings: EK-FAC and LiSSA. Our derivations provide insights into their applicability in MTL settings. The updated paper includes a detailed discussion (see Section 4.2), which we summarize below:

• EK-FAC: This method approximates the Hessian using a blockwise diagonal matrix, which ignores off-diagonal interactions between shared and task-specific parameters. While computationally efficient, this approximation can lead to the loss of significant contributions when computing influence scores, particularly in soft parameter-sharing models where inter-task interactions play a critical role.
• LiSSA: This method approximates the inverse-Hessian-vector-product using an iterative algorithm that supports mini-batch gradients. In MTL settings, however, the empirical Hessian for a data point has a unique structure due to parameter sharing, with non-zero entries restricted to specific sub-blocks. This structure often results in the mini-batch empirical Hessian being ill-posed, characterized by a high condition number, which poses challenges for achieving convergence and numerical stability. In this revision, we ran additional experiments to assess the applicability of LiSSA and added the results to the Appendix. Our empirical results suggest that, as the number of tasks increases, LiSSA requires progressively larger batch sizes to stabilize the stochastic approximation. This scaling significantly raises the computational costs for large-scale MTL problems. Adapting popular methods like LiSSA to address challenges arising from the unique Hessian structure in MTL settings requires nontrivial efforts and would be a valuable direction for future work.

Experiment Baselines:

We have incorporated two gradient-based baselines, TAG [1] and Cosine Similarity [2], into our task-relatedness experiments for both linear regression and neural networks. These baselines are methods for measuring task relatedness in the MTL literature. Each baseline method provides a score of task relatedness for each pair of tasks. We evaluate these methods in terms of the correlation between their scores and the oracle task relatedness obtained from brute-force LOTO retraining as detailed in our paper.

The results, as shown below, clearly demonstrate that our proposed MTIF method outperforms these baselines. Specifically, MTIF achieves consistently higher correlation coefficients with oracle influence estimates, underscoring its superior effectiveness in quantifying task-relatedness. We have included these new results in Appendix Section C.2, and we list below for your convenience.

(Continued) Results for CelebA dataset:

Additional Data Selection Results:

We have also provided the data selection results for the synthetic dataset and the HAR dataset [1] below as well as in Appendix C.3. Overall, We can see significant improvement after applying data selection in each dataset.

The data selection results for synthetic dataset (the error between the learned parameter and the ground-truth for each task, the lower the better):

The data selection results* for HAR dataset (average test classification error over all tasks, the lower the better; Vanilla, Clustered, and Lowrank refer to 3 MTL models used in [2]):

*We note that the HAR dataset is a rather simple dataset, and the test classification errors of the original MTL models trained on the full dataset without data selection are already around 0.01. Therefore, it is difficult to have any further improvement on this dataset. In this experiment, we added noise to a random 5% of the training data points, which makes the original MTL models trained on this noisy dataset have test errors around 0.02. As can be seen from the table above, our data selection method brings the performance back to a 0.01 level of test error.

[1] Reyes-Ortiz, Jorge, et al. "Human Activity Recognition Using Smartphones." UCI Machine Learning Repository, 2013, https://doi.org/10.24432/C54S4K.

[2] Duan, Yaqi, and Kaizheng Wang. "Adaptive and robust multi-task learning." The Annals of Statistics 51.5 (2023): 2015-2039.

We thank Reviewer yCgf for taking the time to review our paper and for their constructive feedback. Please find below our point-to-point response:

Experiment Baselines:

We have incorporated two gradient-based baselines, TAG [1] and Cosine Similarity [2], into our task-relatedness experiments for both linear regression and neural networks. These baselines are methods for measuring task relatedness in the MTL literature. Each baseline method provides a score of task relatedness for each pair of tasks. We evaluate these methods in terms of the correlation between their scores and the oracle task relatedness obtained from brute-force LOTO retraining as detailed in our paper.

The results, as shown below, clearly demonstrate that our proposed MTIF method outperforms these baselines. Specifically, MTIF achieves consistently higher correlation coefficients with oracle influence estimates, underscoring its superior effectiveness in quantifying task-relatedness. We have included these new results in Appendix Section C.2, and we list below for your convenience.

Results for synthetic dataset:

Results for HAR dataset:

We thank Reviewer edvo for taking the time to review our paper and for their constructive feedback. Please find below our point-to-point response:

Equation (1):

Equation (1) is a known result from the prior work [1], which is why its derivation was omitted in the initial version of the paper. For a detailed derivation of Equation (1), we refer the reviewer to the Appendix A of Koh and Liang (2017) [1]. We have also included a footnote in our paper to refer the readers to [1].

Equation (6):

The derivation of Equation (6) relies on learned parameters and is unaffected by early training stages, as highlighted in [1]. While the summation of derivatives over all data points equals zero at the global optimum, individual derivatives remain non-zero. As a result, the influence scores calculated from these derivatives remain meaningful and do not converge to zero, ensuring their interpretability and utility throughout.

Sample Selection:

For the HAR dataset, we partition the entire dataset randomly into training, validation, and test sets using an 8:1:1 ratio. For the synthetic dataset, after data generation, we similarly divide the dataset randomly into training, validation, and test sets with an 1:1:1 ratio. For the CelebA dataset, the data is pre-partitioned into training, validation, and test sets. From each partition, we sample a subset of size 1000 for each task to construct our corresponding training, validation, and test sets. We also randomly sample 9 attributes from all 40 attributes to model as 9 binary classification tasks.

More Complex Dataset (such as CIFAR-100):

We note that our experiments have included a fairly complex dataset, the CelebA dataset. The CelebA dataset was introduced at ICCV 2015 by [3], whereas the CIFAR-100 dataset suggested by the reviewer was introduced in 2009 by [4]. CelebA comprises over 200,000 celebrity images, each annotated with 40 binary attributes, covering a wide range of facial features and expressions. Successful predictions on CelebA data require capturing the nuanced facial features in the image, which is often considered more complex than the object classification task in CIFAR-100.

Furthermore, CelebA has been widely used as a standard benchmark in the MTL literature [5], as it is natural to convert the annotated attributes into multiple tasks. It is less natural to convert CIFAR-100, a multi-class classification dataset, into an MTL benchmark.

Generalization Beyond the Current Scope:

Existing literature suggests that single-task learning (STL) influence functions generalize effectively to more complex datasets and models [2]. By extension, it is theoretically plausible for multitask learning (MTL) influence functions to generalize similarly. However, extending our method to more complex datasets and architectures lies beyond the scope of the current paper. We aim to explore this direction in future work.

We sincerely appreciate the thoughtful feedback, which has helped us improve the clarity and robustness of our work. Thank you again for your valuable comments!

[1] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885–1894. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/koh17a.html.

[2] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023. URL https://arxiv.org/abs/2308.03296.

[3] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015

[4] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

[5] Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Effi- ciently identifying task groupings for multi-task learning. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, volume 34, pp. 27503–27516. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/ file/e77910ebb93b511588557806310f78f1-Paper.pdf.

Comparison with Baselines:

We have incorporated two gradient-based baselines, TAG [1] and Cosine Similarity [2], into our task-relatedness experiments for both linear regression and neural networks. These baselines are methods for measuring task relatedness in the MTL literature. Each baseline method provides a score of task relatedness for each pair of tasks. We evaluate these methods in terms of the correlation between their scores and the oracle task relatedness obtained from brute-force LOTO retraining as detailed in our paper.

The results, as shown below, clearly demonstrate that our proposed MTIF method outperforms these baselines. Specifically, MTIF achieves consistently higher correlation coefficients with oracle influence estimates, underscoring its superior effectiveness in quantifying task-relatedness. We have included these new results in Appendix Section C.2, and we list below for your convenience.

Results for synthetic dataset:

Results for HAR dataset:

We thank Reviewer 4o4x for taking the time to review our paper and for their constructive feedback. Please find below our point-to-point response:

Computational Complexity Analysis:

Thank you for raising this important point. We have added a remark in the method section regarding computational complexity. Specifically, for exact Hessian matrix inverse computation, our method reduces the computational complexity from $\Omega\left(\left(\sum_{k=1}^K d_k + p\right)^w\right)$ to $\Omega\left(\sum_{k=1}^K d_k^w + p^w\right)$, where $w \approx 2.37$, $d_k$ are the dimension of task specific parameters for task $k$, and $p$ is the dimension of shared parameters. This reduction is achieved by decoupling shared and task-specific parameters in the optimization process. For a detailed discussion, please refer to the updated method section in the paper.

Empirical Approximation to the Hessian Inverse:

Motivated by your suggestion, we have expanded the discussion in the method section to address potential approximations for the Hessian inverse, focusing on two widely used approaches: EK-FAC and LiSSA. Our derivations provide insights into their applicability in MTL settings. The updated paper includes a detailed discussion, which we summarize below:

• EK-FAC: This method approximates the Hessian using a blockwise diagonal matrix, which ignores off-diagonal interactions between shared and task-specific parameters. While computationally efficient, this approximation can lead to the loss of significant contributions when computing influence scores, particularly in soft parameter-sharing models where inter-task interactions play a critical role.
• LiSSA: This method approximates the inverse-Hessian-vector-product using an iterative algorithm that supports mini-batch gradients. In MTL settings, however, the empirical Hessian for a data point has a unique structure due to parameter sharing, with non-zero entries restricted to specific sub-blocks. This structure often results in the mini-batch empirical Hessian being ill-posed, characterized by a high condition number, which poses challenges for achieving convergence and numerical stability. In this revision, we ran additional experiments to assess the applicability of LiSSA and added the results to the Appendix. Our empirical results suggest that, as the number of tasks increases, LiSSA requires progressively larger batch sizes to stabilize the stochastic approximation. This scaling significantly raises the computational costs for large-scale MTL problems. Adapting popular methods like LiSSA to address challenges arising from the unique Hessian structure in MTL settings requires nontrivial efforts and would be a valuable direction for future work.

Experimental Scope:

We report the data selection results with different multitask architectures, namely CGC [5], MMoE[3] and DSelect-k [4]. The results demonstrate that, after applying data selection with MTIF, the test accuracy of most tasks shows significant improvement across all model architectures. These results have also been included in the Appendix Section C.3.3, and we list below for your reference.

As for more complex datasets, we note that our experiments have included a fairly complex dataset, the CelebA dataset. The CelebA dataset was introduced at ICCV 2015 by [6], and comprises over 200,000 celebrity images, each annotated with 40 binary attributes, covering a wide range of facial features and expressions. Successful predictions on CelebA data require capturing the nuanced facial features in the image. Furthermore, CelebA has been widely used as a standard benchmark in the MTL literature [5], as it is natural to convert the annotated attributes into multiple tasks.

Results for CelebA:

We appreciate your insightful feedback, which has significantly strengthened our paper. Thank you again for your thoughtful comments!

[1] Fifty, Chris, et al. "Efficiently identifying task groupings for multi-task learning." Advances in Neural Information Processing Systems 34 (2021): 27503-27516.

[2] Azorin, Raphaël, et al. "" It's a Match!"--A Benchmark of Task Affinity Scores for Joint Learning." arXiv preprint arXiv:2301.02873 (2023).

[3] Ma, Jiaqi, et al. "Modeling task relationships in multi-task learning with multi-gate mixture-of-experts." Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018.

[4] Hazimeh, Hussein, et al. "Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning." Advances in Neural Information Processing Systems 34 (2021): 29335-29347.

[5] Tang, Hongyan, et al. "Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations." Proceedings of the 14th ACM Conference on Recommender Systems. 2020.

[6] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015

[7] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Citations for Methods in Table 3:

Thank you for catching this oversight. We have added the appropriate citations for each method listed in Table 3 in the revised version of the paper.

We sincerely appreciate your constructive comments, which have significantly improved the clarity and scope of our work. Thank you again for your thoughtful feedback!

[1] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885–1894. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/koh17a.html.

[2] Fifty, Chris, et al. "Efficiently identifying task groupings for multi-task learning." Advances in Neural Information Processing Systems 34 (2021): 27503-27516.

[3] Azorin, Raphaël, et al. "" It's a Match!"--A Benchmark of Task Affinity Scores for Joint Learning." arXiv preprint arXiv:2301.02873 (2023).

[4] Ma, Jiaqi, et al. "Modeling task relationships in multi-task learning with multi-gate mixture-of-experts." Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018.

[5] Hazimeh, Hussein, et al. "Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning." Advances in Neural Information Processing Systems 34 (2021): 29335-29347.

[6] Tang, Hongyan, et al. "Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations." Proceedings of the 14th ACM Conference on Recommender Systems. 2020.

(continued) Results for HAR dataset:

Results for CelebA dataset:

Negative Transfer Literature:

Thank you for pointing out these valuable works on addressing negative transfer in multitask learning. While these approaches tackle the issue from different perspectives, they provide important context for understanding our contributions. We have now included citations and discussions of these works in the related work section of our paper to provide a more comprehensive review of the literature.

More baseline comprison:

We would like to first clarify that IF-based methods in STL cannot be directly applied to the MTL setting due to the unique parameter structure in MTL models, as detailed in the first point of our response.

To address the reviewer’s concern about limited baselines presented in this paper, we have included the following two sets of additional experiments.

1) We showed that our data selection can combine with different architectures.

We report the data selection results with different multitask architectures, namely CGC [6], MMoE [4] and DSelect-k [5]. The results demonstrate that, after applying data selection with MTIF, the test accuracy of most tasks shows significant improvement across all model architectures. These results have also been included in the Appendix Section C.3.3, and we list below for your reference.

2) We compared our influence score with gradient-based task-relatedness measurements in MTL literature.

We have incorporated two gradient-based baselines, TAG [2] and Cosine Similarity [3], into our task-relatedness experiments for both linear regression and neural networks. These baselines are methods for measuring task relatedness in the MTL literature. Each baseline method provides a score of task relatedness for each pair of tasks. We evaluate these methods in terms of the correlation between their scores and the oracle task relatedness obtained from brute-force LOTO retraining as detailed in our paper.

The results, as shown below, clearly demonstrate that our proposed MTIF method outperforms these baselines. Specifically, MTIF achieves consistently higher correlation coefficients with oracle influence estimates, underscoring its superior effectiveness in quantifying task-relatedness. We have included these new results in Appendix Section C.2, and we list below for your convenience.

Results for synthetic dataset:

We thank Reviewer QvTU for taking the time to review our paper and for their constructive feedback. Please find below our point-to-point response:

The key differences between single-task IF and the proposed multitask IF:

We appreciate your comment regarding the distinction between single-task IF and our proposed multitask IF. Conceptually, our multitask IF is similar to single-task IF, as it builds on the influence function framework introduced by [1]. However, our contribution lies in two significant aspects:

1. Adapting single-task IF to the multitask learning (MTL) setting requires a new framework due to the unique parameter structure in MTL. Specifically, MTL involves both shared and task-specific parameters, and test data predictions in MTL are tied to only a submodel within the overall framework. Addressing these complexities necessitated rethinking the application of influence functions in this context.
2. Our method introduces a natural way to estimate task-relatedness and address negative transfer, two critical challenges in MTL. This contribution is particularly relevant to the MTL literature, as understanding and mitigating negative transfer has significant implications for improving multitask learning performance.

Computational Complexity Analysis:

Thank you for raising this important point. We have added a remark in the method section regarding computational complexity. Specifically, for exact Hessian matrix inverse computation, our method reduces the computational complexity from $\Omega\left(\left(\sum_{k=1}^K d_k + p\right)^w\right)$ to $\Omega\left(\sum_{k=1}^K d_k^w + p^w\right)$, where $w \approx 2.37$, $d_k$ are the dimension of task specific parameters for task $k$, and $p$ is the dimension of shared parameters. This reduction is achieved by decoupling shared and task-specific parameters in the optimization process. For a detailed discussion, please refer to the updated method section in the paper.

Empirical Approximation to the Hessian Inverse:

Motivated by your suggestion, we have expanded the discussion in the method section to address potential approximations for the Hessian inverse, focusing on two widely used approaches: EK-FAC and LiSSA. Our derivations provide insights into their applicability in MTL settings. The updated paper includes a detailed discussion, which we summarize below:

• EK-FAC: This method approximates the Hessian using a blockwise diagonal matrix, which ignores off-diagonal interactions between shared and task-specific parameters. While computationally efficient, this approximation can lead to the loss of significant contributions when computing influence scores, particularly in soft parameter-sharing models where inter-task interactions play a critical role.
• LiSSA: This method approximates the inverse-Hessian-vector-product using an iterative algorithm that supports mini-batch gradients. In MTL settings, however, the empirical Hessian for a data point has a unique structure due to parameter sharing, with non-zero entries restricted to specific sub-blocks. This structure often results in the mini-batch empirical Hessian being ill-posed, characterized by a high condition number, which poses challenges for achieving convergence and numerical stability. In this revision, we ran additional experiments to assess the applicability of LiSSA and added the results to the Appendix. Our empirical results suggest that, as the number of tasks increases, LiSSA requires progressively larger batch sizes to stabilize the stochastic approximation. This scaling significantly raises the computational costs for large-scale MTL problems. Adapting popular methods like LiSSA to address challenges arising from the unique Hessian structure in MTL settings requires nontrivial efforts and would be a valuable direction for future work.

Analysis of Negative Transfer:

MTIF is designed to estimate the influence of a data point or source task on the performance of a target task. The calculated influence score provides a clear and interpretable measure of transfer effects: positive influence scores indicate potential positive transfer, while negative influence scores highlight potential negative transfer. In our experiments, we leverage this property for data selection by filtering out data points with low influence scores to mitigate negative transfer. Our results show that this data selection strategy, guided by our influence scores, effectively improves the performance of MTL algorithms. These findings underscore the utility of MTIF in addressing negative transfer and enhancing multitask learning outcomes.

We appreciate your insightful feedback, which has significantly strengthened our paper. Thank you again for your thoughtful comments!

[1] Fifty, Chris, et al. "Efficiently identifying task groupings for multi-task learning." Advances in Neural Information Processing Systems 34 (2021): 27503-27516.

[2] Azorin, Raphaël, et al. "" It's a Match!"--A Benchmark of Task Affinity Scores for Joint Learning." arXiv preprint arXiv:2301.02873 (2023).

[3] Ma, Jiaqi, et al. "Modeling task relationships in multi-task learning with multi-gate mixture-of-experts." Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018.

[4] Hazimeh, Hussein, et al. "Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning." Advances in Neural Information Processing Systems 34 (2021): 29335-29347.

[5] Tang, Hongyan, et al. "Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations." Proceedings of the 14th ACM Conference on Recommender Systems. 2020.

[6] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015

The results, as shown below, clearly demonstrate that our proposed MTIF method outperforms these baselines. Specifically, MTIF achieves consistently higher correlation coefficients with oracle influence estimates, underscoring its superior effectiveness in quantifying task-relatedness. We have included these new results in Appendix Section C.2, and we list below for your convenience.

Results for synthetic dataset:

Results for HAR dataset:

Results for CelebA dataset:

We thank Reviewer CA63 for taking the time to review our paper and for their constructive feedback. Please find below our point-to-point response:

Generalizability Across Different Datasets and Model Architectures:

We report the data selection accuracy with different multitask architectures, namely CGC [5], MMoE [3] and DSelect-k [4]. The results demonstrate that, after applying data selection with MTIF, the performance of most tasks shows significant improvement across all model architectures. These results have also been included in the Appendix Section C.3.3, and we list below for your reference.

As for more complex datasets, we note that our experiments have included a fairly complex dataset, the CelebA dataset. The CelebA dataset was introduced at ICCV 2015 by [6], and comprises over 200,000 celebrity images, each annotated with 40 binary attributes, covering a wide range of facial features and expressions. Successful predictions on CelebA data require capturing the nuanced facial features in the image. Furthermore, CelebA has been widely used as a standard benchmark in the MTL literature [5], as it is natural to convert the annotated attributes into multiple tasks.

Comparative Analysis with Other Attribution Methods:

To the best of our knowledge, there are currently no existing data attribution methods specifically tailored for multitask learning. Our work is the first to propose a framework that adapts data attribution methods to the MTL setting. In response to this feedback, we have added a comparison of task-relatedness measurements between our method and widely used gradient-based task-relatedness measures from the MTL literature in the updated version of the paper. Specifically, we have incorporated two gradient-based baselines, TAG [1] and Cosine Similarity [2], into our task-relatedness experiments for both linear regression and neural networks. Each baseline method provides a score of task relatedness for each pair of tasks. We evaluate these methods in terms of the correlation between their scores and the oracle task relatedness obtained from brute-force LOTO retraining as detailed in our paper.