MTSAM: Multi-Task Fine-Tuning for Segment Anything Model

We sincerely thank the reviewer for providing valuable comments. You can find our response below for your concerns. Please kindly let us know if you have any further concerns.

Q1. Scaling-up experiments are lacking. Is there any way to see a larger number of tasks beyond 3-4? I really would like to stress test ToRA.

Thank you for your valuable suggestion. Due to time and computational resource constraints, we conducted few-shot experiments on the Taskonomy dataset. Specifically, we used 200 images from six tasks (i.e., segment semantic, depth estimation, surface normal, keypoint detection, edge detection, and reshading) of one view in Taskonomy as the training data and used another view for testing. The results are shown in following table. As can be seen, ToRA achieves better performance than LoRA-STL in 5 tasks and has comparable result in 1 task, which demonstrates the effectiveness of the proposed ToRA.

Q2. Are there any out of domain experiments, same task but out of domain datasets?

First, we would like to humbly clarify that our proposed framework, MTSAM, is primarily designed for efficient multi-task fine-tuning of SAM, rather than for addressing out-of-distribution data scenarios.

Nonetheless, to evaluate its performance on unseen data as suggested, we applied the model fine-tuned on the NYUv2 dataset to make depth predictions on CityScapes dataset. Qualitative results are provided in Appendix E, and illustrate that MTSAM is capable of handling unseen data distributions to some extent.

Q3. Are there any experiments on the speed? ToRA as described has a computational advantage but there does not seem to be any experiments to back that up, unless I missed them.

As mentioned in Section 3.3, the proposed method, ToRA, demonstrates better parameter efficiency compared to LoRA, with sublinear growth in parameter w.r.t the number of tasks. Additionally, similar to LoRA, we froze pre-trained parameters and fine-tune low-rank update tensors during training and inference. Consequently, our method has a similar computational complexity to LoRA, resulting in comparable training and inference speeds.

Q4. Table 3 is slightly disappointing with LoRA-STL (r=32) beating MTSAM on some tasks. Not a show-stopper since overall performance seems ok.

Table 3 shows the results on PASCAL-Context dataset. Compared to LoRA-STL (r=32), MTSAM achieves 2.53% improvement on average with a lower number of trainable parameters. Specifically, MTSAM achieves significantly better performance in the semantic segmentation task and comparable results in the other three tasks. The slight performance decline in the three tasks compared to LoRA-STL is possibly due to conflicts in shared parameters among different tasks or biases from dominant tasks during model training. A feasible solution is to apply optimization algorithms from multi-task learning, e.g., loss-balance methods and gradient-balance methods. We consider this as an interesting direction for our future work.

Q5. The qualitative examples need to be better. Resolution is very low and hard to tell. I know cityscapes are like that, but perhaps testing on some higher res samples would be more convincing.

Thanks for your valuable suggestion. Based on your suggestion, we tested MTSAM trained on the CityScapes dataset on high-resolution images, with results included in Appendix D of the updated manuscript. As can be seen, MTSAM outperforms other baselines across various tasks. In areas highlighted by white boxes, MTSAM generates more accurate results, demonstrating the effectiveness of MTSAM.

I see.

I will finalize my rating at 6. In my opinion, citiscapes and kitti are not zero shot at all, as they are very similar, so there is still question in my about the effectiveness, with such a 5-6 times drop in performance. NYU and cityscapes are completely different so it is hard to use them to gauge what is considered large drop.

I still think your paper has merits, and also I have read the other reviewers' discussions with you. I think rating of 6 is a fair score.

Thank you for taking time to respond to all my questions.

We sincerely thank you for providing valuable comments. You can find our response below for your concerns. Please kindly let us know if you have any further concerns.

Q1. The first and second points in the summary of contribution points appear to be similar, the author is requested to provide good reasons for the significant difference, otherwise it is recommended to merge.

Thanks for your valuable suggestions. In our revised manuscript, we merge the first and second points in the summary of contributions points. Thus, the contributions of this paper are three-fold:

• We propose MTSAM, a novel multi-task learning framework that extends the capabilities of SAM to perform multi-task learning. Specifically, we modify the original architecture of SAM by removing the prompt encoder and adding task embeddings. This modification enhances the flexibility of the original SAM.
• We introduce ToRA, a novel multi-task PEFT method, that applies low-rank decomposition to the update parameter tensor, effectively learning both task-shared and task-specific information simultaneously, with theoretical analysis for its strong expressive power.
• We conduct comprehensive experiments on benchmark datasets, demonstrating the exceptional performance of the proposed MTSAM framework.

Q2. It is hard for me to consider MTSAM as a substantial model innovation improvement, (1) I don't get why the prompt encoder affects the downstream adaptation of the model, (2) the author's improvement of the decoder is more like the integration of multiple decoders for different tasks.

(1) In the SAM architecture, the prompt encoder extracts features from prompts (i.e., points, boxes, and masks), which are then used in cross-attention to compute the final segmentation masks. However, in a multi-task learning scenario for dense prediction tasks, we do not have prompts, so this part of the architecture cannot work during model inference. Therefore, we have removed it.

(2) As mentioned in Figure 1 of the manuscript, applying SAM to multi-task learning faces the challenge of varying numbers of output channels for different tasks. For example, the number of output channels of the semantic segmentation task equals the number of classes, while that for the depth estimation task is always equal to 1. To address this challenge, we introduced task embedding, allowing the model to generate appropriate outputs according to task requirements while maintaining a unified architecture.

Moreover, to verify the effectiveness of our proposed task embedding, we attempted to adjust the output dimensions of the MLP in the decoder to achieve the desired output. As shown in the above table, this approach is less effective compared to using task embedding.

Q3. I am interested in ToRA, but the author's description of the details is unclear. (1) How to constrain U1, U2, and U3 to learn task-relevant and task-irrelevant information, respectively? (explain or experiment) (2) Why is the decomposition into U1+U2+U3+G just enough? What happens if there are more or less U's? What if there were G?

(1) For the three-mode update parameter tensor $\Delta \mathbf{W}$, the first mode represents the output feature dimension, the second mode denotes the input feature dimension, and the third mode is for the task dimension. Hence, according to Tucker decomposition [r1], $U_1$ and $U_2$ reflect the main subspace variation of task-shared information corresponding to the first two modes in $\Delta \mathbf{W}$, while $U_3$ reflects the task-specific subspace structure corresponding to the last mode of $\Delta \mathbf{W}$. We make it clearer in the revision.

(2) The number of $U$'s is determined by the mode of tensor [r1]. As the update parameter tensor $\Delta \mathbf{W} \in \mathbb{R}^{d \times k \times T}$ is a 3-mode tensor, Tucker decomposes it into three factor matrices $U_1$, $U_2$, $U_3$, and a 3-mode core tensor $\mathcal{G}$.

[r1] Some mathematical notes on three-mode factor analysis. Psychometrika.

Q4. The proposed ToRA lacks comparison with other LoRA-based improvement methods.

Thank you for your valuable suggestions. We have included a comparison with MultiLoRA [r2] in our experiments. Moreover, we added the comparisons with the latest published LoRA-based methods, Terra [r3] and HydraLoRA [r4]. The results are shown in the following table, and the complete results can be found in Table 7 of the updated manuscript. As can be seen, ToRA achieves better performance compared to those LoRA-based methods.

[r2] Multilora: Democratizing lora for better multi-task learning. arXiv preprint arXiv:2311.11501 2023.

[r3] Time-Varying LoRA: Towards Effective Cross-Domain Fine-Tuning of Diffusion Models. NeurIPS 2024.

[r4] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning. NeurIPS 2024.

We sincerely thank you for providing valuable comments. You can find our response below for your concerns. Please kindly let us know if you have any further concerns.

Q1. Could you provide a more detailed analysis of the advantages over advanced methods like SwinSTL?

Compared to Swin-based architectures (i.e., VTAGML and SwinMTL), we propose a novel multi-task parameter-efficient fine-tuning framework, MTSAM, which effectively leverages the rich semantic knowledge in the foundation model SAM. Specifically, MTSAM offers the following advantages:

• MTSAM achieves better performance compared to those baselines. Specifically, MTSAM shows 4.38% and 12.45% improvement over SwinMTL on the NYUv2 and CityScapes datasets, respectively.
• MTSAM demonstrates better parameter efficiency, offering advantages in storage and enhancing its practical application value.

Q2. Can you clarify why different evaluation metrics are used for segmentation tasks or others across various datasets? What criteria were used to select these metrics?

We follow the classic setups of [r1,r2] to evaluate the performance of different tasks on different datasets. The evaluation metric of each task has been introduced in Appendix B.1.

• On NYUv2 and CityScapes datasets:
  • For the semantic segmentation task, we use mIoU and Pixel Accuracy (Pix Acc).
  • For the depth prediction task, we use Absolute Error (Abs Err) and Relative Error (Rel Err).
  • For the surface normal estimation task, we evaluate the mean and median of angular errors measured in degrees and the percentage of pixels with angular errors within 11.25°, 22.5°, and 30°.
• On PASCAL-Context dataset:
  • We use mIoU to evaluate the semantic segmentation, human parts segmentation, and saliency estimation tasks.
  • For the surface normal task, we use the mean of angular errors measured in degrees.

[r1] Polyhistor: Parameter-efficient multi-task adaptation for dense vision tasks. NeurIPS 2022.

[r2] End-to-end multi-task learning with attention. CVPR 2019.

Q3. Would it be possible to include more comprehensive examples of image multitasking in the results section?

Thanks for your valuable suggestions. In our revised manuscript, we have added more qualitative results in Appendix D to provide comprehensive examples of image multitasking.

We sincerely thank you for providing valuable comments. You can find our response below for your concerns. Please kindly let us know if you have any further concerns.

Q1. The current work lacks an evaluation of MTSAM’s zero-shot generalization ability, particularly on unseen data distributions.

First, we would like to humbly clarify that our proposed framework, MTSAM, is primarily designed for efficient multi-task fine-tuning of SAM, rather than for addressing out-of-distribution data scenarios.

Nonetheless, to evaluate its performance on unseen data as suggested, we applied the model fine-tuned on the NYUv2 dataset to make depth predictions on the CityScapes dataset. Qualitative results are provided in Figure 12 of Appendix E, and illustrate that MTSAM is capable of handling unseen data distributions to some extent.

Q2. The experiments do not include a direct comparison with full fine-tuning methods, leaving it unclear whether MTSAM’s parameter-efficient fine-tuning can achieve competitive performance without compromising accuracy.

Thanks for your valuable suggestions. We conducted a comparison with the full fine-tuning method on the NYUv2 dataset. As shown in the following table, the results demonstrate that MTSAM achieves significant improvements over the full fine-tuning method. This improvement is attributed to MTSAM’s high parameter and sample efficiency in multi-task parameter-efficient tuning, whereas full fine-tuning requires a larger number of samples to converge and achieve the same level of performance.

As suggested, the result of full fine-tuning method has been added in Appendix C.1 of the revised manuscript.

Q3. There is a lack of comparative experiments to demonstrate the effectiveness of task embedding. Further experiments are needed to confirm the specific advantages of task embedding in multi-task setups.

To demonstrate the effectiveness of the proposed task embedding, we compared it with the method of modifying the MLP output dimensions for different tasks on the NYUv2 dataset. As shown in the table below, task embedding performs better. This improvement is due to the interaction between task embeddings and image features through the cross-attention mechanism, which enables the decoder to better learn the task-specific knowledge and achieve superior results. Those results and analyses have been added in Appendix C.2 of the revised manuscript.