Multi-Task Dense Predictions via Unleashing the Power of Diffusion

Q4: To ensure comprehensive evaluation, it is imperative that the study provides a comparative analysis of the proposed technique against the most recent methods (“DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction” AAAI 2023; “Multi-Task Learning with Multi-Query Transformer for Dense Prediction” TCSVT 2024; “Multi-Task Dense Prediction via Mixture of Low-Rank Experts” CVPR 2024). In addition, the discussion of the decoder-based MTL approach in the related work section lacks an introduction to recent works, such as DeMT AAAI’23, 3DAwareMTL ICLR’24, MQTransformer TCSVT’24 and MLoRE CVPR’24.

A4: Thank you for your advice. Since some of the methods listed do not have performance based on ViT-large, we reproduce them based on ViT-large, and the performances are listed below. In addition, our method serves as a novel pipeline that leverages diffusion models to model the task relation explicitly. It can cooperate with other network architectures to achieve better performance. As a result, we combine the module proposed by MLoRE [1] and our method and it can surpass the performance of the previous SoTA on all tasks, especially in human parsing, normal prediction, and boundary detection. These results have been updated to Tab. 1 in the revision.

Q5: Figure 4 does not seem to demonstrate the visualization advantage of the proposed TaskDiffusion model in the appendix, compared to TaskPrompter.

A5: The differences between our method and TaskPrompter are circled in the figure. For example, the TaskPrompter mistook the background behind the lying person for the chair, while our method can predict the background accurately. It can be seen that the results predicted by our method are closer to the ground truth than TaskPrompter on all the tasks.

Q6: The authors state: “To our knowledge, we are among the first to leverage diffusion models in …”. The DiffusionMTL (CVPR’24) model already took advantage of the diffusion model.

A6: Thank you for your comment. Our method is among the first to leverage diffusion models in fully-labeled multi-task dense prediction, while DiffusionMTL focuses on partially-labeled multi-task dense prediction. We have modified our statement into “We are among the first to leverage diffusion models in fully-labeled multi-task dense prediction ...” in the revision.

Q7: What is Cross-Task Label Encoder based on? Some tasks are labeled with images or data in matlab format. How do authors encode these different data formats?

A7: Our codes are based on Python. Following the code provided by previous works [1，2], we convert all the labels into 4D pytorch tensor for the following process.

[1] Hanrong Ye and Dan Xu. Inverted pyramid multi-task transformer for dense scene understanding. In ECCV, pages 514–530. Springer, 2022a.

[2] Hanrong Ye and Dan Xu. Taskprompter: Spatial-channel multi-task prompting for dense scene understanding. In ICLR, 2022b.

[3] Yuqi Yang, Peng-Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, and Bo Li. Multi-task dense prediction via mixture of low-rank experts

To reviewer QTvQ: Thank you for your valuable comment. We will address your concern in the following point-by-point.

Q1: The value of Delta_m (MTL gain metric) requires the value of the single task (ST) model. It is suggested that the authors release the value of ST in Table 3. Following the calculation criteria of MTL on Delta_m metric, showing Baseline ST and MT values will have a beneficial impact on the development of MTL. The reviewer saw Table 7, but it appeared in the appendix rather than in the main paper.

A1: Thank you for your suggestion. We agree with the importance of the performance of the ST model. We have added them to Tab.3 in the revisions.

Q2: From the results in Table 3, the core diffusion decoder designed in this paper does not seem to bring much growth to the model. Table 3 also reveals that a good encoder leads to greater performance gains. The relative performance improvement brought by the proposed Cross-Task Diffusion Decoder is not even comparable to ATRC and InvPT.

A2: It can be seen from Tab. 3 that the proposed cross-task diffusion decoder brings an improvement of 1.69% in $\Delta_m$ compared to the baseline, and 0.91% compared to the task-specific diffusion decoder. Following previous works [1], a performance gain in $\Delta_m$ for about 1.5%(e.g. In Tab. 1 the performance gain after +SPrompt is 1.52%) can be considered effective. In addition, our method focuses on proposing a better decoder that can improve the performance with tolerable efficiency loss, though, changing the encoder to a heavier backbone will bring more computation cost and parameter number. It is hard for a decoder to bring as much performance gain as a heavier encoder does. At last, the InvPT [1] is the first multi-task dense prediction method that leverages the ViT-large as the backbone. The performance of ATRC with ViT-large listed in Tab. 1 is reproduced in Taskprompter [2]. As a result, it is natural for InvPT, a method designed for the ViT-based backbone, to have a significant performance gain compared to ATRC, whichis designed for the CNN-based backbone.

Q3: Task-fused features U are generated by A_F in line 302. Why the features after A_F are fused features? The same question arises on line 308. Why can this matrix multiplication represent feature fusion and what is its mathematical interpretation? Both task-fused features and task-fusing features mean the same map U.Moreover, the same problem has $p$ in 302 line and $p$ in Eq.2.These inconsistency are confusing to the reader.

A3: In line 302, the $A_F$ represents the task-relation maps. More specifically, $A^{s, p}$ represents the weight for task $s$ to gain from the task-specific feature from task $p$. In other words, the task-fused feature $U^s$ for task $s$ is the weighted sum of task-specific features from different tasks, like the attention mechanism. In addition, the inconsistency of task-fused/task-fusing feature and annotation $p$ has been corrected in the revision.

Thanks the authors for the response. Most of the my concerns have been addressed.

Please ensure that the additional content from your rebuttal are incorporated into the main text of your paper. This will enhance the clarity and completeness of your work, ensuring it effectively communicates your contributions to the community. Adhering to this commitment will significantly strengthen your paper.

I decide to keep my original rating.

To reviewer T3fK:

Thank you for your valuable comment. We will address your concerns in the following point-by-point.

Q1: The proposed modules have limited novelty. The cross-task diffusion decoder builds upon existing works, such as DDP, while incorporating interaction techniques from multi-task learning. The cross-task label encoder functions as an inverse process of task-specific heads in standard multi-task models. It would be better if the paper could demonstrate more insights in the utility of diffusion for multi-task dense prediction.

A1: Thank you for your comment. The novelty of our method can be summarized in two points. Firstly, the diffusion models are proven to have outstanding ability in capturing the underlying distribution of each single task in DDP. However, the ability to capture the cross-task relations is overlooked by the existing method. In addition, the cross-task relation is the key point to improve the overall performance of different dense prediction tasks in multi-task learning. To achieve our goal, we designed the cross-task diffusion decoder that leverages the diffusion process to model the task relation explicitly in a coarse-to-fine manner. In other words, our method is inspired by the impressive performance of the diffusion process and leverages the diffusion process in the key problem of multi-task learning.

Secondly, the cross-task label encoder is proposed to unify the encoding of discrete labels and continuous labels. The inverse process of task-specific heads and our cross-task label encoder is different in the target feature space. The inverse process of task-specific heads maps the different labels to their corresponding feature spaces, which are also task-specific. On the contrary, our cross-task label encoder maps the continuous or discrete spatial map to a feature space shared by all the tasks. This encoding mechanism can get rid of cumbersome task-specific encoding and encode the discriminative information from Wdifferent tasks to gain better performance.

Q2: The authors are recommended to compare their method with more state-of-the-art methods, especially those published in 2024, such as MLoRE and TSP-Transformer.

A2: Thank you for your suggestion. Our method proposes a novel diffusion-based multi-task learning pipeline that can be used to improve the task-relation generation in other multi-task learning networks. As a result, we combine the MLoRE module proposed in previous work [1] with our proposed pipeline and achieve a new SoTA in the PASCAL-Context dataset. The performance of it and previous SoTA are listed below. It can be seen from the result that our method can surpass the performance of the previous SoTA on all tasks, especially in human parsing, normal prediction, and boundary detection. These results have been updated to Tab.1 in the revision.

Q3: There is room for improvement in presentation, including language and figures. For example, the text in Figure 1 right is too small to be easily readable.

A3: Thank you for your advice. We will polish the language and figures carefully in the camera-ready version. In addition, the text in Fig. 1 has been bigger in the revision.

Q4: In Section 3.2, task-specific auxiliary heads are used to generate intermediate predictions that are supervised by ground truth labels. It is interesting to know the contribution of these auxiliary heads to the final results, as they have different training procedure with the diffusion process.

A4: The task-specific auxiliary heads can generate intermediate predictions from task-specific branches. Without the auxiliary heads, there will be no guarantee that these branches are capturing task-specific features. As a result, it will be hard for the cross-task diffusion decoder to generate feasible task-relation which results in performance degradation. The performance of our method without the auxiliary heads is listed below. It can be seen that the auxiliary head brings a performance gain of 0.29% in $\Delta_m$, which supports our argument.

Q5: In Algorithm 2, what is the relationship between masks_pred and m_preds?

A5: Thanks for your comment. The “masks_pred” in Algorithm 2 should be “m_preds”. We have corrected this typo in the revision.

Q6: In Table 1, the FLOPs of the proposed method with a ViT-large backbone is reported as 610G. However, in Table 6, the ablation study indicates that the model with 3 steps utilizes 667G FLOPs. Why does the model with the ViT-base backbone has a higher computational cost than the ViT-large backbone?

A6: In the ViT-base backbone, we follow the previous method [2,3] and use more channels in the decoder. In multi-task dense prediction, since the decoder is branched for different tasks, the computation costs for the decoder can outweigh the encoder in some cases. As a result, the multi-task dense prediction method may have more computation costs for the ViT-base backbone than the ViT-L backbone because of increased computation costs in the decoder. For instance, TaskExpert [3] have more computation cost for the ViT-base backbone (782GFLoPs) than the ViT-large backbone (622GFLoPS).

[1] Yuqi Yang, Peng-Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, and Bo Li. Multi-task dense prediction via mixture of low-rank experts

[2] Hanrong Ye and Dan Xu. Taskprompter: Spatial-channel multi-task prompting for dense scene understanding. In ICLR, 2022b.

[3] Hanrong Ye and Dan Xu. Taskexpert: Dynamically assembling multi-task representations with memorial mixture-of-experts. In ICCV, 2023.

Q5: Some magnificent results are missing. In Table.1 and 2, the single task should be considered a baseline to figure out the impact of the negative transfer and the improvement. And should be provided in Tables 1 and 2.

A5: For PASCAL-Context, we list below the STL performance and $\Delta_m$ in Tab. 7.

Also, the STL performance for NYUD-v2 is listed as follows.

These results have been added to Tab.1 and Tab.2 in the revision.

Q6: The presentation of Table.3 and 4 should be improved. How the $\Delta_m$ is calculated? If it is based on the single task baseline, it should be provided.

A6: As proposed by the previous work [3], $\Delta_m$ is the average per-task performance drop of a task $m$ with respect to the single-task learning baseline $b$. The calculation of $\Delta_m$ can be formulated as follows: $\Delta_m=\frac{1}{T}\sum_{i=1}{T}(-1)^{l_i}(M_{m, i}-M_{b, i})/M_{b_i},$ where $l_i=1$ if a lower value means better for measure $M_i$ of task $i$. The performance of the single-task learning baseline has been added to Tab.3 in the revision.

[1]Hanrong Ye and Dan Xu. Taskprompter: Spatial-channel multi-task prompting for dense scene understanding. In ICLR, 2022b.

[2]Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Mti-net: Multi-scale task interaction networks for multi-task learning. In ECCV, pages 527–543. Springer, 2020.

[3]Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos. Attentive single-tasking of multiple tasks. In CVPR, pages 1851–1860, 2019

To reviewer Cv9P:

Thank you for your valuable comments. We will address them in detail in the following.

Q1: The related works of MTL dense prediction are quite old. Some recent research[1-4] related to MTL dense prediction should be considered as references.

A1: Thank you for your advice. We have added these works to the Related Work in the revision.

Q2: The claim “This strategy utilizes the diffusion model to model the task relations explicitly in a coarse-to-fine process during denoising, …” in lines 79-80 should be clarified more deeply. To be specific, the illustration of $A_i$ in Fig.3 is hard to comprehend. What is the relationship between a pair of task?

A2: Thanks for the suggestion. To utilize the diffusion model to model the task relation, we introduce the pixel-wise attention map $A_i^{s, p}$ generated in the diffusion process. As introduced in Sec.3.2, $A_i^{s, p}$ is a $H\times W$ map that represents the pixel-wise attention map from task $s$ to another task $p$. $A_i^{s, p}$ can fuse the task-specific feature from different tasks to the target task dynamically, and can act as an explicitly modeled task relations. During the diffusion process, $A_i^{s, p}$ will focus on a more specific area as shown in Fig. 3, which can support our claim in lines 79-80. In Fig.3, we visualize the $A_i^{s, p}$ mentioned in Sec.3.2 for specific $s$, $p$ and $i$. We annotate task pairs on the bottom of each figure. For example, in the visualized $A_i^{s, p}$ in the left-top figure in Fig. 3, $s$ represents the task index for the saliency detection and $p$ represents the index for boundary detection.

Q3: Some experimental details should be corrected and provided. For instance, the original PASCAL-Context does not provide the label for the surface normal task. And what is the resolution of input?

A3: Thank you for your suggestions. We follow the previous works [1, 2] and use the PASCAL-Context dataset with labels on five tasks. The original PASCAL-Context has annotations for semantic segmentation, human part segmentation, and edge detection. The surface normals and saliency labels are obtained from [3]. These labels are distilled from pre-trained state-of-the-art models. The input resolution is set to 512, which also follows the previous works. We have added this information in the Sec.4.1 of the revision.

Q4: Some comparisons are not fair. For example, the results of HRNet18 or HRNet48 should not be compared to ViT-large, since the backbone affects the performance remarkably.

A4: In Tab. 1, we compare the performance of our method and previous SoTA. Some of the previous methods, including PAD-Net, MTI-Net, ATRC, MQTransformer, and DeMT, did not report their performances based on the ViT-large backbone in their original papers. For a fair comparison, we borrow performances of PAD-Net, MTI-Net, and ATRC reimplemented with ViT-large from TaskPrompter. In addition, we reimplement MQTransformer and DeMT based on the ViT-large backbone by ourselves and report their performances in the revision. It can be seen that our method can still surpass the performance of these previous SoTA when they are reimplemented with ViT-Large. In addition, for a clearer and fairer comparison with the previous SoTA, we modify Tab.1 and Tab.2 in the revision and make sure all the compared methods are based on ViT-large.

Thank you! Most of my concerns have been addressed. I still have some questions about the $A_i$ in Fig.3. For example, why does the saliency task focus on the human area of the boundary's feature? And why human parsing concentrate on the corners of the semseg's feature? For the other two attention maps, I still have the same questions. More explanations and clarifications would be better.

Thank you for your response. I do recommend the authors add the corrected statements and analysis to the next revision. I will raise my score.

To reviewer YCpJ:

Thank you for your valuable comments. We will address your concerns in the following point-by-point.

Q1: The figures lack sufficient detail. The task interaction module and level interaction modules should be included in Figure 2 for better clarity.

A1: Thank you for your advice. As suggested by the reviewer, we add a more detailed model architecture of task-interaction modules and level-interaction modules in Fig. 4 in the revision, which is an extension of Fig. 2. We hope this can make the understanding of the proposed method clearer.

Q2: In Algorithm 2, the term “masks_pred” is not mentioned in the context. Should it be “m_preds”?

A2: Thanks for your comment. The “masks_pred” in Algorithm 2 should be “m_preds”. We have corrected this in the revision.

Q3: The novelty is somewhat limited, as previous research has already demonstrated that diffusion-based multi-task learning is feasible.

A3: Although diffusion-based multi-task learning has been proven to be feasible by previous methods, some challenges remain to be solved. These challenges include the lack of digging task relations, the inefficiency resulting from the iterative denoising process, and the cumbersome encoding design for labels of different tasks. To solve these challenges, we present a novel multi-task diffusion pipeline.

Our novelty can be summarized in two points. Firstly, inspired by the outstanding ability of the diffusion models to capture the underlying distribution of prediction for each task, we want to leverage the diffusion models in the key problem of multi-task learning, which is the generation of cross-task relations. With the help of our proposed cross-task diffusion decoder, our model can model the cross-task relations explicitly in a coarse-to-fine manner, and results in better overall performance than the previous diffusion-based multi-task dense prediction method, DiffusionMTL, as shown in Tab.1 in the paper. In addition, by denoising all the tasks jointly, our proposed cross-task diffusion decoder can speed up the overall diffusion process in inference and needs less computation cost than DiffusionMTL as shown in Tab.1.

Secondly, we propose the cross-task label encoder to unify the encoding of different tasks. Specifically, it encodes the continuous or discrete spatial map to a shared feature space. The proposed label encoder can get rid of the cumbersome task-specific encoding and encode the discriminative information from different tasks to improve the overall performance.

Q4: Can you visualize the cross-task map at different steps to better understand it?

A4: Thanks for the suggestion. We visualize the cross-task map at different steps in Fig.6 and analyze it in Sec.A.3 in the revision.