Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models

W3: Comparison with Marigold [1] and StableNormal [2]

First, we would like to clarify that while our work, Marigold [1], and StableNormal [2] are all relevant to the surface normal estimation task, they pursue different exploration directions and aim to achieve distinct objectives. Marigold [1] and StableNormal [2] repurpose diffusion models for dense prediction by finetuning the denoising network, which results in the loss of the model's RGB generation capability. In contrast, our framework integrates both generation and discrimination, preserving the ability for RGB generation while further extending it to multi-modal generation.

Second, we provide a comparison with methods that repurpose diffusion models for discriminative tasks, empirically demonstrating that our framework is more flexible for such tasks. Below, we mainly focus on a comprehensive comparison with Marigold [1], as StableNormal has not released the official training/finetuning code.

• We compared three variants of Marigold on NYUv2 surface normal estimation in total. Variant 1, denoted as Marigold (pretrain), is their released checkpoint that is trained on a mixed large dataset excluding NYUv2. For variant 2, we adopt the same setting as ours, using the 795 training samples to train Marigold from the Stable Diffusion checkpoint until convergence, denoted as Marigold (SD). For variant 3, we further finetune their released checkpoint with 795 training samples from NYUv2, denoted as Marigold (finetune). The results of the three variants, together with ours are reported below, as well as in Table F in the revised manuscript.

• All three variants lagged behind our model, indicating the effectiveness of our model design with the self-improving learning mechanism. Moreover, we observe that Marigold gets inferior performance when adapted to a specific domain with a limited amount of training data (795 samples). While finetuning from a well-trained model can help mitigate this issue, it still does not work as well as our proposed method. The reason is that tuning these diffusion-based perception models like Marigold, which require finetuning the denoising U-Net, is computationally expensive. In comparison, our approach only requires training a lightweight task head, which makes our framework more flexible and easier to train or fine-tune for new domains.

• In addition, the comparison of the computational cost of our method and Marigold with a batch of images of shape (2, 512, 512, 3) is reported below:

Our framework is a more efficient and effective solution compared with Marigold.

---

We sincerely thank you once again for your valuable comments, which have greatly helped improve our work. We hope that the explanation above can well address your concerns.

We sincerely appreciate the detailed feedback you provide for our work! Below, we provide the following clarifications to address your concerns:

---

W1-1: Writing improvement on Figure 2 and Section 4.3

Thanks for your comment! The original Figure 2 contained some inconsistent notation in the color illustration as mentioned by Reviewer Got7's [W3], which may be the reason for the cause of confusion. We have addressed this issue by providing a corrected version of Figure 2 in the revised manuscript. We have also made revisions on Lines 264-265, 269, 311-314, 318-320 to clarify and better motivate the self-improving component. Basically, the underlying motivation of our proposed self-improving learning mechanism is that while synthetic multi-modal data typically demonstrates high visual fidelity, its direct utility for discriminative learning remains uncertain, so we need to have a self-improving stage to efficiently utilize the synthetic multi-modal data for discriminative perception tasks. We are glad to further revise the corresponding figures and texts if you have remaining concerns about them.

---

W1-2: Self-improving learning performance in Table 7

Thank you for your comment! First, we would like to kindly clarify that Table 7 is not intended to ablate the self-improving technique, but rather to assess the quality of the generated multi-modal data. More specifically,

• In Table 7, we quantitatively evaluate the quality of the generated multi-modal data, by training an external discriminative model (Bae et al.) using a mixture of the original samples and generated samples. This evaluation serves as an “offline” assessment of the multi-modal data quality, and importantly, no self-improving learning is involved in the training of the external model (Bae et al.).

• The results in Table 7 show that both "GT + Syn (Before Self-improving)" and "GT + Syn (After Self-improving)" outperform the basic version of "GT Only", demonstrating the high-quality and usefulness of the multi-modal data generated from our framework.

• However, it is important to note that the comparison between "before/after self-improving" in Table 7 should not be interpreted as a rigorous ablation of the self-improving technique. This is because the setting in Table 7 does not leverage the full potential of the self-improving process --- in Table 7's setting, we only use the synthetic multi-modal data to train an external model (Bae et al.) without incorporating the closed-loop interaction between generation and discrimination.

Second, we provide an ablation to directly ablate the effect of self-improving learning by directly comparing the variants of our model with and without the self-improving mechanism using Bae et al. on the NYUv2 surface normal benchmark, as below:

We can observe that the self-improving mechanism indeed further boosts the performance of the model, and the improvement is evident.

---

W2: Ablation study on the critical aspects of the framework

Thanks for the comment! Following the suggestions, we conducted the ablation study on whether or not to include self-improving learning, as shown in response to the previous concern W1.

Moreover, we would like to kindly ask for clarification on the suggested ablation of "a variant without using diffusion-denoising process". Based on our understanding, it is hard to conduct such an ablation. The reason is that the unique denoising-diffusion process is integral to our unified diffusion model, which enables both multi-modal data generation and dense pixel prediction. This dual capability further serves as the foundation for the self-improving learning mechanism --- a wise way to better utilize the generated multi-modal data to boost discriminative learning. Therefore, it is not feasible to remove the diffusion-denoising process from our current framework.

We are glad to provide more ablations for a better understanding of our framework if the reviewer could further clarify or suggest other experiments.

We sincerely appreciate the constructive feedback you have given for our work! Below, we provide the following clarifications to address your concerns:

---

W1: Extension of our approach for other types of tasks

Thank you for the comment! We would like to first kindly point out that our method is a general framework designed to accommodate different types of tasks by incorporating task-specific prediction heads. In this work, we focus on pixel-level dense prediction tasks, which are a series of primary and important tasks in computer vision. Nevertheless, following the suggestion, we conduct an additional evaluation on the multi-task benchmark of Tiny-Taskonomy, with categorization as one of the target tasks. The results are shown below:

The results validate that our framework can also provide improvement on other types of perception tasks beyond dense pixel prediction.

---

W2 & W3: Discussion on improving RGB image generation

Thanks for the comment! First, we would like to kindly point out that, in our default setting, the diffusion U-Net remains frozen, ensuring that the image generation capability remains consistent between our model and the backbone model (Stable Diffusion). More importantly, our framework extends this capability to enable multi-modal generation, producing diverse and consistent image-annotation pairs --- a property not explored or achieved by previous approaches.

Second, we have evaluated the generation performance both quantitatively and qualitatively. (1) In Section 5.4, we have assessed the quality and usefulness of the generated multi-modal data by measuring downstream task performances, as there are currently no direct metrics for evaluating such data. These results demonstrate the high quality of our generated paired data. (2) We have provided extensive visualizations of multi-modal generation results, as shown in Figures E-H in the appendix.

Third, while our primary focus in this paper is on multi-modal generation, we appreciate the reviewer's insight that our framework holds the potential to leverage multi-modal information to enhance RGB generation. Conceptually, generating multi-modal visual attributes such as surface normals, depth, and semantic labels explicitly can provide more precise geometric and semantic information, which could significantly benefit RGB generation. We leave this as a very interesting direction for future exploration.

We have revised the text in Lines 66-67, 130 to provide more clarification in the paper.

---

W4: Ablation study on the self-improving strategy

Thanks for the comment! We include the ablation study on using/not using the self-improving mechanism with Bae et al. on the NYUv2 surface normal benchmark as below:

We observe that the self-improving mechanism indeed further boosts the performance of the model by fostering an effective interaction between the discriminative and generative components in our framework.

---

W5: Performance improvement on semantic segmentation and depth estimation

For semantic segmentation, it is because the data regime between the settings of semantic segmentation and normal prediction is different. In the surface normal prediction task, there are only 795 available training samples as shown in Table 1. In comparison, for the semantic segmentation benchmark on ADE20K, there are about 20K training samples in the standard setting. The improvement of our method is more significant in a lower data regime.

For depth estimation in Table 3, it is because this task is optimized together with other tasks in a multi-task setting. Therefore, the model also needs to balance and compromise among all the tasks, which makes the performance gain in Table 3 not as significant as the improvement in Table 1. However, when we place our scope of performance gain comparison to the same multi-task setting in Table 3, we can see that the performance gains of depth estimation and surface normal estimation within the single multi-task setting (Table 3) are on a comparable level.

---

We sincerely thank you once again for your valuable comments, which have greatly helped improve our work. We hope that the explanation above can well address your concerns.

We are sincerely grateful for your positive feedback! We provide the following clarifications in response to your concerns:

---

W1: Reduce the content of preliminary and add visualizations

We thank the reviewer for the suggestion! We have adjusted the content of the preliminary and included more visualizations in the main paper in the revised manuscript accordingly, as shown in Lines 143-147 and Figure 5.

---

W2: Performance improvement in Table 2

We want to first kindly clarify that both DA-Fusion and our method are built upon the same baseline model (VPD). Therefore, the comparison between these two methods should be based on the performance gain over the baseline model instead of the absolute numbers. With this perspective, the performance gain from our method is 0.8 for the segmentation task, which is notably more significant than the performance gain of 0.3 brought by another data augmentation method DA-Fusion. As a reference, the performance gain of VPD over previous MAE/ConvNeXt-based methods is only 0.1 (from 53.6 to 53.7). Thus, we believe our improvement in the segmentation task is significant.

---

Q1: Decreased performance on saliency detection

Thanks for the great question! The reason is that, in multi-task learning where the model is optimized with multiple objectives, there are inherent trade-offs among all the tasks, especially when the optimization direction of some tasks conflicts with others, and the primary goal is to achieve the best overall performance across all tasks. In Table 4, we achieved improved performances for all the other tasks, and the overall performance of our method did get better, demonstrating the effectiveness of our method. However, if we want to improve the performance of a specific task under a multi-task setting, a simple yet effective way is to increase the weight of the objective corresponding to that task.

---

We sincerely thank you once again for your valuable comments, which have greatly helped improve our work. We hope that the explanation above can well address your concerns!

Dear reviewer,

We sincerely thank you for the reply! We are glad that your concerns have been well addressed!

Thank you once again for your constructive feedback!

Best,

Authors

We are sincerely thankful for your insightful feedback on our work! We provide the following clarifications in response to your concerns.

---

W1: The overall writing lacks clarity. The paper's main focus is on improving discriminative learning using synthetic data. Claiming it as "A single, unified diffusion-based model for both generative and discriminative learning" may not be proper, as the method does not improve generative learning.

Thanks for the comment! We want to point out that improving discriminative learning is not our sole focus. We present a single integrated model for diffusion models to perform both discriminative perception and multi-modal generation, and we have also assessed the quality and usefulness of the generated multi-modal data in Section 5.4.

Our work is significantly different from existing works like Marigold [a] or GeoWizard [b] which repurpose diffusion models for perception tasks. They only consider the optimization on the discriminative tasks by tuning the whole denoising U-Net. These approaches completely change the functionality of diffusion models from text-to-image generation to image-to-X (X is the predicted modality). Therefore, they lose the original capability of diffusion models to generate RGB images. In contrast, we still preserve the original generation capability of diffusion models and extend it to enable consistent multi-modal generation, including but not limited to RGB image synthesis.

Additionally, we have revised Lines 66-67, 130 to provide further clarification in the updated manuscript.

[a] Ke et al. Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation. CVPR 2024.

[b] Fu et al. GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image. ECCV 2024.

---

W2: Tunable parameters

Thanks for the comment! In our default settings, the pretrained latent diffusion remains frozen, with only the lightweight task head being tunable, therefore enabling more efficient and low-computation-budget tuning. Two exceptions are the VPD-based variant in Table 2, where finetuning U-Net is required by VPD, and the ablation of the finetuning strategy in Table B. We have revised related texts in Lines 322-323, 909-912 to clarify this.

---

W3: Clarification in Figure 2

We are sincerely grateful that the reviewer spotted this issue! There was indeed inconsistent notation in color illustration in the original version. We have updated Figure 2 in the manuscript.

---

W4: Extension of our method to fully fine-tuned models

We thank the reviewer for this insightful comment! As mentioned in our response to W1 above, one of the primary differences between our method and the recommended fully-tuned approaches is that those methods repurpose diffusion models for dense prediction by finetuning the denoising network, which makes them lose the capability of RGB generation. In comparison, our framework maintains the capability of RGB generation and further extends it to multi-modal generation. This primary difference makes it difficult to directly instantiate our framework to the recommended models since their objective breaks the RGB generation. However, we do appreciate the suggestion and also believe that extending the general idea of bridging generation and dense perception to boost their performance is a promising future research direction.

---

W5: Boosting discriminative capacity on data with large domain shifts

This is a good point! We would like to first kindly point out that the diversity of the synthetic data generated with the SDEdit-style solution can be controlled by the hyperparameter $T$ which is the timestep of the diffusion process. Evidence from Figure 4 shows that a higher $T$ will lead to a more diverse generation of the multi-modal data while a smaller $T$ boosts the faithfulness of the generated data.

Moreover, we conducted a cross-domain evaluation experiment in Table 6 showing that the out-of-distribution performance of our model is even better than the in-distribution performance of the baseline model that we build upon, which indicates that our framework provides improvement in situations with domain shifts.

Finally, we want to point out that, naturally, it is challenging to fully solve the severe domain shift issue using a single source of training data. A more common solution is to train a model with multiple sources of data. In this case, the data generation strategy within our framework can further increase the diversity of the training data, thereby further boosting the robustness of our model on out-of-distribution test samples.

---

We sincerely thank you once again for your valuable comments, which have greatly helped improve our work. We hope that the explanation above can well address your concerns!

W6: Performance gain over DA-Fusion

We want to argue that the performance gain over DA-Fusion is significant across all the benchmarks shown in the paper.

First, we want to clarify that the way to compare the performance gain between our method and DA-Fusion is not to compare the actual numbers of the two, but the performance gain of these two models over the baseline models (e.g. in Table 2, it is "the discrepancy between the last line and the third to last line" v.s. "the discrepancy between the second to last line and the third to last line")

With this understanding, for surface normal prediction, the performance gain acquired by our method compared to DA-Fusion when iDisc as the base model is shown as follows:

It is clear to see that our performance gain compared with DA-fusion is significant.

For semantic segmentation, our performance gain of 0.8 as shown in Table 2 is also obviously more significant than the gain of 0.3 brought by DA-Fusion. For reference, the performance gain of VPD compared with previous MAE/ConvNext-based methods is only 0.1 (from 53.6 to 53.7).

For the multi-task benchmarks, the discrepancy between our method and DA-Fusion is also larger than the gap between DA-Fusion and the baselines in most metrics as shown in Tables 3 and 4.

---

Q2: Potential extension of this work to recent approaches

Thanks again for your insightful suggestions! As noted in our response to W4 above, while our work and the recent approaches suggested by the reviewer all leverage diffusion models to address visual perception tasks, they pursue different exploration directions and aim to achieve distinct objectives. These recent approaches like Marigold repurpose diffusion models for discriminative tasks by fine-tuning the entire diffusion U-Net, thereby changing the original function of diffusion models and losing the capacity of RGB generation during the fine-tuning process. In contrast, our method appends a lightweight task head while preserving the RGB generation capability – an essential property for enabling self-improvement within our framework. As a result, it is not straightforward to directly apply our approach to their architectures. Nevertheless, we sincerely appreciate the suggestion and believe that extending the general idea of bridging generation and dense perception to improve performance is a promising direction for future research.

---

We sincerely thank you once again for your valuable comments, which have greatly helped improve our work. We hope that the explanation above can well address your concerns!

W4: Missing recent related work

Thanks for the suggestion! We have added the discussion of these works in the related work section in Lines 120-126. At a high level, [1, 2, 3] take a different strategy of using diffusion models for discriminative learning – they repurpose diffusion models from text-to-image generation to dense prediction by finetuning the denoising U-Net. With such a design, they achieve promising results with the cost of totally losing the capability of generation. In comparison, we are exploiting the capability of diffusion models to discriminative perception, and at the same time, preserving the original RGB generation capability, and further expanding to multi-modal generation. Thanks to this unique design, we can utilize the multi-modal data to benefit discriminative learning, and further allow us to introduce a self-improving stage to enable interactions between generative and discriminative learning.

We also provide a comparison with methods that repurpose diffusion models for discriminative tasks, empirically demonstrating that our framework is more flexible for such tasks. Below, we mainly focus on a comprehensive comparison with a representative method Marigold [1].

• We compared three variants of Marigold on NYUv2 surface normal estimation in total. Variant 1, denoted as Marigold (pretrain), is their released checkpoint that is trained on a mixed large dataset excluding NYUv2. For variant 2, we adopt the same setting as ours, using the 795 training samples to train Marigold from the Stable Diffusion checkpoint until convergence, denoted as Marigold (SD). For variant 3, we further finetune their released checkpoint with 795 training samples from NYUv2, denoted as Marigold (finetune). The results of the three variants, together with ours are reported below, as well as in Table F in the revised manuscript.

• All three variants lagged behind our model, indicating the effectiveness of our model design with the self-improving learning mechanism. Moreover, we observe that Marigold gets inferior performance when adapted to a specific domain with a limited amount of training data (795 samples). While finetuning from a well-trained model can help mitigate this issue, it still does not work as well as our proposed method. The reason is that tuning these diffusion-based perception models like Marigold, which require finetuning the denoising U-Net, is computationally expensive. In comparison, our approach only requires training a lightweight task head, which makes our framework more flexible and easier to train or fine-tune for new domains.

• In addition, the comparison of the computational cost of our method and Marigold with a batch of images of shape (2, 512, 512, 3) is reported below:

Our framework is a more efficient and effective solution compared with Marigold.

---

W5: Unrelated works

Thanks for the suggestions! We have removed this part and, as the reviewer suggests, replaced it with the discussion of more closely related works in the revised manuscript. Please refer to Section 2 in the revised manuscript for more details. In the original submission, we included a more general background for readers who may not be familiar with visual perception, because this paper focuses on dense visual perception.

We sincerely appreciate the detailed feedback you provided for our work! We provide the following clarifications in response to your concerns:

---

W1: Claim of the "unified framework"

We thank the reviewer for the suggestion! We would like to first kindly clarify that the generation aspect of our framework focuses on "multi-modal generation" rather than just RGB image generation, for which we generate both RGB images and their corresponding pixel-wise annotations (as highlighted in Lines 16-17, 50-52). Under this design, our model is a "unified" model with the task-specific head serving as a shared component used for both discriminative and generative tasks. Nevertheless, to avoid confusion, we have introduced additional clarifications (Lines 66-67, 130, etc.) and revised the manuscript (please refer to Lines 15, 48, 52, 74, etc.) changing the term "unified" to "integrated". We are happy to discuss and make further revisions.

---

W2 & Q1: Details of $\theta_E$ and $\theta_C$

Thanks for the comment! As indicated by Equations 3 and 4, $\theta_E$ and $\theta_C$ represent the entire parameters of our model. Notice that, not all parameters in $\theta_E$ / $\theta_C$ are required to be tuned for a more efficient and low-computation-budget finetuning.

In our default experimental setting, we only finetuned the task head, except for the VPD-based variant (where we followed the full-weight finetuning strategy from VPD for a fair comparison) in Table 2 and the ablation of the finetuning strategy in Table B. However, whether or not finetuning the U-Net parameters does not affect the correctness of our self-improving mechanism in Equation 5, this equation always holds no matter whether we finetune the entire parameters or not. For our revised manuscript, we have revised the corresponding sections in Lines 322-323, and 909-912 for clarity.

---

W3: Novelty in Core Components

We would like to first kindly point out that the design of the core component is not the sole criterion for considering novelty. For example, VPD itself uses the standard design of plain FPN as its mask decoder; and both DIFT [a] and "A tale of two features" [b] even did not use any components except the pretrained diffusion models. We would like to further emphasize our key contributions as follows.

• First, existing works, such as VPD and Marigold, utilize diffusion models solely for discriminative tasks with the loss of generation capability. As the reviewer acknowledges, "the integration of generative and discriminative learning is an interesting direction" – our proposed framework, which integrates both generation and discriminative perception, fills a gap in the current research landscape and offers a significant contribution for future work to continue exploring in this direction. The novelty and promising exploration of the idea and our framework that integrates dense perception and multi-modal generation has also been acknowledged by Reviewers 8JiW, Ao2e, and Got7.
• Second, we want to emphasize that the generation task we focus on is the capability of "multi-modal generation", instead of sole RGB generation. Therefore, the fact that our model, which is trained only with discriminative objectives, also gains the capability of consistent multi-modal generation that can be used to boost the discriminative process is non-trivial and novel.
• Finally, although EMA updates are broadly studied for settings like semi-supervised learning, which handles the case with unlabeled real data, we further extend its application to the context of synthetic data utilization. Specifically, we use EMA guidance from a model optimized for discriminative perception to update a model for multi-modal generation, while the multi-modal generation model, in turn, provides the synthetic data back to the model for discriminative perception. The underlying motivation is that while synthetic multi-modal data typically demonstrates high visual fidelity, its direct utility for discriminative learning remains uncertain. Therefore, we believe that this unique interaction between the generative and discriminative components, facilitated by the EMA mechanism, has seldom been studied and explored before.

[a] Tang et al. Emergent Correspondence from Image Diffusion. NeurIPS 2023.

[b] Zhang et al. A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence. NeurIPS 2023.

Dear reviewer,

We have posted our responses to your concerns and updated our manuscripts according to your suggestions, please check whether your concerns have been resolved and we are happy to have further discussions! Thanks again for your detailed and valuable review of our paper!

Best,

Authors

Dear Reviewer,

Thanks again for your detailed review! In our previous response, we have clarified and elaborated on our concept of an integrated framework, where a single model handles both "multi-modal data generation" (generative learning) and "dense visual perception" (discriminative learning). Here, we would like to present additional experiments demonstrating that unfreezing the denoising U-Net in our framework and jointly learning both RGB generation and the discriminative objective enables mutual benefits, enhancing both RGB generation and discriminative perception through the joint modeling.

More specifically, we unfreeze the denoising U-Net in the diffusion model, and add a loss term on noise prediction for the RGB generation (the same generation loss term in Stable Diffusion) alongside the original discriminative loss. We find that the performance of both generative and discriminative tasks can be improved as follows:

• The FID of the synthetic RGB images for our self-improving stage improves from 48.53 (original Stable Diffusion checkpoint) to 40.40 (after our joint finetuning).
• The discriminative task on surface normal with Bae et al., the performance improvement is as follows: |Settings|11.25$^\circ(\uparrow)$|22.5$^\circ(\uparrow)$|30$^\circ(\uparrow)$|Mean ($\downarrow$)|Median ($\downarrow$)|RMSE ($\downarrow$)| |:---|:---:|:----:|:----:|:----:|:---:|:---: |Discriminative objective only (reported in the paper) | 67.4 | 83.4 |88.2 | 13.2 | 6.5 | 22.0 | |Generative and discriminative objectives (the new setting) | 68.6 | 84.4 | 89.1 | 12.8 | 6.4 | 21.6 |

We can observe that, with the joint modeling of generative and discriminative training objectives, the quality of the generated images gets improved, further boosting the performance of the discriminative perception task. As mentioned in the previous response, we froze the diffusion model in the original manuscript to prioritize building a more efficient and lightweight framework; but the experiments above show that with sufficient computational resources, fine-tuning the whole denoising U-Net can further improve the performance. We will include these new results into the next revision of our manuscript.

Thank you once again for your thoughtful comments! We are looking forward to hearing from you, and are always glad to have further discussions!

Best,

Authors

Dear Reviewer LDAd,

As the discussion period will come to an end in less than 24 hours, we would like to send you a reminder about our responses as above to solve your concerns. Please check whether your concerns have been addressed. We are sincerely looking forward to hearing from you, and are always happy to have more discussions with you!

Thank you once again for your insightful reviews!

Best,

Authors