Q1: It would be better to show the possibility of using this framework in other tasks.

Thanks for your suggestion. We have conducted additional experiments on semantic segmentation and diffuse reflectance prediction. For further details, please refer to Common Concern-1. Regarding object detection, it is not a dense prediction task, and therefore, we have not conducted experiments on it.

Q2: More explanations are necessary for the detail preserver.

Thanks for your valuable advice. We have conducted experiments from the view of frequency to explain the effect of the proposed detail preserver. Please refer to Common Concern-2 and the Sec. G of the supplementary material for more details.

Q3: More analysis on number of time-steps.

Thanks for your valuable feedback. As mentioned in Lines 337-344, unlike image generation, which is trained on billions of images, dense prediction tasks face challenges in obtaining high-quality paired training data. This makes multi-step diffusion models for dense prediction harder to optimize. As shown in Fig. 7 of the main paper, when the amount of training data decreases, the performance of multi-step formulations degrades more rapidly than single-step ones, validating our assumption.

The theoretical foundation can be found in Min-SNR[1] and ANT[2]. The multi-step diffusion training can be regarded as a multi-task learning problem, where the training of diffusion process contains T different tasks, each task represents an individual time-step. It is obvious that optimizing a multi-task model is significantly more challenging than a single-task model. Consequently, under limited training data, single-step models perform better than multi-step ones.

Additionally, 1) the diffusion priors already encode semantic information and spatial layouts, eliminating the need to relearn them; 2) dense prediction tasks are simpler than image generation as they do not require generating new semantic informations (textures or colors). These reasons also explain why the multi-steps formulations is unnecessary in dense prediction tasks, single-step diffusion formulation can achieve even superior dense prediction performance with significantly accelerated inference speed.

[1] Efficient Diffusion Training via Min-SNR Weighting Strategy

[2] Addressing Negative Transfer in Diffusion Models

Dear Reviewer vkmx,

Thank you for taking the time to review our paper and provide your valuable feedback. We have submitted detailed responses to your comments and suggestions, addressing the key points raised. As the discussion is scheduled to end on November 26th, we would greatly appreciate it if you could review our responses and provide any further clarifications at your earliest convenience. Your feedback is extremely important to us.

Thank you again for your time and effort.

Best,

Authors of Submission 13

Dear Reviewer vkmx,

Thanks so much for your time and effort in reviewing our submission! We deeply appearciate it.

We highly value any opportunities to address any potential remaining concerns before the discussion closes, which might be helpful for improving the rating of this submission. Please do not hesitate to comment upon any further concerns. Your feedback is extremely valuable!

Thanks,

Authors of submission 13

Q1: Clarify the performance gap of "direct adaptation" in Table 3 compared to other recent conditional diffusion approaches.

The "direct adaption'' in our paper is directly adapted from the stable diffusion originally for image generation, i.e., specifically, it is implemented using the diffusers library (https://github.com/huggingface/diffusers/tree/main/examples/text_to_image). Unlike methods like Marigold, our direct adaption does not introduce specialized techniques for dense prediction, such as annealed multi-resolution noise, or test-time ensembling. These differences are the primary reasons for the performance gap.

There are two main reasons why we did not base our implementation on Marigold: 1) The Marigold open-source training code was not available when we first began our experiments. 2) We start our analysis from the most straightforward way to leverage the standard diffusion formulation into dense prediction, in order to maximize the universal applicability of our conclusions. And we were also uncertain whether these specialized designs would impact our analysis. Therefore, we kept the original designs for image generation unchanged.

In response, we have implemented our designs based on Marigold, as shown in the table below. The results are consistent with our original findings (especially on NYUv2 dataset) and even achieve better performance on NYUv2. In our experiments, Marigold (v-prediction) represents its original setting, and we did not use test-time ensembling for fast evaluation.

Q2: Evaluating on more comprehensive datasets

Thanks for your valuable advice. In the supplementary material, we added the results on DIODE dataset for depth estimation in Tab. E, F (using true-depth space and disparity space, respectively) and the results on DIODE and OASIS datasets for normal estimation in Tab. H.

Q3: Some numbers in Table 2 differ from those I have seen floating around

The results marked with * in Tab. 1 and Tab.2 were evaluated by ourselves using the tools provided by Marigold (https://github.com/prs-eth/Marigold/blob/main/eval.py) and DSINE (https://github.com/baegwangbin/DSINE/blob/main/projects/dsine/test.py). As the results of Marigold, DSINE, DepthAnything series, StableNormal can be reproduced, we can ensure the correctness of the results for the other methods as well. We will releases the code for the evaluation on other methods soon.

Q4: Whether the larger variability in noise prediction is an algorithmic artifact or a reflection of inherent predictive uncertainty in ill-posed tasks.

Thanks for your comment. The variance is indeed a reflection of inherent predictive uncertainty in ill-posed tasks, however, it should not compromise accuracy significantly. As shown in Fig. 6, the larger variance of noise prediction (illustrated by the banded regions around the line, where wider areas indicate greater variance) leads to higher errors, which is undesirable.

Additionally, we evaluated the results using a testing-time ensemble strategy, similar to Marigold and GeoWizard, combining multiple inferences (10 samples in our experiments). The AbsRel scores on the NYUv2 dataset are presented in the table below. The testing-time ensemble strategy can be regarded as a Monte Carlo estimation, which approximates the true distribution through random sampling. As shown in the table below, even with the testing-time ensemble strategy, the performance of noise prediction remains inferior to that of $x_0$ prediction, indicating that the large variance in noise prediction does indeed impact prediction accuracy.

Q5: Relate, compare and discuss the work Diffusion-E2E-FT

Thanks for your valuable advice! We have added the comparisons between our Lotus and Diffusion-E2E-FT [Garcia et al., 2024] on depth estimation in the Tab. I of supplementary material. For normal estimation, since there are important differences in the training settings between Lotus and Diffusion-E2E-FT, we are re-training the Lotus-G model for a more fair comparison. We will add the comparison results once the training is finished.

The main contribution of Diffusion-E2E-FT is an interesting phenomenon that Marigold and similar models used an inconsistent pairing of timestep and noise, leading to poor single-step predictions. Fixing this issue via setting the “timestep spacing” to "trailing" mode in the config of schedulers will avoid harmful leak of “GT” signals during inference [1], enabling more accurate single-step predictions.

While the final pipeline and evaluation performance of Lotus-D and Diffusion-E2E-FT are quite similar, Lotus is developed from the systematic analysis of leveraging the standard stochastic diffusion formulation into dense prediction. It also incorporates innovations like the detail preserver to further improve the accuracy especially in detailed areas. Additionally Lotus (Lotus-G) is a stochastic model and, unlike the deterministic Diffusion-E2E-FT, it enables uncertainty predictions.

[1] Common diffusion noise schedules and sample steps are flawed.

Q6: Some writing problems

Thanks for your valuable advice. We will continue to review and address similar issues to ensure the article is as professional and polished as possible before the final version.

Dear Reviewer d5L4,

Thank you for taking the time to review our paper and provide your valuable feedback. We have submitted detailed responses to your comments and suggestions, addressing the key points raised. As the discussion is scheduled to end on November 26th, we would greatly appreciate it if you could review our responses and provide any further clarifications at your earliest convenience. Your feedback is extremely important to us.

Thank you again for your time and effort.

Best,

Authors of Submission 13

Regarding Q4, doesn’t that contradict your reply to Q1, where x0-prediction worked no better than v-prediction and only marginally better than eps-prediction?

Q5: thank you. These results really belong in the main paper Table. Also, E2E-FT should be at least briefly discussed in the main paper, given that you agree its pipeline and performance are very similar to yours. Just to be sure, I repeat here that the two works were concurrent, E2E-FT does not detract from the novelty of your findings w.r.t. single-step inference.

Q6: Beyond wording, please also pay attention to my comment regarding a balanced and scientific interpretation of the results. Don’t say “we are the best” when the numbers say “there isn’t much difference”. Don’t say all your extensions are important if your Marigold experiments (c.f. Q1 above ) suggest they maybe aren’t. Etc.

Thank you for the response.

Regarding Q1: I am afraid the new results do not really clarify things. If anything, they show that x0-prediction brings no real advantage over v-prediction. Single-step inference seems to indeed work better on NYUv2, but actually a bit worse on KITTI. On the latter, is exclusively the detail-preserver that gets you back to where the original baseline already was, whereas the same detail preserver does almost nothing on NYUv2. Control experiments (v-prediction + single-step, v-prediction without single-step but with detail-preserver) are missing. So no conclusive answer to my doubts whether the proposed extensions are still necessary and effective when starting from a properly tuned “direct adaptation”. This is not a killer argument against the paper, but it does mean the authors would have to (1) seriously tone down their claims and (2) add the results from the discussion to the paper, so that readers are aware of the open questions.

Regarding Q2: thanks for adding these datasets. Is there a chance they can be added to Tables 1 and 2 in the main paper, and included in the calculation of the average ranks? It is fairly confusing for readers - and skews the ranking - if some of the results from the same set of experiments are in the main paper and some in the appendix.

Regarding Q3: ok, fine. I was just a bit surprised to see Marigold normals, AFAIK that paper never mentions normal estimation.

Overall, I slightly lower my rating. The paper is interesting enough to be published, but seems premature. It is unclear and not properly evaluated whether some of the claimed extensions (x0-prediction, detail-preserver) will stand and have any lasting impact. Single-step inference is probably here to stay, if only to be faster and for situations where a deterministic output is preferred. But I am a bit uncomfortable with publishing a paper where the majority of the contributions is not solidly backed up by experiments.

Thanks for your valuable feedback! We highly value every opportunity to address your concerns. For clarity and your convenience, we divided your questions into two parts : 1) the explanation of the effect of the proposed designs, and 2) revisions to the paper, all revised parts of the main paper are highlighted in red.

Part 1: The explanation of the effect of the proposed designs.

Q1. The performance gap of "Direct Adaptation" in Table 3 compared to other recent conditional diffusion approaches.

We sincerely apologize that our response to Q1 still leaves some issues unresolved. Over the past few days, we carefully re-analyzed this question. The main reason that our experiments in Q1 do not show significant advantages of our designs is the "annealed multi-resolution noise (AMRN)" strategy proposed in Marigold, which significantly enhances the performance of multi-step diffusion models for dense prediction.

As stated in section titled “Annealed multi-resolution noise” on Page 5 and section titled “Training noise” on Page 8 in the Marigold paper, AMRN offers two advantages:

1. Accelerating model convergence. While effective, the Marigold paper does not explain this in detail.
2. Improving the performance. As stated on Page 8 of the Marigold paper, "training with multi-resolution noise leads to more consistent predictions given different initial noise at inference time, and annealing further enhances this consistency." This means that the AMRN strategy, proposed by Marigold for dense prediction, is designed to reduce the model’s variance, particularly in the multi-step formulation with v-prediction, thereby enhancing performance. Marigold also provides experiments in Sec. S2 to demonstrate this.

Since our original experiments for Q1 were based on the Marigold codebase, the AMRN diminishes the impact of our proposed designs. Therefore, we argue that “Direct Adaption” would be better to directly adapt diffusion models originally for image generation, rather than incorporating designs specifically for depth prediction (or dense prediction), such as AMRN. Analyzing from the original diffusion formulation allows us to better understand how to more effectively and efficiently leverage diffusion priors for dense prediction from a more fundamental perspective.

Following your suggestion, we revised our experiments by adding additional ablation experiments (Tab. A) and conducting experiments also based on Marigold codebase but w/o AMRN (Tab. B). We believe these new experiments can better validate the effect of our proposed designs.

There are two tables:

1. Tab. A: The difference between this table and the one in our previous response to Q1 is that we now evaluate the models on the full NYUv2 and KITTI datasets. We also apologize for not noticing earlier that the Marigold codebase only validates on subsets (100 images) for faster evaluation. The validation configuration file in Marigold can be found here: https://github.com/prs-eth/Marigold/blob/main/config/dataset/dataset_val.yaml. All the experimental settings of Lotus (main paper, suppl., the below Tab. A and B, etc.) strictly uses the complete set of all evaluation datasets. Moreover, both Tab. A and B are conducted under the same seed.
2. Tab. B: The difference between Tab. A and Tab.B is that the experiments in Tab. A use AMRN while the experiments in Tab. B do not. Most of the settings in Tab. B are consistent with those in Tab. 3 of our paper, but there are some differences in the training setup: 1) All experiments in Tab. B are trained on a mixed dataset, whereas Tab. 3 uses the single hypersim dataset; 2) Tab. B uses a decaying learning rate, while Tab. 3 employs a constant learning rate.

Dear Reviewer d5L4,

1. Thanks so much for your extremely thorough and constructive review of our submission and responses! The additional experiments based on your comments significantly strengthened the solidness of this submission. We deeply appreciate your valuable suggestions!

2. Furthermore, please accept our sincere gratitude for your positive feedback and the significantly improved rating! We are so excited!

3. Following your latest response, we will definitely include the additional experiments and corresponding discussions in both the public archive version and the camera ready version (if accepted :D) of this submission. Once again, thank you for your extremely responsible and thoughtful comments!

Thanks so much and have a great day!

Best, Authors of submission 13

Q1: It may be beneficial to explore using ZtZ_tZt with different ttt values in one-step training and inference.

Thanks for your comment. I am a bit unclear about what you mean by "ZtZ_tZt" and "ttt." Are you referring to $Z_t$ and $t$? In response, we trained our model using different values of $t$ (even though this deviates from the diffusion formulation) on Hypersim dataset and evaluated on NYUv2 dataset. Since our model follows the diffusion formulation, which predicts the annotation starting from noise in one step, the $Z_t$ remains Gaussian noise for different values of $t$ and is concatenated with the input image. The results are shown in the table below. As observed, the model performs best when $t=T$ (t=1000). Changing t leads to a slight degradation in performance.

Q2: The loss function should be adjusted to reflect this "either-or" relationship.

Thanks for your advice. In fact, our model either predicts annotations or reconstructs images by using different task switchers. As shown in Eq. 6 of the main paper, we employ different task switcher $s_x$ and $s_y$ to represent this "either-or" relationship, where $s_x$ is used for reconstruction and $s_y$ for prediction.

Q3: What is the relationship between the proposed network and existing conditional restoration methods?

To the best of our knowledge, restoration methods aim to recover high-quality or nearly realistic images from low-quality or degraded inputs. There are two main differences between Lotus and restoration models:

1. Different model outputs. Our model’s primaty outputs are generated predicted annotations, whereas restoration models aim to generate reconstructed clear images.
2. Different model inputs. Our model takes clear images with noise as inputs. The noise input is introduced specifically for the diffusion process. In contrast, restoration models take degraded images as inputs.

Q4: Could this design be extended to other dense prediction tasks, such as segmentation?

Yes, we can. We have conducted additional experiments on semantic segmentation and diffuse reflectance prediction. Please refer to Common Concern-1 and Sec. F of the supplementary materials for further details.

Dear Reviewer myC3,

Thank you for taking the time to review our paper and provide your valuable feedback. We have submitted detailed responses to your comments and suggestions, addressing the key points raised. As the discussion is scheduled to end on November 26th, we would greatly appreciate it if you could review our responses and provide any further clarifications at your earliest convenience. Your feedback is extremely important to us.

Thank you again for your time and effort.

Best,

Authors of Submission 13

Dear Reviewer myC3,

Thanks so much for your prompt and kind response! We are so excited to see that!

We highly value any opportunities to address any potential remaining concerns before the discussion closes, which might be helpful for improving the rating of this submission. Please do not hesitate to comment upon any further concerns. Your feedback is extremely valuable!

We deeply appearciate your time and effort in reviewing our submission and writing responses.

Thanks,

Authors of submission 13

Q1: About "removing noise'' of Lotus on line 430

The ``removing noise'' version of Lotus, Lotus-D, is a new model that directly inputs the 4-channel image latent into the UNet model without concatenating it with noise. The purpose of Lotus-D is just to align with the deterministic nature of traditional models and the deterministic nature of benchmarks. Unlike Lotus-G, Lotus-D does not strictly adhere to the diffusion formulation. However, its value lies in the powerful diffusion priors. Only by leveraging the diffusion priors without the full diffusion process, it is still able to achieve superior performance.

Q2: The proposed method appears relatively simplistic and may lack substantial innovation

Thanks for your comment, however, we respectfully disagree with it. We would like to emphasis that our contribution is the proposed fine-tuning protocol for dense prediction tasks, which not only achieves new SoTA performance but is also significantly faster than most existing diffusion-based methods. Our contribution is encouraged by reviewer myc3, d5L4, and vkmx, as our strengths. The related works mentioned in your comment are specifically for a boarder realm of image generation, instead of image dense prediction.

Additionally, as we stated in our paper, ``the standard diffusion formulation originally designed for image generation is unsuitable for dense prediction tasks''. Experimentally, we directly adapted the diffusion formulation for image generation to dense prediction task and observed poor performance (see Lines 263–289). Therefore, it is necessary to investigate a more tailed formulation for dense prediction that can more effectively leverage diffusion priors, which is what our paper achieves.

Q3: Expanding the experiments to include intra-dataset inference or testing in varying contexts.

Thanks for your valuable suggestion. We have conduct additional experiments in two other contexts: semantic segmentation and diffuse reflectance prediction. Both tasks were evaluated in intra-dataset (in-domain) settings, and Lotus significantly outperforms the baseline in these scenarios. Please refer to Common Concern-1 and the Sec. F of supplementary material for further details.

However for depth and normal estimation, we evaluate Lotus on zero-shot benchmarks because 1) zero-shot evaluation is more challenging and more close to the practical unknown scenarios during application, and 2) nearly all recent method, no matter in depth and normal estimation, report their results in zero-shot benchmarks.

Q4: Training the model with datasets of varying sizes, and scaling up the training dataset.

1. Please see the Fig. 7 of the main paper, we have conducted experiments using training datasets of varying sizes, with the corresponding values presented in the table below. The performance of our single-step formulation improves as the training data scales up. This experiment demonstrate that our model has the potential to benefit further from increased data scaling, with similar or better data quality.
2. However, we are currently unable to scale up our training dataset to the same extent as DepthAnything, as our hardware is not capable in training large models with 60 million samples.

Dear Reviewer forA,

Thank you for taking the time to review our paper and provide your valuable feedback. We have submitted detailed responses to your comments and suggestions, addressing the key points raised. As the discussion is scheduled to end on November 26th, we would greatly appreciate it if you could review our responses and provide any further clarifications at your earliest convenience. Your feedback is extremely important to us.

Thank you again for your time and effort.

Best,

Authors of Submission 13

Dear Reviewer forA,

I hope this message finds you well.

Thank you for your time and effort for reviewing our submission. As the discussion deadline (November 26th) approaches, we would greatly appreciate it if you could review our responses and provide any further clarifications at your earliest convenience. Your feedback is extremely valuable!

Thanks again for your time and effort!

Best,

Authors of Submission 13

Dear Reviewer forA,

We sincerely thank you for your time and effort for reviewing our submission, and writing your responses.

However, as the discussion deadline approaches in less than 2 days (only 1 day left for reviewers to post a message), we would greatly appreciate it if you could carefully review our latest responses and provide further clarifications at your earliest convenience. We highly value any opportunities to address any potential remaining issues, which we believe will be helpful for a higher rating. Your timely feedback is extremely important to us!

Once again, thanks so much for your time and effort, we deeply appearciate it!

Best,

Authors of Submission 13

Q1: The relationship of our proposed method between diffusion and UNet-based model.

Q1-1: Why the proposed method is called ``diffusion model''?

Lotus is called "diffusion-based'' method in our title. More precisely, the reasons are: 1) Lotus, especially the Lotus-G, strictly follows the principle of diffusion formulation at time-step $T$ to generate annotation from noise. It contains single-step noising and denoising process. 2) Lotus, either Lotus-G or Lotus-D, relies on the diffusion priors (which is the valuable and powerful pre-trained network parameters) requiring adapting and fine-tuning under the basic diffusion pipeline. Therefore, the claim of "diffusion-based'' model is reasonable.

Q1-2: What is the difference between the proposed method with UNet-based model?

Firstly, diffusion-based model and UNet-based model are interconnected, as UNet serves as the denoiser in diffusion models.

Secondly, Lotus (Lotus-G) still follows the principle of diffusion denoising formulation, the UNet denoises the concatenated latent feature of Gaussian noise and the input image, and producing the latent feature of the dense prediction. Thus, it is not merely a simple UNet-based model.

Lastly, and importantly, we emphasize that our main contribution lies in exploring the most suitable diffusion formulation to effectively leveraging the powerful diffusion priors for dense prediction tasks, rather than focusing on model structure. Through systematic analysis and extensive experiments, we proposed a novel adaptation protocol that significantly outperforms previous diffusion-based methods in both efficiency and accuracy.

Q1-3: Dose the discriminative version just fall back to a simple UNet-based method?

Although our Lotus-D does not strictly follow the diffusion formulation, as the noise input is removed, to meet the discriminative requirements of perceptions tasks.

Its value lies not in its model structure, but in demonstrating that even without the noise input in diffusion formulation, SoTA performance can still be achieved with the support of pre-trained diffusion priors. This result provides an intriguing insight for bridging the fields of generation and perception.

Q2: The effect of Detail Preserver

Thanks for your comment. In our paper, the “original detailed generation ability” does not mean the ability to generate new textures or colors like that in image generation task. It is also not desired to “imagine” new details, which may lead to deviation from input image. Instead, it means to preserve those details that should be clearly expressed in dense predictions, e.g., the geometrical details for depth/normal prediction.

The original diffusion model excels at generating detailed images but may lose its ability to generate intricate details with dense annotation tasks due to catastrophic forgetting. To address this, we propose a regularization strategy called Detail Preserver. This approach uses a switcher: when set to $s_y$, the model focuses on predicting annotation; when set to $s_x$. it reconstructs the input image to retain its detail generation ability. This dual focus improves dense predictions in intricate regions and enhances evaluation performance.

Please refer to Common Concern-2 and Sec. G of the supplementary materials for more details.

Q3: If the authors claim it can be served as a visual foundation model, it needs to show more capability of generalization or potentials of other tasks.

Thanks for your suggestion.

We have trained Lotus on two additional, representative dense prediction tasks: semantic segmentation and diffuse reflection prediction, to further support our claim. Please refer to Common Concern-1 and Sec. F of the supplementary materials for detailed information.

Q4: Do we need to fine-tune the VAE or decoder for dense representation?

We do not fine-tune the pre-trained VAE and only adapt the annotations into 3-channel input. Depth maps are repeated to 3 channels, surface normal maps (which originally are 3-channels) remain unchanged, and segmentation maps are converted to color maps. The generalization capability of VAE on dense annotations is also validated in previous works, e.g., Marigold[1] and GeoWizard[2].

[1] Repurposing diffusion-based image generators for monocular depth estimation.

[2] Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image.

Q5: How did the author evaluate the generative methods given the randomness?

All evaluations are performed with the same seed across Lotus and all baselines. To ensure a fairer comparison, we also report the mean and variance values of the metrics for our Lotus over 10 independent runs, as shown in the tables below. These results show that the variance of our method is small, only approximately e-3~e-5, and the mean values across 10 runs are consistent with those reported int the Tab. 1 and Tab. 2 of the main paper, highlighting the robustness of our method.

We have updated these experiments in Sec. H of our supplementary material for your kind reference.

The Lotus-G’s results in depth estimation, in true-depth space.

The Lotus-G’s results in normal estimation.

Dear Reviewer kYir,

Thank you for taking the time to review our paper and provide your valuable feedback. We have submitted detailed responses to your comments and suggestions, addressing the key points raised. As the discussion is scheduled to end on November 26th, we would greatly appreciate it if you could review our responses and provide any further clarifications at your earliest convenience. Your feedback is extremely important to us.

Thank you again for your time and effort.

Best,

Authors of Submission 13

Dear Reviewer kYir,

I hope this message finds you well.

Thank you for your time and effort for reviewing our submission. As the discussion deadline (November 26th) approaches, we would greatly appreciate it if you could review our responses and provide any further clarifications at your earliest convenience. Your feedback is extremely valuable!

Thanks again for your time and effort!

Best,

Authors of Submission 13

One possible ablation might be taking the same architecture, and training the model from scratch.

Thanks for your valuable feedback. We have trained our Lotus-G and Lotus-D models from scratch (without pre-trained diffusion priors), as shown in the table below. All models were trained on the same mixed dataset as in our main paper and evaluated on the NYUv2 and KITTI datasets. These experiments demonstrates that “the good performance comes from the diffusion priors”, rather than the model structure itself, especially in zero-shot scenarios with limited training data. Based on this insight, we would like to emphasize that the goal of our paper is to investigate how to effectively and efficiently leverage diffusion priors for dense prediction tasks.

And taking a pre-trained diffusion model as feature extractor or training from a pre-trained diffusion model are not that novel.

We appreciate the reviewer’s feedback, but we believe there is a misunderstanding regarding the novelty and contribution of our work.

We agree that adopting pre-trained diffusion models into dense prediction tasks is a promising trend and has shown good performance. However, existing methods still face challenges in both accuracy and inference time. This is because they just directly adopt pre-trained diffusion models without carefully exploring more suitable formulations for dense prediction tasks. In our paper, we provide a systematic analysis and conduct extensive experiments on the key factors of diffusion formulation, offering significant insights in this area. Based on these insights (which are not a shot in the dark), we propose a more tailored and efficient adaptation protocol for dense prediction, rather than simply reusing existing designs or the UNet structure.

We would like to clarify that contributions to a field do not solely lie in proposing new model structures or very novel paradigms (e.g., single-step diffusion, there are many follow-up works after the famous consistency model, many of them use the similar distillation idea). New insights and a deeper understanding can be obtained when a model or paradigm is effectively adapted to new task (e.g., from generation to perception) are equally important and can lead to meaningful advancements. The effectiveness of Lotus is mainly demonstrated by SoTA (or comparable to SoTA) performance with minimal of training data.

We highlight the key contributions of our work:

1. We systematically analyze the diffusion formulation and find its parameterization type, designed for image generation, is unsuitable for dense prediction. Specifically, we observe that $x_0$-prediction has lower variance than $\epsilon$-prediction, leading to improved accuracy.

2. Based on our analysis, we also find that the computation-intensive multi-step diffusion process is also unnecessary and challenging to optimize. We propose fine-tuning the pre-trained diffusion model with fewer training time-steps. Under limited training data, the single-step formulation is the most efficient way to leverage the diffusion priors and achieves promising performance in both accuracy and inference time.

3. We introduce a novel regularization strategy, called Detail Preserver, to improve prediction accuracy in detailed areas. We also analyze its effectiveness from a frequency perspective in Sec. G of the supplementary material and report the ablation studies in Tab. 3 of the main paper.

Regarding the Section F of Supp, do we have some comparison with other SoTA on these two tasks?

The comparison of our method with other SoTA foundation models on semantic segmentation is shown in the table below. All other methods are evaluated on the Hypersim dataset for a fair comparison. These experiments demonstrate that the proposed diffusion adaptation protocol, Lotus, has the potential to be applied to other foundational tasks in computer vision. Our model not only outperforms our baseline method “Direct Adaption”, but also achieves comparable performance with SoTA method. We are refining the comparisons on both semantic segmentation and diffuse reflection, e.g., comparing with more SoTA methods, training Lotus with more data, training Lotus with task-specific head, etc.

We provide additional clarification regarding the novelty of our method.

We uncover a critical yet under-explored principle: we can just fine-tune the pre-trained diffusion model in only single-step with x0-prediction, which is a both more effective and efficient fine-tuning protocol for dense prediction compared to existing methods.

The exploration process should not be overlooked simply because the final method or model structure is simple. This simple and effective diffusion fine-tuning protocol is obtained through step-by-step systematic analysis, aiming to better leverage the diffusion priors for dense prediction in both accuracy and efficiency.

Thus, we kindly disagree that our method just simply reuses the UNet structure.

There are some evidences that proves the importance of diffusion priors, rather than the UNet structure itself.

1. As discussed in this authors’ response (https://openreview.net/forum?id=stK7iOPH9Q&noteId=yjMGqpefHt), the diffusion priors (pre-trained parameters) play a vital role in leveraging diffusion models for dense prediction tasks, where the performance is significantly degraded without diffusion priors, demonstrating that the good performance does not lie in the UNet structure itself.
2. The rules of basic diffusion formulations should better be followed. Violating them will lead to performance degradation. For example, in Sec. 4.2 of our main paper, we reduced the number of training time-steps of diffusion formulation to only one, and fixing the only time-step $t$ to $T$ as formulated in Eq. (5), following the basic diffusion formulation (i.e., we denoise from pure Gaussian noise, which corresponds to $t=T$, best matching the diffusion formulation). To further investigate the impact of violating the diffusion formulation, we sampled 5 time-steps at equal intervals—$t=1,250,500,750,1000$—within the range [1,T] (i.e., [1,1000]) and trained 5 diffusion models using these different time-steps. The results are shown in the table below. As observed, the model performs best when $t=T$ ($t=1000$). $t \neq T$ (means the violation of basic diffusion formulation) leads to a slight degradation in performance.

Dear Reviewer kYir,

We sincerely thank you for your time and effort for reviewing our submission, and writing your responses.

However, as the discussion deadline approaches in less than 2 days (only 1 day left for reviewers to post a message), we would greatly appreciate it if you could carefully review our latest responses and provide further clarifications at your earliest convenience. We highly value any opportunities to address any potential remaining issues, which we believe will be helpful for a higher rating. Your timely feedback is extremely important to us!

Once again, thanks so much for your time and effort, we deeply appearciate it!

Best,

Authors of Submission 13

For one-step diffusion, xt (t=1000) contains some very weak information of x0, and it may help the training. Another ablation can be adding pure random noises (break the relationship between x0 and xt) to finetune the pretrained diffusion model. My two-cents are the noise channels hare not important enough to affect the performance, so in this way of training, the performance may not drop too much. It may show that "diffusion" is not necessary for dense prediction.

In diffusion models, the forward process progressively adds Gaussian noise to clean samples according to a pre-defined variance schedule, i.e., $\beta_1,\cdots,\beta_T$: $q\left(x_t \mid x_{t-1}\right)=\mathcal{N}\left(x_t ; \sqrt{1-\beta_t} x_{t-1}, \beta_t \mathbf{I}\right)$

Let $\alpha_t=1-\beta_t$ and $\bar{a}_t$=$\prod\alpha_s,\ s\in[1,t]$, $x_t$ can be sampled as:

$q\left(x_t \mid x_0\right)=\mathcal{N}\left(x_t ; \sqrt{\bar{a}_t} x_0,\left(1-\bar{\alpha}_t\right) \mathbf{I}\right)$

Equivalently: $x_t=\sqrt{\bar{a}_t} x_0+\sqrt{1-\bar{a}_t} \epsilon, \epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$. We can define the SNR as: $\text{SNR}(t)={\overline{a}_t}/{(1-\overline{a}_t)}$. In the standard DDIM scheduler: $\sqrt{\bar{a}_T}\rightarrow0$ and $\sqrt{1-\bar{a}_T} \rightarrow1.0$, which indicates the input, i.e., $x_T$, always contains a small amount of signal of $x_0$ during training.

Breaking the relationship between $x_0$ and $x_t$ to fine-tune the pretrained diffusion model (in our single-timestep setting, it means breaking the $x_0$ and $x_T$), will be a very important ablation. We have implemented the “breaking operation” by forcing the $\sqrt{\bar{a}_T}=0$ and $\sqrt{1-\bar{a}_T} =1.0$, thus, $\text{SNR}(T)={\overline{a}_T}/{(1-\overline{a}_T)}=0$, via setting rescale_betas_zero_snr=True and timestep_spacing="trailing” in schedulers. Also as discussed in [1,2], we believe that removing the leaked GT signal will be very helpful in further understanding the performance of Lotus-G, which preserves the basic diffusion formulation, i.e., the stochastic nature. Due to limited time, our model is still training, we will add these ablation studies in the near future.

We believe that stochasticity (with well-controlled variance in the Lotus-G model, which does not affect accuracy) is a valuable characteristic, and if the performance is comparable with Lotus-D, there seems to be not very necessary to not retain it. Thus, Lotus-G, which adheres to the diffusion formulation, and Lotus-D, which does not strictly follow the diffusion formulation for better fitting the deterministic dense prediction benchmarks, are both proposed in this submission, acting as two different models to meet different needs. Also as we discussed here (https://openreview.net/forum?id=stK7iOPH9Q&noteId=2fqmzk6A1d), when using Lotus-G, as demonstrated in experiments with different time-steps, it is better to strictly follow the diffusion formulation to achieve the best performance.

[1] Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think. ACCV 2024.

[2] Common diffusion noise schedules and sample steps are flawed. WACV 2024.

To be honest, I felt excited when first seeing the title of the paper, since I guessed the paper is about training a vision foundation model from scratch by purely using image data, until I found it a finetuning work with some proposed engineering tricks. And when I read the one-step, I felt excited again and thought it provides some new idea of formulation. So somehow we are leveraging the hot topics to make the paper attractive, but yet not there.

Thanks for your feedback! The topics and ideas you mentioned is indeed fascinating! In our paper, due to existing related works suffer from limited performance and slow inference time, our primary goal is to explore how to effectively and efficiently leverage the pre-trained diffusion models for dense prediction. The fine-tuning protocol proposed in this submission is simple but effective and efficient. Moreover, it is applicable to a variety of dense prediction tasks. Now besides depth and normal estimation, experiments in Semantic Segmentation and Diffuse Reflection also support that the proposed fine-tuning protocol works more effectively and effciently compared with directly adapting the pre-trained diffusion model with standard diffusion formulation, as detailed in Sec. F and Fig. D of the supplementary material. For solidness, we will further revise our paper to better highlight our contributions in refining the comparisons on both semantic segmentation and diffuse reflection, e.g., comparing with more SoTA methods, training Lotus with more data, training Lotus with task-specific head, etc.

Overall, thanks again for your prompt response and your kind improved rating! We will update the discussion and corresponding qualitative and quantitative experiments in the camera ready and public archive versions of this submission. Once again, please accept our sincere gratitude for your positive feedback and the improved rating! We sincerely thank you for your time and effort for reviewing our submission, we deeply appreciate it!

Thank you and have a great day,

Authors of submission 13

Dear reviewer kYir,

Thanks so much for your improved rating (5→6)! We are very excited and deeply appreciate it!

Thanks so much for your really thorough and constructive review of our submission and prompt responses! The additional experiments based on your suggestions significantly strengthened the solidness of this submission. We deeply appreciate your valuable comments!

We also sincerely thank you for your further comments to this submission, please see our point-by-point responses to your latest comment below:

Could you elaborate the latest results? Are you using pure Gaussian noises with different time steps?

Thanks for your feedback. The noises in our experiments are sampled using the DDPM sampler, as in the original diffusion models, i.e., $x_{t^{\text{noise}}}$=$\sqrt{\bar{a}_{t^\text{noise}}}x_0$+$\sqrt{1-\bar{a}_t^\text{noise}} \epsilon$

In our single-step formulation, we fix the time-step $t^{\text{noise}}$ for the DDPM sampler to $t^{\text{noise}}=1000$ . However, to investigate the impact of violating the diffusion formulation, the time-step $t^{\text{denoiser}}$ for the denoiser model (UNet) can be treated as a hyper-parameter, and we experimented with 5 different values for $t^{\text{denoiser}}$. Our experiments show that the best performance is achieved when the time-step is matched. i.e., $t^{\text{denoiser}}=1000$. We will provide more details about our experiments in our main paper and supplementary material for clarity.

It is not that clear why the noise channels are necessary, given Lotus-D also work well which simply follows the traditional training using paired data and reconstruction loss. The different from previous works are a better / larger pretrained model which was used for training generative models and provided better features. And jointly training images / depth map reconstructions help to preserve the details. But why noises matter? You showed in Fig.9 and demonstrated the uncertainly regions, but could we expand the discussions?

The main reason we remain the noise input in Lotus-G is that: in our systematic analysis (for better leveraging the valuable diffusion priors for dense prediction, considering both the efficiency and effectiveness), we try to keep the basic diffusion formulation unchanged. A key factor is the stochastic nature.

Strictly speaking, the presence or absence of noise input should belong to two different tasks, because the output of generative version of Lotus (Lotus-G) is a distribution (because the input Gaussian noise represents a two-dimensional Gaussian distribution), and the output of discriminative version of Lotus (Lotus-D) is a single deterministic sample. Considering that dense prediction (and most perception tasks) is originally deterministic, and the evaluation process does not take into account the evaluation of uncertainty, besides Lotus-G, we also proposed Lotus-D to better fit the evaluation settings, which is a bold attempt. So it is understandable that Lotus-D is slightly better.

Although Lotus-G may be slightly inferior in discriminative benchmarks, does it mean it has no value? Actually not. As we mentioned before, Lotus-G has the advantage over Lotus-D that it retains the input of Gaussian noise, satisfies the basic diffusion formulation, and outputs distribution:

1. As for dense prediction, although its benchmarks currently hardly support uncertainty evaluation, in fact, for example, in monocular depth estimation, the predictions of the sky; the edge of objects that are parallel with camera rays; detailed objects that are difficult to express clearly under limited pixels such as hair; semi-transparent media (such as water, glasses); fully reflective media (such as mirrors); blurred areas, etc., naturally have uncertainties.

2. Lotus-D can output one reasonable solution, while Lotus-G can output multiple reasonable solutions for the input image. By analyzing these solutions, such as simply calculating the mean and std (uncertainty) of the depth value of each pixel, then we can infer the uncertainty maps: the area with large std values can be regarded as the area where the model is highly uncertain, i.e., low confidence. For example, when training a self-supervised model, the uncertainty map can assist in excluding low-confidence predictions and only the high-confidence predictions are used for self-training to improve the performance continuously. We believe that the uncertainty map produced by the stochastic model can also support a wide range of interesting downstream applications.