BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation

Thank you for the thoughtful comments. We kindly ask the Reviewer to read the top-level global response first. Our detailed responses to the comments in the weaknesses (denoted as W) and questions (denoted as Q) sections are listed below.

W-1: Motivation

Both the training data and the model architecture are important for performance. The very recent method Depth Anything V2 [R4] uses synthetic data for better details. Although promising improvements are achieved in Depth Anything V2, BetterDepth still shows better performance as discussed in the detail evaluation section of global response, thanks to the iterative refinement of diffusion models.

Enriching the training dataset helps to boost MDE performance, but (i) obtaining high-quality labels for real datasets is difficult due to depth sensor limitations, (ii) synthetic datasets offer high-quality labels but are costly to generate at scale, and (iii) training on large datasets is both time-consuming and resource-intensive.

BetterDepth efficiently combines the strengths of feed-forward and diffusion-based MDE models, achieving robust performance with fine-grained details with minimal training effort.

W-2: Table 1

(i) Feed-forward models can easily use large-scale datasets (both synthetic and real ones) to gain robust performance, e.g., Depth Anything V2 [R4], so we include both in Tab. 1 for generality. (ii) Recent works [9,11,14] validate the effectiveness of capturing fine details by training diffusion models on synthetic datasets. While techniques like depth infilling [34] enable training on real datasets, the resulting depth maps tend to be smoother due to the sparse and noisy labels. Thus, we focus on the recent diffusion-based MDE methods with synthetic data. We will explicitly indicate this to avoid confusion. (iii) Diffusion-based models could generalize better in tasks with diverse, accurately annotated datasets, e.g., recognition, but the sparse/noisy labels of real datasets and the limited diversity of synthetic datasets in the MDE task make it challenging. Tab. 2 also supports the significant performance gap between the state-of-the-art diffusion and feed-forward MDE approaches, e.g., Marigold v.s. Depth Anything.

W-3: SOTA improvements

BetterDepth aims to achieve robust MDE performance with fine-grained details. Despite achieving state-of-the-art performance, Tab. 2 cannot show the full advantages of BetterDepth (especially detail extraction) due to the sparse and noisy depth labels (Fig. A6-A15), which is also observed in [R4]. Thus, we provide the detail evaluation experiment in the global response, and the results in Tab. T3 verifies our significant improvements over the state-of-the-art model.

Q-1: Training strategies

To test patch masking with only depth labels, we use a smoothed version of depth label as conditioning to imitate the estimation of pre-trained MDE models. As shown in the table below, patch masking with only depth labels (denoted as Label Only) shows inferior performance due to the significant distribution gap of depth conditioning in the training and inference stages.

For patch masking, discarding patches will indeed result in loss of visual information. However, since BetterDepth utilizes the rich geometric prior from the pre-trained model, training on limited visual information already yields promising results, e.g., BetterDepth-2K is comparable with our full model in Tab. 2.

Q-2: Large datasets

BetterDepth aims to achieve both robust performance and fine details. Although large-scale real datasets can be employed to gain better generalizability, the sparse and noisy labels hinder models from extracting fine details. Synthetic datasets provide high-quality labels for fine detail extraction, but their low diversity limits the learned geometric priors. Similar discussions can be also found in Sec. 2-3 of [R4].

Q-3: Fine details

Training with synthetic datasets could help improve detail extraction, but the model architecture is also important. The recent Depth Anything V2 [R4] employs synthetic training data for better details, and we perform comparison in the detail evaluation section of the global response. Thanks to the iterative refinement scheme, BetterDepth shows the best performance on detail extraction (Tab. T3).

Q-4: Fig. 3

Fig. 3 is a schematic diagram where X_(M_FFD, {D_syn, D_real}) and X_(M_DM, D_syn) are fixed distributions representing the characteristics of different methods (Tab. 1). By contrast, \hat{X} indicates the learned output distribution of BetterDepth and we mainly analyze its change under different training strategies. Thus, only the \hat{X} is affected by global pre-alignment in the middle sketch. We will clarify this to avoid confusion.

Q-5: Filtering

The percentage of discarded patches on the training dataset is 36.6% in BetterDepth. Although a small $\eta$ will lead to fewer valid patches, BetterDepth works well with small-scale training datasets even under the information loss (as discussed in Q-1). We empirically find the optimal value $\eta=0.1$ to achieve the best performance balance (Fig. A2).

Q-6: Diffusion MDE + real data

Both [34] and [R5] employ depth infilling techniques to train diffusion-based MDE models on real datasets and achieve promising results. However, these works primarily show the feasibility of applying diffusion models on MDE without exploring fine detail extraction like recent methods, e.g., Marigold. Besides, they focus on in-domain testing instead of zero-shot evaluation. We will add [R5] to the related work section.

Limitation

Thank you. We will explore metric depth and improve information utilization in future works.

Thank you for your thoughtful comments. We kindly ask the Reviewer to read the top-level global response first. Our detailed responses to the comments in the weaknesses (denoted as W) and questions (denoted as Q) sections are listed below.

W-1: Contribution

Apart from the depth-conditioned diffusion refiner, a key contribution of BetterDepth is the proposed training strategies to achieve both robust performance and fine details. While Depth Anything gives state-of-the-art results, naively conditioning on it without our training strategies only yields inferior results as shown in Tab. T1 (Naive Conditioning v.s. BetterDepth) and discussed in the contribution section of the global response. In addition, the advantages of BetterDepth, e.g., lower error bar, also come from the proposed training strategies. We compare the standard deviation (std) based on the settings in Sec. D, and the results below show that the naive conditioning model is even more unstable than Marigold. This is because the injection of additional conditioning information makes it harder to determine which prior to follow, and the performance of BetterDepth further highlights the importance of our training methods.

W-2: Training strategies

To achieve robust MDE performance with fine details, the key challenge is how to ensure conditioning strengths while enabling the learning of detail refinement. We agree that our training strategies are not difficult to implement, but the motivation to use them during training is more important.

• To ensure conditioning strength, we propose to narrow the distance between depth conditioning and labels in a global-to-local manner. The pre-alignment first eliminates the global differences caused by unknown scale and shift, and the patch masking further addresses the local estimation bias in depth conditioning.
• For detail refinement, global pre-alignment and local patch masking together contribute to fine-grained detail extraction. Although significantly different regions are excluded to ensure conditioning strength, the combination of pre-alignment and patch masking still enables the learning of detail refinement as shown in Fig. S1 of the attached PDF, e.g., the basket.

Thus, the proposed training strategies are critical for better MDE performance and fine-grained details, which is also supported by the results/analyses in W-1.

Q-1: Wobbles

Diffusion-based MDE methods tend to introduce subtle variation due to the random noise in the diffusion process. This can be fixed by using the mean instead of the median in test-time ensembling (Fig. S2 in the attachment), which also achieves better performance as follows.

Q-2: Max-Pooling

Max-pooling is used to convert pixel-level masks to latent space. We follow Marigold to employ 8x8 max-pooling for mask downsampling as the VAE encoder in our diffusion model performs 8x downsampling for pixel-to-latent conversion. Since the patch size is also set to 8x8, max-pooling only happens within each patch without affecting a larger region. We will clarify this in revision.

Q-3: Outlier-aware method

Better alignment like outlier-aware methods could indeed have more patches survive for better performance. Although the data efficiency experiments (Tab. 2) show that fewer valid training patches lead to lower model performance, the models trained with small datasets, e.g., BetterDepth-2K, already achieve comparable results to our full model, indicating that limited patches can be sufficient. Nevertheless, better alignment methods could improve patch preservation to further benefit efficient training.

Q-4: Metric depth

Transferring affine-invariant depth to metric depth is a promising direction to benefit practical use but poses challenges, e.g., scale/shift ambiguity and diverse depth ranges. Nonetheless, BetterDepth shows potential to boost metric depth estimation. We employ the metric Depth Anything model and apply BetterDepth in a plug-and-play manner on zero-shot datasets iBims-1 [R6] and SUN RGB-D [R7]. Due to the unknown scale and shift in the current BetterDepth, we align the outputs of Depth Anything and BetterDepth to depth labels and compare the quality of depth maps. The table below (metrics are AbsRel $\downarrow$ / $\delta1 \uparrow$) and Fig. S3 of the attachment verify the superiority of BetterDepth. One promising next step is to model the scale/shift in an implicit learnable manner (as suggested by Reviewer EiFe), and we will further study this in future works.

Q-5: Inference

The inference time includes the pre-trained MDE model, i.e., Depth Anything. To measure the time spent at each step, we reproduce the experiment, and the inference time of the pre-trained model and diffusion model are 0.02 and 0.38 seconds per sample, respectively. The ensemble size is set to 1.

Q-6: Appendix C

The model without geometric prior uses the same network and fine-tuning method as Marigold but estimates inverse depth (following Depth Anything) instead of relative depth. For the model without image prior, we follow Stable Diffusion to train the latent UNet from scratch and keep the pre-trained VAE unchanged.

Q-7: Detail evaluation

Thanks for the suggestions. We conduct quantitative evaluation for detail extraction in the detail evaluation section of the global response, and the results in Tab. T3 demonstrates the state-of-the-art performance of BetterDepth in detail extraction.

Thank you for your thoughtful comments. We kindly ask the Reviewer to read our top-level global response first. Our detailed responses to the comments in the weaknesses (denoted as W) section are listed below.

W-1: Novelty

One of our key contributions is the proposed training strategies that combine the merits of feed-forward and diffusion-based MDE methods in an efficient manner. We agree that building a conditional diffusion model to combine two components is straightforward, but properly leveraging advantages of both is challenging. Although the depth map estimated by Depth Anything provides good initialization, directly building a diffusion refiner on top of it without our training strategies (denoted as Naive Conditioning) yields inferior results as shown below (metrics are AbsRel $\downarrow$ / $\delta1 \uparrow$).

This is because the naive conditioning overfits the small training datasets and thus underutilizes the prior knowledge in the pre-trained MDE model, resulting in degraded zero-shot performance (Lines 276-279 of the main paper). With the proposed training strategies, BetterDepth learns to utilize the geometric prior in pre-trained models for zero-shot transfer and the image prior in diffusion models for detail refinement, efficiently achieving robust performance with fine-grained details (as discussed in the contribution section of the global response).

W-2: Local patch masking

To achieve better details without disregarding the geometric prior learned in pre-trained MDE models, the key challenge is to ensure the depth conditioning strength and simultaneously enable learning detail refinement. Thus, we design the patch masking mechanism to:

• improve depth conditioning strength at local regions. By excluding significantly different patches, BetterDepth learns to follow the depth conditioning and better utilizes the geometric prior for robust estimation, which is important for zero-shot performance as the comparison shown in Section W-1 (Naive Conditioning v.s. BetterDepth).
• learn detail refinement within a reasonable range. With the filtered patches, BetterDepth learns detail refinement without overfitting the training data distribution, achieving better detail extraction while maintaining robust MDE performance. A visual example is provided in Fig. S1 of the attached PDF, where the model can learn to improve the detail of the basket (Fig. S1c) according to the depth label (Fig. S1b).

However, higher conditioning strength limits the capacity for detail refinement, and lower conditioning strength results in degraded performance, e.g., the naive conditioning model in W-1. Therefore, the threshold $\eta$ is used to balance these two contradicting properties. We conduct experiments with larger $\eta$ (0.5 and 1) and combine the results in Fig. A2 as follows (metrics are AbsRel $\downarrow$ / $\delta1 \uparrow$). Too large $\eta$ often leads to worse performance as the geometric prior is not well utilized, and we choose the optimal $\eta=0.1$ in BetterDepth.

W-3: Training samples and method performance

All the training samples are synthetic data in BetterDepth (Lines 249-253 of the main paper). For the two smaller training sets, we randomly select 400 and 2K samples from the full 74K synthetic dataset (composed of Hypersim and Virtual KITTI, which is the same synthetic dataset used in Marigold), and our experiments (Tab. 2) show promising performance of BetterDepth even with such small-scale training sets.

The effectiveness of quantitative evaluation heavily relies on the quality of depth labels in the benchmark. However, due to the limitation of depth sensors, the commonly adopted test benchmarks often contain incomplete and noisy depth labels. For example, Fig. A14-A15 illustrates significantly incorrect annotations and noise in the ground-truth of the DIODE dataset, and such label noise makes the reported metrics not fully reliable on a single dataset (similar discussions can be found in Tab. 2 and Sec. 6.1 of Depth Anything V2 [R4]). A more reliable evaluation is to compare the performance across diverse datasets, so we additionally provide the average ranking of the five benchmarks in Tab. 2 and our BetterDepth achieves the overall best performance. In addition, we further validate the superiority of BetterDepth on fine-grained detail extraction, as discussed in the detail extraction section of the global response.

I am a bit confusing: In the last row of Table T2, which method is embedded into the proposed BetterDepth?