Depth Pro: Sharp Monocular Metric Depth in Less Than a Second

Thanks for the feedback!

Figure 3: The different patches (tiles) in Fig. 3 are concatenated along the batch dimension, so that the resulting input image tensor to the ViT encoder can be batch-processed very efficiently. We clarified this in the updated paper. We also added a more detailed description of the merge operation in Sec. C.1 of the appendices.

Training data. Please note that Tab. 13 (now Tab. 15) also contains test datasets. Overall, Depth Pro is trained on approximately 5.1M real and 2.1M synthetic depth maps. This is significantly less than e.g., the 16.2M samples reported by Metric3D v2.

Performance gap to Marigold. Our controlled experiments indicate that a high native output resolution is a necessary condition for predicting high-frequency details. This alone is not sufficient however, as demonstrated by Depth Pro outperforming approaches with varying output resolutions. We observe further improvements on boundary metrics through the proposed architecture, loss functions, and training curriculum.

Varying performance on zero-shot datasets. The performance of all tested methods varies significantly across evaluation datasets. A possible explanation could be the selection of training datasets, which differs for each method and may result in biases towards certain domains. Hence, for fair comparison we purposefully collect a diverse set of zero-shot test datasets that represents different domains. On average Depth Pro outperforms all other approaches, which indicates a robust performance across diverse domains.

Thank you for the feedback!

Please find our response to the question on the most critical components for performance in the global response above and remaining responses below.

Effect of predicting inverse disparity. Thanks for the suggestion. To assess the effect of predicting inverse depth, we conducted the following experiment (Tab. 10): We trained a single ViT encoder and DPT decoder to predict inverse depth, depth, and log-depth on Hypersim. We then evaluate the accuracy in predicting depth on several depth ranges. Across all ranges we observe lower errors for predicting inverse depth than for depth or log-depth.

Tradeoff between boundary accuracy and metric depth. To shed more light on this tradeoff, we supplement our controlled experiments with boundary evaluation results. For stage 1 losses, our proposed loss configuration yields the best depth accuracy and boundaries. For stage 2 losses, our proposed configuration of first and second order losses balances metric depth and boundary accuracy. Here, removing the second order loss improves metric depth at the cost of less accurate boundaries. Further removing the MSGE loss reverses this effect. For the curriculum we observe improved metric depth by training the two proposed stages. Training only a single stage slightly improves boundaries at the cost of metric depth accuracy.

Calculating final depth. That is correct. When the focal length is available however (e.g. through the EXIF data or in the dataset), we will use it instead of the predicted focal length.

Median-scale depth in experiments. That is correct. Table 1 and 4 report absolute errors, and we do not use any kind of median scaling or alignment to the ground-truth. Boundary metrics in Table 2 are scale-invariant, which allows us to run comparisons against non-metric methods such as DPT, Marigold or DepthAnything-relative v1 and v2.

Fine-tuning experiments. In-domain fine-tuning should almost always improve the in-domain accuracy on downstream tasks, usually at the cost of diminished generalization abilities. As we focus on zero-shot generalization we deemed in-domain fine-tuning experiments beyond the scope of the paper.

Training time & compute. Depth Pro trains in approximately 50K GPU-hours on a Volta GPU with FP32 precision.

Thank you for the feedback!

Limited architectural innovation: We propose a novel architecture that leverages pretrained ViT backbones in a highly efficient way - it is more accurate and faster than upscaling existing ViT backbones (see comparison to DINOv2 in Tab. 8, comparison to Metric3D v2 in Tab. 1 & Tab. 5) and is capable of accurately tracing thin structures (Tab. 2). This is in contrast to prior state-of-the-art approaches such as DepthAnything v1 and v2, and Metric v2, which all relied on plain ViT encoders, DINOv2 weights, and a plain DPT decoder. Beyond the novel architecture of the depth estimation branch, we propose a simple yet effective approach to handing images without camera metadata. This is in contrast to almost all other approaches to metric depth estimation (which rely on ground truth metadata) and outperforms prior state of the art on focal length estimation (Tab. 3). We further propose a novel metric for assessing the quality of predicted depth boundaries and benchmark prior work. And we demonstrate the importance of accurate depth boundary prediction on three downstream applications.

Reliance on synthetic data: Although reliance on synthetic data during training could hypothetically impact generalization, we demonstrate that this is not the case here. Figures 1, 4, 5, 6, 7, 8, 9 show qualitative demonstrations of Depth Pro predicting thin structures in complex environments on unseen real-world images. Quantitative demonstrations come from the zero-shot evaluation of boundary metrics in Tab. 2, which we conduct on several datasets from different domains to specifically assess generalization to unseen environments. The quantitative and qualitative results on matting datasets in particular demonstrate excellent generalization to real-world data outside of the training domains.

Thank you for the feedback!

Limited novelty: Please note that our contributions go beyond loss functions and training strategy. As outlined in Sec. 3.1, we propose an efficient architecture that allows high-resolution metric depth prediction with excellent zero-shot capabilities. Our proposed approach reaches higher depth accuracy than alternatives (Tab. 1 & Tab. 9), produces more accurate boundaries (Tab. 2), predicts more accurate focal lengths (Tab. 3) and is more efficient than prior work (Tab. 5). In contrast to almost all other work on single image metric depth prediction, it works on images without metadata by predicting the focal length (Sec. 3.2). We further propose a novel metric for evaluating the quality of predicted depth boundaries (Sec. 3.3).

Temporal consistency: We thank eL31 for the suggestion and agree that the exploration of further applications would be interesting. As we focus on single-image depth estimation in this paper and already explore three closely related downstream applications (novel view synthesis, controlled image synthesis, synthetic depth of field) in the Appendix, we politely leave the exploration of multi-frame use cases to future work.

Method not clear / missing details. We appreciate the feedback and added additional sections on the feature merging (Sec. C.1) and focal length head (Sec. C.2) to the appendices and updated Sec. 3.1. We further updated Figure 3. We summarize here for convenience:

• Feature merging: The patch encoder ingests image patches of 384^2 and yields feature patches of 24^2. At the coarsest resolution of 384^2, we directly obtain Feature 5 of size 24^2. The same happens with the image encoder, yielding Feature 6. At the next higher scale, we split the 768^2 image into 3x3 = 9 overlapping patches of 384^2 each and obtain 9 patches of 24^2 each. The patches are merged to Feature 4 of size 48^2. For merging patches we generate a Voronoi partition of the feature map with patch centers as seeds. Each cell then corresponds to the area of a feature patch we can directly copy. Figure 11 in the appendices illustrates the merging. Features 1-3 are merged analogously.
• Patch encoder: Details on the backbone architectures can be found in Sec. B.2 of the Appendix. We studied several backbone variants (see Tab. 7 of the Appendix) and opted for a ViT-L DINOv2 architecture.
• Focal length head: The focal length encoder uses the same architecture and pre-trained weights (ViT-L DINOv2) as the image encoder and the patch encoder. Same as the image encoder, it operates on the full image downsampled to 384^2 and yields a feature tensor of 24^2. To estimate the focal length, we simply add the feature maps from both image encoders and pass them through a 3-layer convolutional network, which predicts the focal length. The network reduces the channel dimension from 128 to 64, to 32, to 1 via kernels of size 3, 3, 6, and strides 2, 2, 1. Convolutions are interleaved with ReLUs. The focal length head does not operate on multi-scale features but only features from the image encoder(s) at a single scale. As this was not clear from Figure 3, we revised it.

Effect on novel view synthesis. Thank you for the feedback, we supplemented Fig. 4 with a more detailed explanation. Novel view synthesis requires depth maps to be metric to display objects at the right size and depth boundaries to be well aligned with image edges. Fig. 4 focuses on the artifacts due to misaligned boundaries and missed thin structures. Here, all prior work produce ghosting artifacts as shown in the insets of the horse and zebra images. For the windmill, we further observe thin structures being missed by Marigold and Depth Anything v2.

Arbitrary image sizes. We resize images of all sizes to 1536^2 via bicubic interpolation for processing by our network and resize depth predictions to the original input resolution. We found this simple scheme to work robustly across arbitrary image sizes as demonstrated through its strong zero-shot performance for metric depth and boundary accuracy.

Runtime. We summarize runtimes in Fig.1, Fig. 2, and report detailed runtime numbers for various input resolutions in Sec. A.3 of the Appendix. To summarize here for convenience: Among all approaches with a fixed output resolution, Depth Pro has the highest native output resolution, processing more than 3 times as many pixels as the next highest, Metric3D v2. Yet Depth Pro has less than half the parameter count and requires only a third of the runtime compared to Metric3D v2. The variable-resolution approaches (PatchFusion and ZeroDepth) have a considerably larger runtime, with the faster model, ZeroDepth, taking almost 4 times as long as Depth Pro, even for small VGA images.