Metric from Human: Zero-shot Monocular Metric Depth Estimation via Test-time Adaptation

Thank you for the time and effort to review our paper. Below please find our specific answers to the questions.

1. Practicality.

   We acknowledge that MfH is not currently efficient. The runtime shown in Figure 5 is based on a sequential generate-and-estimate process, where painted images are processed one after another, illustrating a linear correlation between computational cost and the number of painted images. In Table A, we provide a breakdown of runtime with 32 images to paint when paralleled, revealing that the majority of the time is consumed by generative painting with diffusion models. Given the rapid advancements in diffusion sampling [R1, R2], we anticipate further improvements in MfH’s inference speed in the near future.

   HMR	Generative Painting	MRDE	Optimization	Total
   2.4s	5.5s	0.1s	0.3s	8.3s

   Table A. Runtime breakdown for an input image.

   [R1] Consistency Models, ICML 2023

   [R2] One-step Diffusion with Distribution Matching Distillation, CVPR 2024

2. Offline settings.

   We appreciate your suggestion to explore offline settings, which is a direction we find promising. Both solutions have their advantages. The offline approach requires training with abundant in-the-wild images and pseudo ground truths. While being expensive to train and prone to scene dependency, it offers fast inference. In contrast, our MfH provides a more cost-effective solution that is training-free and directly benefits from advancements in support models.

3. Inherently limited by the support models.

   We agree that the performance of MfH is related to the support models. However, our experimental results demonstrate that MfH, using current HMR and generative painting models, can predict metric depths satisfactorily. We further conduct ablation study in Tables B and C to show the impacts of different generative painting models and HMR models. They further indicates the potential for improved MMDE results with more advanced support models.

   Model	$\delta_1$ $\uparrow$	AbsRel $\downarrow$	SI$_{\log}$ $\downarrow$	RMSE $\downarrow$
   SD v1.5	74.0	16.8	11.5	0.642
   SD-XL	78.5	15.9	11.3	0.533
   SD v2	83.2	13.7	9.78	0.487

   Table B. Ablation study for different generative painting models on NYUv2.

   Model	$\delta_1$ $\uparrow$	AbsRel $\downarrow$	SI$_{\log}$ $\downarrow$	RMSE $\downarrow$
   HMAR [R3]	82.0	14.2	9.83	0.489
   TokenHMR [R4]	80.4	14.9	9.55	0.495
   HMR 2.0 [20]	83.2	13.7	9.78	0.487

   Table C. Ablation study for different HMR models on NYUv2.

   [R3] Tracking People by Predicting 3D Appearance, Location & Pose, CVPR 2022

   [R4] TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation, CVPR 2024

4. Will failure cases lead to failure result?

   Our random generate-and-estimate process provides tolerance for failures in each component. Since the generative painting model can paint plausible humans in most cases, a few failures will not significantly impact the overall result. A sufficiently large number of painted images dilutes the influence of these failures, providing reasonable predictions. However, the idea of fine-tuning pre-trained Stable Diffusion for metric depth estimation is interesting, especially without metric depth annotations. We are excited to explore this dedicated problem in the future.

5. Works only under certain assumptions.

   Thank you for pointing out the potential assumptions of our method. We address each concern below:

   1. To assess whether MfH performs well with close-up or wide-view shots, we examine the MMDE results on the ETH3D dataset, which includes both indoor and outdoor scenes with various shot types. We annotate ETH3D images by differentiating their shot distances and plot the AbsRel comparisons in Figure R1 of the attached PDF. The results indicate no significant performance degradation when handling close-up or wide-view inputs, demonstrating that MfH can effectively handle general close-up and wide-view shots. Similar conclusion can also be drawn from the in-the-wild qualitative results in Figure R2. However, as we acknowledged in Section 5, MfH may not perform well in extreme cases where humans cannot be present in the scene.
   2. Since MfH only aligns the MRDE of unpainted areas ($\mathbf{D}^\text{rel}$ vs. $\{\hat{\mathbf{D}}^\text{rel}_n\}$), and MMDE of human areas ($\mathbf{D}^*_n$ vs. $\{\hat{\mathbf{D}}^\text{m}_n\}$), the potential non-human artifacts are not taken into account and will not affect the optimization process. Although artifacts can be semantic meaningless, we do not observe them significantly impacting the aligned depths of other contents, as evidenced by the point clouds in Figure 6 column 5 and 6.

6. Image inpainting technique.

   We adopt Stable Diffusion v2 for generative painting, which is based on Conditional Latent Diffusion [56]. Starting from a text-conditioned diffusion model checkpoint, the denoising UNet is finetuned with additional input channels for VAE-encoded masked image to result in a diffusion-based generative painting model. We will clarify this in our revised manuscript.

7. Optimization range of Eq. 5.

   Your understanding is correct. Eq. 5 is optimized over the pixels of humans.

Thank you for your valuable comments. We agree that including direct comparisons with recent advanced metric depth prediction methods would be beneficial. Since reviewer 4QJ3 emphasizes the importance of robustness on in-the-wild inputs, we also provide this comparison here as a reference.

For in-the-wild inputs, where ground truths are unavailable, we further conduct a user study. This study includes all images shown in Figure R2 of the rebuttal PDF with MMDE results from UniDepth-C, UniDepth-V, ZoeDepth-NK, ZeroDepth, Metric3D-v1, Metric3D-v2, DepthAnything-N, DepthAnything-K, and our proposed MfH. Participants are presented with input images and corresponding MMDE results from all methods, along with a color bar mapping depth values to colors. They are then asked to select the most reasonable MMDE result for each input sample.

To analyze the results, we take each input image as a separate sample, and add one count to the corresponding method if its MMDE result is selected as the most reasonable MMDE given the corresponding input image and the meter bar. We then calculate the selection rate for each method, representing the proportion of selected results for this method out of the total number of selections. So far, we have received 45 responses with the overall results in Table D. Further, we break down the results according to the maximum value of the meter bar as in Tables E-G.

These results indicate that our MfH method achieves the highest selection rate across all depth ranges, demonstrating its robustness. Metric3D-v2 also performs well, securing the second-highest selection rate. In contrast, other methods shows variability in performance across different depth ranges. For example, DepthAnything-N has a high selection rate for short-range inputs but is not selected in inputs with larger maximum depths. This is probably due to its scene dependency. Since it is trained on NYUv2, an indoor scene dataset, its MMDE ability focus more on short-range scenes. In our revised manuscript, we will include all MMDE results (also as qualitative comparisons), these quantitative results, and discussions. We will also keep updating the results with more responses received.

We hope this user study, along with Tables 1-2 in the main paper, and Table R1 and Figure R2 in the rebuttal PDF, offers a more comprehensive comparison between our MfH and recent advanced metric depth prediction methods.

Table D. Overall selection rate as the most reasonable MMDE result.

Table E. Selection rate as the most reasonable MMDE result for short-range (10m-15m at max) inputs.

Table F. Selection rate as the most reasonable MMDE result for medium-range (20m-40m at max) inputs.

Table G. Selection rate as the most reasonable MMDE result for long-range (80m at max) inputs.

Thank you for the time and effort to review our paper. Below please find our specific answers to the questions.

1. Experimental results.

   We acknowledge that currently our zero-shot MfH does not always outperform state-of-the-art many-shot methods. However, we would like to highlight our main contribution as pointing out the scene dependency problem of fully supervised many-shot MMDE and offering a potential solution. Our comparisons demonstrate MfH’s strong zero-shot MMDE performance across diverse scenarios, while fully supervised many-shot methods may degrade on unseen scenes. Additionally, we view MfH as a general framework for zero-shot MMDE using test-time adaptation, with performance that can be further enhanced with improved MRDE, HMR, and generative painting models.

2. MfH can benefit MMDE models like UniDepth.

   We replace the current MRDE model with UniDepth in MfH and present the MMDE results in Tables A, B, and C below. The results show that MfH with UniDepth achieves better results on iBims-1 and ETH3D, but worse on DIODE (Indoor), than original UniDepth. This confirms strong metric scale priors extracted by MfH can enhance MMDE models on unseen scenes. The degradation on DIODE (Indoor) is because it contains extreme close-up scenes, where painting humans can meet difficulties, as we acknowledge in Section 5. For such scenarios, we see potential in incorporating objects other than humans into the generate-and-estimate pipeline as metric landmarks.

   Model	$\delta_1$ $\uparrow$	AbsRel $\downarrow$	SI$_{\log}$ $\downarrow$	RMSE $\downarrow$
   UniDepth-V	79.8	18.1	10.4	0.760
   MfH (Ours) w/ Depth Anything	42.2	34.5	13.2	1.363
   MfH (Ours) w/ UniDepth-V	43.5	32.6	11.9	1.390

   Table A. Performance comparisons of our MfH with different MRDE methods and UniDepth on DIODE (Indoor).

   Model	$\delta_1$ $\uparrow$	AbsRel $\downarrow$	SI$_{\log}$ $\downarrow$	RMSE $\downarrow$
   UniDepth-V	23.4	35.7	6.87	1.063
   MfH (Ours) w/ Depth Anything	67.7	23.3	9.73	0.738
   MfH (Ours) w/ UniDepth-V	69.8	20.0	8.99	0.664

   Table B. Performance comparisons of our MfH with different MRDE methods and UniDepth on iBims-1.

   Model	$\delta_1$ $\uparrow$	AbsRel $\downarrow$	SI$_{\log}$ $\downarrow$	RMSE $\downarrow$
   UniDepth-V	27.2	43.1	8.93	1.950
   MfH (Ours) w/ Depth Anything	47.1	24.0	8.16	1.366
   MfH (Ours) w/ UniDepth-V	51.2	25.5	9.17	1.489

   Table C. Performance comparisons of our MfH with different MRDE methods and UniDepth on ETH3D.

3. Detailed analysis of human scale priors with respect to scenes.

   To analyze the contribution of metric information from humans, we look into the MMDE results on ETH3D, which includes both indoor and outdoor scenes with diverse type of shots. Specifically, we annotate ETH3D images with two shot-related attributes and plot the AbsRel comparisons in Figure R1 of the attached PDF. They confirm that MfH can robustly recover metric depths, as it consistently achieves low errors across various type of shots. We also identify that the metric information from humans helps the most for level-angle inputs. This is likely because MRDE models tend to interpret similar semantics, such as different parts of a human body, as having similar depths. This interpretation aligns well with standing humans, which are typically generated in level-angle images. Moreover, we do not observe significant degradation with varying the distance of shots. This indicates MfH can effectively handle general close-up and wide-view shots. We will include these analysis in our revised manuscript.

4. Dependency on human recovery models and generative painting models.

   We acknowledge MfH depends on HMR and generative painting models. To assess their impacts, we conduct ablation study in Tables D and E. Table D demonstrates that MfH combined with stronger generative painting models yields better performance, likely due to its superior ability to generate realistic paintings. It is possible that if a generation model can better capture the real-world 2D image distributions, it has a better sense of scale, serving as a more effective source of metric scale priors. Table E indicates that different HMR models do not significantly affect MfH’s performance, probably because current HMR models can robustly assist in extracting metric information for MMDE within our MfH framework. Overall, we anticipate better MMDE results with more advanced support models, which can be integrated in a plug-and-play manner.

   Model	$\delta_1$ $\uparrow$	AbsRel $\downarrow$	SI$_{\log}$ $\downarrow$	RMSE $\downarrow$
   SD v1.5	74.0	16.8	11.5	0.642
   SD-XL	78.5	15.9	11.3	0.533
   SD v2	83.2	13.7	9.78	0.487

   Table D. Ablation study for different generative painting models on NYUv2.

   Model	$\delta_1$ $\uparrow$	AbsRel $\downarrow$	SI$_{\log}$ $\downarrow$	RMSE $\downarrow$
   HMAR [R1]	82.0	14.2	9.83	0.489
   TokenHMR [R2]	80.4	14.9	9.55	0.495
   HMR 2.0 [20]	83.2	13.7	9.78	0.487

   Table E. Ablation study for different HMR models on NYUv2.

   [R1] Tracking People by Predicting 3D Appearance, Location & Pose, CVPR 2022

   [R2] TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation, CVPR 2024

Thank you for your valuable comments and for acknowledging that most of your questions have been resolved. We agree that including direct comparisons with recent advanced metric depth prediction methods would be beneficial. Since reviewer 4QJ3 emphasizes the importance of robustness on in-the-wild inputs, we also provide this comparison here as a reference.

For in-the-wild inputs, where ground truths are unavailable, we further conduct a user study. This study includes all images shown in Figure R2 of the rebuttal PDF with MMDE results from UniDepth-C, UniDepth-V, ZoeDepth-NK, ZeroDepth, Metric3D-v1, Metric3D-v2, DepthAnything-N, DepthAnything-K, and our proposed MfH. Participants are presented with input images and corresponding MMDE results from all methods, along with a color bar mapping depth values to colors. They are then asked to select the most reasonable MMDE result for each input sample.

To analyze the results, we take each input image as a separate sample, and add one count to the corresponding method if its MMDE result is selected as the most reasonable MMDE given the corresponding input image and the meter bar. We then calculate the selection rate for each method, representing the proportion of selected results for this method out of the total number of selections. So far, we have received 45 responses with the overall results in Table F. Further, we break down the results according to the maximum value of the meter bar as in Tables G-I.

These results indicate that our MfH method achieves the highest selection rate across all depth ranges, demonstrating its robustness. Metric3D-v2 also performs well, securing the second-highest selection rate. In contrast, other methods shows variability in performance across different depth ranges. For example, DepthAnything-N has a high selection rate for short-range inputs but is not selected in inputs with larger maximum depths. This is probably due to its scene dependency. Since it is trained on NYUv2, an indoor scene dataset, its MMDE ability focus more on short-range scenes. In our revised manuscript, we will include all MMDE results (also as qualitative comparisons), these quantitative results, and discussions. We will also keep updating the results with more responses received.

We hope this user study, along with Tables 1-2 in the main paper, and Table R1 and Figure R2 in the rebuttal PDF, offers a more comprehensive comparison between our MfH and recent advanced metric depth prediction methods.

Table F. Overall selection rate as the most reasonable MMDE result.

Table G. Selection rate as the most reasonable MMDE result for short-range (10m-15m at max) inputs.

Table H. Selection rate as the most reasonable MMDE result for medium-range (20m-40m at max) inputs.

Table I. Selection rate as the most reasonable MMDE result for long-range (80m at max) inputs.

Thank you for the time and effort to review our paper. Below please find our specific answers to the questions.

1. Results from DSLR and smartphone.

   We demonstrate qualitative results for DSLR and smartphone captured images in Figure R2 of the attached PDF. Depth predictions are truncated at different maximum values and displayed in color maps. These results show that our MfH can handle in-the-wild MMDE well, even for inputs with distortions, e.g., the first and last rows of the right column. Also, we observe fully supervised MMDE methods like UniDepth often provide bounded metric depths, inheriting from the limited range of sensors used in their training ground truths. In contrast, our MfH can provide more flexible results.

2. Practical issue when the camera deviates from pinhole.

   We do observe distortions in some in-the-wild images, where using a linear metric head might not be ideal since some pixels might appear closer than the others. For general slight distortions, we expect the MRDE model, as one component of MfH, to handle these, as it leverages abundant training data which might contain distortions. To further compensate for stronger distortions, we can use a radius-related metric head to transform relative depths to metric depths, formally, the scale $s = s(r)$ and the translation $t = t(r)$. Then we believe our MfH can still work since our generate-and-estimate strategy with random painting allows local optimization of scales and translations.

3. Reference errors.

   We appreciate your suggestion and have checked and fixed the reference errors accordingly in our revised manuscript.

4. Source code and pre-trained weights.

   We appreciate your suggestion and will release our source code and pre-trained weights of support models upon acceptance.

Thanks authors.

I have checked your rebuttal, and do not have questions anymore.

Best,

Reviewer bPK1.

Thank you for the time and effort to review our paper. Below please find our specific answers to the questions.

1. Comparisons with state-of-the-art methods.

   We will update Table 1 in our revised manuscript as Table R1 in the attached PDF. Our previous Table 1 aims to provide a fair comparison based on the availability of metric depth annotations (the number of shots). So we focus on zero/one/few-shot methods, excluding many-shot methods like Metric3D and UniDepth.

2. Comparative experiments with different generative painting methods.

   In Table A, we ablate the effect of using different generative painting models in MfH. The results indicate that current generative painting models generally work well with MfH in MMDE. MfH combined with SD v2 produces the best outcomes, likely due to its superior ability to generate realistic paintings. It is possible that if a generation model can better capture the real-world 2D image distributions, it has a better sense of scale, serving as a more effective source of metric scale priors. Hence, we anticipate further performance gain of MfH with more advanced generative painting models.

   Model	$\delta_1$ $\uparrow$	AbsRel $\downarrow$	SI$_{\log}$ $\downarrow$	RMSE $\downarrow$
   SD v1.5	74.0	16.8	11.5	0.642
   SD-XL	78.5	15.9	11.3	0.533
   SD v2	83.2	13.7	9.78	0.487

   Table A. Comparative experiments with different generative painting models on NYUv2.

3. Comparative experiments with different HMR methods.

   In Table B, we examine the effect of different HMR models in MfH. We find that our approach is not sensitive to such changes. This suggests that humans can serve as relatively universal landmarks for deriving metric scales from images. Also, current HMR models can robustly help extract metric scales for MMDE with our MfH framework.

   Model	$\delta_1$ $\uparrow$	AbsRel $\downarrow$	SI$_{\log}$ $\downarrow$	RMSE $\downarrow$
   HMAR [R1]	82.0	14.2	9.83	0.489
   TokenHMR [R2]	80.4	14.9	9.55	0.495
   HMR 2.0 [20]	83.2	13.7	9.78	0.487

   Table B. Comparative experiments with different HMR models on NYUv2.

   [R1] Tracking People by Predicting 3D Appearance, Location & Pose, CVPR 2022

   [R2] TokenHMR: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recovery

4. Discussion and analysis of failure cases.

   We show three typical failure cases in Figure 8 of our appendix in the main PDF. They include 1) the generative painting model producing non-human objects with human-like appearances, 2) the generative painting model incorrectly capturing the scene scale and producing out-of-proportion humans, and 3) the HMR model predicting meshes that penetrate each other. Since the generative painting model can paint plausible humans in most cases, a few failures will not significantly impact the overall result. Our random generate-and-estimate process with sufficient painted images makes MfH robust to outliers. Further, we speculate prompt engineering, as well as better sampling and filtering strategies in human painting, can improve the performance of MfH.

5. Discussion of limitations.

   We discuss the limitations of MfH in Section 5, pointing out two main assumptions MfH based on:

   1. We assume humans can exist in the scene so that generative painting is possible to paint humans on the input image. While this holds for most usages of MMDE, it might not be ideal for some cases, e.g., close-up scenes. To this end, one future direction is incorporating objects other than humans into the generate-and-estimate pipeline as metric landmarks.
   2. We assume the MRDE predictions align with true metric depths up to affine. Since the MRDE predictions can contain non-linear noises, a simple linear metric head as in MfH might not be optimal. Exploring alternative parameterizations of the metric head remains an open question.

6. Computational complexity.

   We acknowledge that MfH is not currently efficient. The runtime shown in Figure 5 is based on a sequential generate-and-estimate process, where painted images are processed one after another. In Table C, we provide a breakdown of runtime with 32 images to paint when paralleled, revealing that the majority of the time is consumed by generative painting with diffusion models. Given the rapid advancements in diffusion sampling [R3, R4], we anticipate further improvements in MfH’s inference speed in the near future.

   HMR	Generative Painting	MRDE	Optimization	Total
   2.4s	5.5s	0.1s	0.3s	8.3s

   Table C. Runtime breakdown for an input image.

   [R3] Consistency Models, ICML 2023

   [R4] One-step Diffusion with Distribution Matching Distillation, CVPR 2024

7. Human bodies not completely overlapping with human point clouds.

   Since we only optimize a scale and a translation to convert MRDE to MMDE, each point cloud is stretched with the same set of scale and translation. Hence, we do not expect all human bodies to overlap perfectly with their corresponding point clouds but try to seek a “mode” of MRDE-to-MMDE transformation. This simplistic parameterization also provides a regularization against outliers during the generate-and-estimate process.

8. In-the-wild evaluation.

   We present qualitative results for DSLR and smartphone captured images in Figure R2 of the attached PDF. These results demonstrate our MfH can effectively handle in-the-wild inputs. Also, we observe fully supervised MMDE methods like UniDepth often provide bounded metric depths, inheriting from the limited range of sensors used in their training ground truths. In contrast, our MfH can provide more flexible results.

For in-the-wild inputs, where ground truths are unavailable, we further conduct a user study. This study includes all images shown in Figure R2 of the rebuttal PDF with MMDE results from UniDepth-C, UniDepth-V, ZoeDepth-NK, ZeroDepth, Metric3D-v1, Metric3D-v2, DepthAnything-N, DepthAnything-K, and our proposed MfH. Participants are presented with input images and corresponding MMDE results from all methods, along with a color bar mapping depth values to colors. They are then asked to select the most reasonable MMDE result for each input sample.

To analyze the results, we take each input image as a separate sample, and add one count to the corresponding method if its MMDE result is selected as the most reasonable MMDE given the corresponding input image and the meter bar. We then calculate the selection rate for each method, representing the proportion of selected results for this method out of the total number of selections. So far, we have received 45 responses with the overall results in Table D. Further, we break down the results according to the maximum value of the meter bar as in Tables E-G.

These results indicate that our MfH method achieves the highest selection rate across all depth ranges, demonstrating its robustness. Metric3D-v2 also performs well, securing the second-highest selection rate. In contrast, other methods shows variability in performance across different depth ranges. For example, DepthAnything-N has a high selection rate for short-range inputs but is not selected in inputs with larger maximum depths. This is probably due to its scene dependency. Since it is trained on NYUv2, an indoor scene dataset, its MMDE ability focus more on short-range scenes. In our revised manuscript, we will include all MMDE results (also as qualitative comparisons), these quantitative results, and discussions. We will also keep updating the results with more responses received.

We hope this user study, along with Tables 1-2 in the main paper, and Table R1 and Figure R2 in the rebuttal PDF, offers a more comprehensive comparison between our MfH and recent advanced metric depth prediction methods. We sincerely hope this addresses your concerns regarding in-the-wild comparisons, and will appreciate it if you could kindly reconsider the rating. Thank you.

Table D. Overall selection rate as the most reasonable MMDE result.

Table E. Selection rate as the most reasonable MMDE result for short-range (10m-15m at max) inputs.

Table F. Selection rate as the most reasonable MMDE result for medium-range (20m-40m at max) inputs.

Table G. Selection rate as the most reasonable MMDE result for long-range (80m at max) inputs.