DCDepth: Progressive Monocular Depth Estimation in Discrete Cosine Domain

1. Clarification about Contribution: Thank you for your comment. We believe there may be a misunderstanding regarding our contributions for the following reasons:

   1. As stated in lines 53-54, our main contribution is the first to formulate monocular depth estimation as a progressive regression task in the discrete cosine domain, achieving state-of-the-art performance. All other reviewers consider our idea a novel insight (Reviewer HHm4), quite interesting (Reviewer fZWS), and significantly different from previous methods (Reviewer tNUP).
   2. The global-to-local or coarse-to-fine concept is broad, and researchers can achieve this in various ways. The two approaches you mentioned employ tandem networks to perform pixel-wise coarse depth estimation and final depth refinement in the spatial domain. In contrast, our method leverages discrete cosine transformation to segregate depth information into various frequency components and progressively predict them from low-frequency to high-frequency components based on an iterative machenism. Our approach is fundamentally different from the two approaches you mentioned.
   3. Besides the novel formulation for monocular depth estimation, we also proposed two innovative modules: the progressive prediction head and the pyramid feature fusion module. We demonstrated their effectiveness through ablation studies.

2. Clarification about Grouping Strategy: Thank you for your comment. The lower-frequency groups contain fewer frequency components to be predicted, while the higher-frequency groups contain more frequency components. Each group of components is predicted through one iteration, thus the average number of prediction steps for higher-frequency components is less than that for lower-frequency components. Due to space limitations, we only report part of the evolution result in Figure 2. Please refer to Figure 1 of the attached PDF for a detailed illustration.

3. Clarification about DCT-based Downsampling: Thank you for your comment. We believe you may have missed some of the explanations about the DCT-based downsampling in our main text. In Section 3.1, we review the Discrete Cosine Transform (DCT) and introduce its energy compaction property. In lines 154-162, we elaborate on the workflow of the DCT-based downsampling strategy, explaining that the key information of feature maps is preserved during downsampling by leveraging the energy compaction property of DCT. Additionally, we illustrate the workflow of DCT-based downsampling in the bottom-left corner of Figure 3.

4. Clarification about Experimental Comparison: Thank you for your comment. Compared with our method, the approaches you mentioned have significant advantages in experimental configurations. We summarize them as follows:

   Method	Backbone	Pretraining	Traning Set Size
   AiT [3]	Swin-Large V2	SimMIM	Same as Our
   Depth Anything [4]	ViT-Large	DINO V2	Over 63.5 Million
   DepthGen [5]	Efficient U-Net	Image-to-image self-supervised pretraining & supervised image-to-depth pretraining	Supervised pretraining: 8.5 million & NYU-Depth-V2 finetuning: 50000
   Our	Swin-Large V1	ImageNet-22k classification	NYU-Depth-V2: 24231

   We did not include the comparison with these methods due to the large gap in experimental configurations. To further demonstrate the advancement of our method, we integrated our method into Depth Anything and finetuned it in NYU-Depth-V2 following the settings in the Depth Anything paper. We denote the fine-tuned Depth Anything model as Depth Anything ft., and our fine-tuned method as Our ft. DA. We also compared our method with DepthGen on both NYU-Depth-V2 and KITTI Eigen datasets. The results are reported as follows:

   Method	Backbone	Abs Rel	RMSE	$log_{10}$	$\delta<1.25$	$\delta<1.25^2$	$\delta<1.25^3$
   Depth Anything ft.	ViT-Large	0.056	0.206	0.024	0.984	0.998	1.000
   DepthGen	Efficient U-Net	0.074	0.314	0.032	0.946	0.987	0.996
   Our	Swin-Large	0.085	0.304	0.037	0.940	0.992	0.998
   Our ft. DA	ViT-Large	0.055	0.204	0.024	0.985	0.998	1.000

   Method	Backbone	Abs Rel	RMSE	$RMSE_{log}$	$\delta<1.25$	$\delta<1.25^2$	$\delta<1.25^3$
   DepthGen	Efficient U-Net	0.064	2.985	0.100	0.953	0.991	0.998
   Our	Swin-Large	0.051	2.044	0.076	0.977	0.997	0.999

​ Our finetuned method outperforms the depth anything couterpart, which demonstrate the superiority of our method. Furthermore, our method significantly outperforms DepthGen on all metrics of KITTI Eigen and several metrics on NYU-Depth-V2, despite DepthGen employing a large amount of data for supervised pretraining.

5. About References: Thanks for your advice and we will revise our paper according to your suggestion.

1. Discussion on TOFDC Result: Thanks for your valuable suggestion. The TOFDC dataset is challenging due to its small number of training images with limited diversity. NewCRFs, PixelFormer, and IEBins independently predict pixel-wise depth without modeling the correlations among them. We believe this approach makes it difficult to learn general depth estimation knowledge from a small-scale dataset, causing these large models to overfit on the training set and resulting in degraded depth estimation performance. In contrast, models with smaller parameters, such as BTS and AdaBins, achieve better depth estimation results.

   Unlike the large models mentioned above, our proposed method models local depth correlations by making patch-wise predictions in the frequency domain. This approach allows our model to better exploit general depth estimation knowledge. Additionally, we include two regularizations in the training loss to encourage the model to output smoother depths, helping to avoid overfitting and achieve better performance.

2. Depth Prediction Evolution Visualization: Thank you for your professional suggestion. We have included the visualization of frequency prediction evolution in Figure 1 of the attached PDF. We will incorporate this result into our revised paper.

3. Depth Prediction Visualization in Figure 3: Thank you for your suggestion. The depth results at the bottom of Figure 3 are produced by our model. However, they appear indistinguishable due to the small resolution. We will enhance the presentation in our revised version to ensure the differences are more visible.

4. Supervision with Complete Depth Map: Thank you for your constructive advice. We will explore using the dense depth map generated by the depth completion model as supervision to improve the performance of our method in future work.

1. Table Result Presentation: Thank you for your valuable feedback! Based on your suggestions, we plan to update our tables in the following two aspects:

   1. Updating the accuracy metrics in Tables 1 and 2 to percentages and multiplying the error metrics in Table 1 by 100, keeping two decimal places for reporting quantitative results.
   2. Highlighting only the best-performing methods for each metric.

   Below are the updated comparison results of VA-DepthNet, which is closest to our method in performance on the NYU-Depth-V2 and TOFDC datasets:

   Method	Reference	Backbone	Abs Rel	Sq Rel	RMSE	$log_{10}$	$\delta<1.25$	$\delta<1.25^2$	$\delta<1.25^3$
   VA-DepthNet	ICLR-2023	Swin-Large	8.56	3.88	30.44	3.68	93.68	99.20	99.87
   Our	---	Swin-Large	8.51	3.87	30.43	3.66	93.95	99.20	99.84

   Method	Reference	Backbone	Abs Rel	Sq Rel	RMSE	$RMSE_{log}$	$\delta<1.25$	$\delta<1.25^2$	$\delta<1.25^3$
   VA-DepthNet	ICLR-2023	Swin-Large	0.234	0.029	0.619	0.373	99.550	99.890	99.969
   Our	---	Swin-Large	0.188	0.027	0.565	0.352	99.490	99.891	99.970

   In the revised version, we will update our tables according to these principles.

2. Including Publication Year and Venue Information for Comparison Methods: Thank you for your valuable advice! We will include the publication year and venue information of the compared methods in our tables to help readers better understand the context of the comparisons.

Dear Reviewer fZWS,

Thanks for your comments to our paper. As the deadline for the discussion period is approaching, we wanted to kindly remind you that your feedback is very important to us. We have submitted our responses to your comments and would greatly appreciate any additional questions or feedback you may have.

Thank you for your time and consideration.

Best regards,

The Authors

1. Clarification of Experimental Comparison: Thank you for your comment. We would like to clarify the following points:

   1. Our research goal is different from that of Depth Anything and Metric3D. We focus on the novel network and algorithm design for monocular depth estimation task, while Depth Anything and Metric3D focus on improving the generalization of the depth estimation model. This difference leads to different experimental setup for these two kinds of methods, which we summarize as follows:

      Method	Pretraining	Traning Set Size	Ratio to Our Training Set
      Depth Anything	DINO-V2	Over 63.5 Million	2621 / 6350 / 2742
      Metric 3D	ImageNet-22K	Over 8 Million	330 / 800 / 345
      Our	ImageNet-22K	NYU-Depth-V2: 24231 / TOFDC: 10000 / KITTI Eigen: 23158	1 / 1 / 1

      Due to the significant differences in the experimental setup, it is unfair to directly compare our method with these two methods, thus we do not include these comparison results in our paper.

   2. We disagree with your comment that “the methods currently compared are outdated,” as we have included recent approaches such as IEBins [1], BinsFormer [2], and VA-DepthNet [3] in our comparison, as shown in Tables 1, 2, and 3 of the main text and Table 7 of the supplementary material.

   3. Under the same experimental settings, our method surpasses existing methods on three datasets, achieving state-of-the-art performance. To further demonstrate the effectiveness of our method, we additionally designed the following comparative experiments:

      • We use the weights of Depth Anything as initialization, transplant the proposed progressive prediction head into the network of Depth Anything, and fine-tune it on NYU-Depth-V2 following the settings in the Depth Anything paper. We denote the fine-tuned Depth Anything model as Depth Anything ft., and our fine-tuned method as Our ft. DA.
      • We use the weights from Metric3D encoder as initialization and randomly initialize the decoder. Then we fine-tune Metric3D and our model on the NYU-Depth-V2 dataset for 10 epochs, denoted as Metric3D ft. and Our ft. M3D, respectively.

      The experimental results are reported as follows:

      Method	Reference	Backbone	Abs Rel	RMSE	$log_{10}$	$\delta<1.25$	$\delta<1.25^2$	$\delta<1.25^3$
      Depth Anything ft.	CVPR-2024	ViT-Large	0.056	0.206	0.024	0.984	0.998	1.000
      Metric3D ft.	ICCV-2023	ConvNeXt-Large	0.065	0.232	0.028	0.971	0.996	0.999
      Our ft. DA	---	ViT-Large	0.055	0.204	0.024	0.985	0.998	1.000
      Our ft. M3D	---	ConvNeXt-Large	0.062	0.229	0.027	0.970	0.996	0.999

      Our finetuned methods outperform their counterparts, demonstrating the superiority of our approach.

   [1] Shao et al. “IEBins: Iterative Elastic Bins for Monocular Depth Estimation", NIPS 2023.

   [2] Li et al. "BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation", TIP 2024.

   [3] Liu et al. "VA-DepthNet: A Variational Approach to Single Image Depth Prediction", ICLR 2023.

2. Clarification on Per-Pixel Basis Statement: Thank you for your feedback. On lines 27-28, we stated that “to predict depth on a per-pixel basis within the spatial domain,” which indicates that the mentioned approaches independently predict depth for each pixel in the spatial domain. In contrast, our method models local depth correlations by predicting the frequency spectrum in the discrete cosine domain. The term “per-pixel basis” refers to the output, not the input. We will enhance the expression in the revised version to clarify this distinction.

Dear Reviewer HHm4,

Thanks for your comments to our paper. As the deadline for the discussion period is approaching, we wanted to kindly remind you that your feedback is very important to us. We have submitted our responses to your comments and would greatly appreciate any additional questions or feedback you may have.

Thank you for your time and consideration.

Best regards, The Authors