Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation

Thank you for your thoughtful review and for recognizing the strengths of our paper. We appreciate your feedback and would like to address your concerns:

Q1. Technical contribution and novelty:

While we acknowledge that utilizing unlabeled data is not a new concept, our work makes several novel contributions specific to 360-degree depth estimation:

• a) We introduce a novel cube projection technique with random rotation preprocessing, which significantly reduces cube artifacts and improves depth estimation quality for 360-degree images with cross camera projection knowledge distillation.

• b) We develop a tailored masking strategy using SAM for 360-degree images, addressing the sky and watermark regions that are undesire in real-world scenes but don’t often appear in datasets.

• c) Our method is architecture-agnostic and can be applied to various 360-degree depth estimation models, as demonstrated in our experiments with UniFuse, BiFuse++, and additional models such as HoHoNet[33] (horizontal compression-based) and EGformer[50] (transformer-based):

  Method	Train	Test	Abs Rel ↓	δ₁ ↑	δ₂ ↑	δ₃ ↑
  HoHoNet	M-all	SF	0.095	0.906	0.975	0.991
  HoHoNet (our)	M-all, ST-all (p)	SF	0.088	0.920	0.979	0.992
  EGformer	M-subset	SF	0.169	0.764	0.924	0.972
  EGformer (our)	M-subset, ST-all (p)	SF	0.148	0.814	0.946	0.982

  These results show that our method generalizes well to different architectures, highlighting its broader impact on the field.

Q2. Affine-invariant loss and metric scale:

We understand your concern about the loss of metric scale. The affine-invariant loss, as you mentioned, is designed for cross-dataset training to leverage more data for real-world applications, which aligns with our approach of using both labeled and unlabeled datasets. Several previous works, such as MiDaS[4] and Depth Anything[45], also employ this training technique. To address the issue of metric scale, we will perform metric depth fine-tuning with our pretrained model in the final version. For real-world applications, we demonstrate exceptional improvement on in-the-wild scenes, as shown in Figure 2 of the PDF and Figure 6 of the original paper.

Additionally, we'd like to highlight some key aspects of our work that underscore its significance:

1. Bridging the gap between perspective and 360-degree depth estimation: Our method effectively transfers knowledge from well-established perspective depth models to the 360-degree domain, addressing the scarcity of labeled 360-degree data.

2. Improved generalization: We've conducted zero-shot experiments on unseen datasets, demonstrating our method's ability to generalize in Table 3 in the main paper. Our method especially shines in in-the-wild scenarios which can be observed in Figure 6 in the main paper and Figure 2 of the pdf.

3. Scalability: Our approach allows for the efficient utilization of large amounts of unlabeled 360-degree data, which is increasingly available but often underutilized in current methods.

We believe these points, combined with the additional experiments and clarifications, address your concerns about the technical contribution and real-world applicability of our work. Our method not only improves upon existing 360-degree depth estimation techniques but also introduces novel concepts that can be broadly applied in the field.

Thank you again for your valuable feedback. We are confident that addressing these points will strengthen our paper and highlight its contributions to the field of 360-degree depth estimation.

Thank you for your thorough review and constructive feedback. We appreciate your recognition of our method's strengths and innovative techniques. We'll address your concerns and questions point by point:

Q1. Applicability to non-dual-projection fusion models:

We acknowledge this limitation in our current evaluation. To address this, we've conducted additional experiments with HoHoNet[33] (horizontal compression-based) and EGFormer[50] (transformer-based):

These results further demonstrate our method's effectiveness across different architectures. We will include these analyses in the paper and discuss the broader applicability of our approach. Due to the limited time available for rebuttal and the lengthy process of reproducing EGformer[50], we ran it on a subset of Matterport3D, with the size of $1/5$, and will add the results from the full set to our final version. For PanoFormer, we would like to clarify that it was initially included in our model list, but we removed it due to incorrect implementation and evaluation in their official code, which led to a marginal difference between the official code and the results reported in their paper.

Q2. Cross-domain experiments (synthetic to real):

Thank you for the insightful suggestion. We agree that training on synthetic data for real-world scenarios is an important research direction. Our method focuses on leveraging the large amount of unlabeled real-world data, whereas synthetic data often includes depth ground truth and emphasizes domain adaptation. We will add a literature review on these two distinct research topics.

Q3. Median Alignment and Fair Comparisons:

We apologize for any confusion regarding median alignment. All methods discussed in our paper (Tables 2, 3, and 4 in the original paper) were re-trained using affine-invariant loss for relative depth training, with the same evaluation criteria applied for a fair comparison. The metric depth values presented in the paper are listed only for reference and are not used for direct comparison.

Q4. Additional Related Work:

We have conducted additional experiments, as shown in the table in Q1, and will include these results in the final version.

Q5. Complexity and Computational Resources:

We would like to clarify that our method does not require stereo pair generation nor synthetic generation. While our method does introduce additional complexity during training, this is a one-time cost. During the inference stage, the runtime and complexity are the same as those of the chosen SOTA 360 methods.

Q6. Dataset Dependency and Generalizability:

Our proposed method demonstrates strong generalization ability across algorithms, as evidenced by the interchangeable use of SOTA 360 depth models in the table in Q1 and in Table 3 of the original paper. Our method particularly excels in in-the-wild real-world scenarios, showing significant improvements in generalization ability on diverse data, as illustrated in Figure 2 of the PDF and Figure 6 of the original paper.

These results demonstrate our method's ability to generalize to unseen datasets, supporting its effectiveness in real-world scenarios.

We sincerely appreciate your valuable feedback, which has helped us identify areas for improvement and additional experiments. We believe these changes and clarifications significantly strengthen our paper and address the main concerns raised in your review. Thank you for your time and expertise.

Thank you for your suggestion. In addition to those already cited in the original paper, we will add the following papers to our related work discussion.

Panoramic Depth Estimation

• $𝑆^2$ Net: Accurate Panorama Depth Estimation on Spherical Surface
• High-Resolution Depth Estimation for 360-degree Panoramas through Perspective and Panoramic Depth Images Registration
• Adversarial Mixture Density Network and Uncertainty-based Joint Learning for 360 Monocular Depth Estimation
• Learning high-quality depth map from 360 multi-exposure imagery
• Distortion-Aware Self-Supervised 360 Depth Estimation from A Single Equirectangular Projection Image
• SphereDepth: Panorama Depth Estimation from Spherical Domain
• 360 Depth Estimation in the Wild -- the Depth360 Dataset and the SegFuse Network
• GLPanoDepth: Global-to-Local Panoramic Depth Estimation
• Neural Contourlet Network for Monocular 360 Depth Estimation
• HiMODE: A Hybrid Monocular Omnidirectional Depth Estimation Model
• Geometric Structure Based and Regularized Depth Estimation From 360 Indoor Imagery
• Deep Depth Estimation on 360 Images with a Double Quaternion Loss

Panoramic Depth Completion

• Cross-Modal 360° Depth Completion and Reconstruction for Large-Scale Indoor Environment
• 360 ORB-SLAM: A Visual SLAM System for Panoramic Images with Depth Completion Network
• Deep panoramic depth prediction and completion for indoor scenes
• Multi-Modal Masked Pre-Training for Monocular Panoramic Depth Completion
• Distortion and Uncertainty Aware Loss for Panoramic Depth Completion
• Indoor Depth Completion with Boundary Consistency and Self-Attention

Please let us know if any specific paper is missing.

Thank you for your detailed review and constructive feedback. We appreciate your recognition of our method's strengths and will address your concerns point by point:

Q1. Indoor/Outdoor Data Clarification:

We apologize for the confusion. While Figure 2 in the original paper shows outdoor examples of pseudo labels, our pseudo label dataset actually consists of both indoor and outdoor scenes. In the indoor panoramas from the pseudo label dataset, regions with infinite distance or no physical meaning, such as windows and watermarks, still appear frequently. SAM segments help mask out these undesired regions during the loss calculation in the training phase of depth estimation.

Q2. Baseline Model Selection:

We acknowledge your point about code availability, and our previous statement was imprecise. HRDFuse (https://github.com/haoai-1997/HRDFuse) does not provide any README documentation or instructions, while BiFuse (https://github.com/yuhsuanyeh/BiFuse) does not provide their training code. We attempted to reproduce their results but were unsuccessful. PanoFormer was initially included in our model list, but we removed it due to incorrect implementation and evaluation of their official code, which led to a marginal difference between their official code and the paper results. We have conducted additional experiments on HoHoNet[33] and will include them in the final version. Due to the limited time available for rebuttal and the lengthy process of reproducing EGformer[50], we ran it on a subset of Matterport3D, with the size of $1/5$, and will add the results from the full set to our final version.

We have shown the results in the table, where both HoHoNet[33] and EGformer[50], when trained with our semi-supervised approach, demonstrate consistent improvement.

Q3. Performance on UniFuse with SP-all (p):

Our improvements may appear small in some datasets quantitatively, but they consistently enhance performance across different methods and datasets. We have presented the results in the table in Q2, where both HoHoNet and EGformer[50] show significant improvements with our proposed method. Our analysis suggests that UniFuse's architecture is less capable of leveraging additional unlabeled data, as SP-all (p) differs more from SF. However, UniFuse with SP-all (p) shows extraordinary improvements in in-the-wild scenes, as demonstrated in Figure 2 of the PDF and Figure 6 of the original paper. We will include a discussion of this limitation and potential improvements in the final version.

Q4. Typographical Errors:

Thank you for catching these. The correct Abs Rel value for BiFuse++ (M-all, SP-all(p)) in Table 3 is 0.086. We will correct this and the "UniFise" typo in Table 2.

Q5. Semi-Supervised Learning Ratios:

Your point about exploring different ratios is well-taken. We've conducted additional experiments with varying ratios in the following Table:

These results show our method is robust across different ratios starting from 1:1. We will include this analysis in the paper.

Q6. Adaptability to Other Models

We applied our methods to HoHoNet[33] (horizontal compression-based) and EGFormer[50] (transformer-based) and demonstrated significant improvements in these two non-dual projection methods, as shown in Q2.

Q7. Projection Method:

We appreciate your suggestion about cube projection vs. tangent projection. We will conduct comparative experiments using both projection methods, including the 10 and 18 face polyhedron suggestions from [1]. These results will be included in the final version to provide empirical evidence for our projection method choice.

Thank you again for your valuable feedback. We believe these additions and clarifications will significantly strengthen our paper.

Thank you for your detailed review of our paper. We appreciate your recognition of our method's novelty and practical utility. We'd like to address your concerns and questions:

C1. Regarding the straightforwardness of our method:

While our approach may appear straightforward, we believe this simplicity is a strength. It allows for easy integration with existing models and datasets. The performance gains, which you describe as marginal, are actually significant in the context of 360-degree depth estimation. For example, on the Stanford2D3D zero-shot test, our method improves AbsRel from 0.09 to 0.082 for BiFuse++, a 8.9% relative improvement. In computer vision, such improvements are considered substantial, especially for zero-shot scenarios. Moreover, our method demonstrates extraordinary improvements in the in-the-wild scenes, as shown in Figure 6 of the original image and Figure 2 of the PDF.

C2. Limited application to specific models:

We chose UniFuse and BiFuse++ as they are widely recognized baselines in 360-degree depth estimation. However, our method is designed to be model-agnostic. To demonstrate this, we've run additional experiments with a third model, HoHoNet[33]. Another method that has been mentioned by reviewers is EGformer[50]. Due to limited of training time during the rebuttal period, we have conducted zero-shot experiments on a subset of Matterport3D with $1/5$ of the original dataset size for EGformer[50]. The full dataset version will be added to the final version.

These results further demonstrate our method's effectiveness across different architectures. We will include these analysis in the paper and discuss the broader applicability of our approach.

C3. Inconsistencies in decimal points:

We apologize for this oversight. We will standardize all results to three decimal places for clarity and ease of comparison in the final version.

Regarding your specific questions:

Q1. Choice of UniFuse and BiFuse++:

We chose UniFuse and BiFuse++ as they are widely recognized baselines in 360-degree depth estimation. However, our method is designed to be model-agnostic. To demonstrate this, we've run additional experiments with HoHoNet [33] and EGformer [50] as shown in the Table in C2.

Q2. Discrepancy in Abs Rel for BiFuse++:

Thank you for catching this error. The correct value should be 0.085, not 0.858. This was a typo in our submission. We sincerely apologize for this mistake and will correct it in the final version.

Additional points:

• Broader impact: Our method addresses a significant challenge in 360-degree vision and is highly adaptable due to the interchangeable teacher and student models. This flexibility has the potential to benefit a range of applications, including virtual reality, autonomous navigation, and more.
• Efficiency: Our approach allows for better utilization of limited 360-degree data by leveraging abundant perspective data, which is particularly valuable given the scarcity of labeled 360-degree datasets.

We believe these clarifications and additional results address your main concerns. Our method, while conceptually straightforward, offers significant and consistent improvements across multiple models and datasets. We'd be happy to discuss any further questions or concerns you may have.

Thank you for your thoughtful review. We appreciate your positive assessment of our work's originality, significance, and clarity. We're glad you found our approach well-motivated and effective. We'll address your questions and concerns point by point:

Q1. Comparison to FoVA-Depth (Lichy et al. 3DV 2024):

Thank you for pointing this out. While there are indeed similarities in leveraging perspective depth data and using cube maps, our work differs in several key aspects: - We focus on monocular depth estimation rather than stereo depth. - Our approach is architecture-agnostic and can benefit any 360 depth estimation network. - We introduce novel techniques like valid pixel masking and mixed labeled/unlabeled training. We will add a discussion of these similarities and differences in the related work section.

Q2. Baseline of directly projecting pseudo GT:

We have tried this setting as an initial setting, and due to the un-aligned scale, the training leads to unstable results and artifacts as shown Figure 1 of the pdf. As can be seen, our method significantly outperforms direct projection, demonstrating the value of our training approach beyond just leveraging the pre-trained model

Q3. Metric depth estimation:

We focused on relative depth estimation due to its generalizability across different datasets. However, your point is well-taken, we will follow previous work (MiDaS[4]/Depth Anything[45]) and conduct metric depth finetuning in the final version.

Q4. Scaling of pseudo GT vs. real GT:

We have conducted an ablation study on the ratio between real GT and pseudo GT in a batch. These results show our method is robust across different ratios, and shows the effectiveness from a ratio of 1:1. We will include this analysis in the paper. We will add to the final version.

Q5. Student vs. teacher network quality:

Our proposed method offers a cross-camera projection knowledge distillation. Our student models are SOTA methods designed for 360 images where as teacher model is initially designed for perspective images. Therefore, the teacher model is not yet a bottleneck due to cross-domain knowledge distillation, which is also shown in Table 3 in the original paper. We’ll include this discussion.

Regarding limitations:

• We agree that the scarcity of 360 data, even unlabeled, is a limitation. We'll explicitly mention this.
• While we focused on equirectangular images, our approach should indeed generalize to other projections. We'll note this potential extension.

Thank you again for your insightful feedback. We believe addressing these points will significantly strengthen our paper.