A Simple yet Universal Framework for Depth Completion

[bXoG] W.3 Not SOTA Performance on 1-shot Experiments

VPP4DC [P1], which utilizes a stereo matching network for depth completion, creates a model robust in cross-domain generalization for the task. As the reviewer mentioned, P1 achieves accurate performance by pretraining the model with a thousands number of synthetic datasets, slightly outperforming our method in the 1-shot setting. P1 estimates depth by predicting disparity and converting it with baseline and camera parameters, which might lead to slight differences in evaluation protocol. Additionally, it seems that P1 only measures performance on the left camera (Camera_02) for KITTI (50% of official validation), which could cause metric protocol discrepancies. To fairly compare the model and ours, we conduct experiments under the UniDC problem we proposed. Table.I demonstrates that while P1 performs similarly to our method in the 1-shot setting, it does not show significant improvement in the 10 or 100-shot settings. This suggests that VPP4DC, pretrained on a synthetic dataset, struggles with learning new representations. This is consistent with in-domain experiments conducted in the paper, where VPP4DC shows weaknesses in the in-domain performance, due to insufficient learning from the scene appearance in NYU and the sparse data obtained from new sensors.

[Table.I] Performance Comparison between VPP4DC and Ours

[bXoG] Minor Comments & Limitations

• [1,2] We will fix the typos in the revised version. Thanks!
• [3] Fig 1. Initial relative depth of (b) DepthPrompting. We implement the DepthPrompting [19] method with MiDaS. We report the initial depth with a metric scale which is inferred by a pretrained model in 1-shot environment. We agree that the depth map may confuse readers to understand, so we will change the depth map into the output of the pretrained depth foundation model in relative scale.
• Limitation of VFM and rescaling solution of [19]: We think the major concern is how to change the relative-scale depth in metric scale with the guidance, sparse depth from depth sensor. The pioneer work [19] successfully utilizes the output of the monocular depth model, but it shows poor adaptation performance.

[Table.G] Experiments on VFM-less Methods with VFM (RMSE/MAE/DELTA1.25)

[Table.H] Varying Density Experiment (RMSE/MAE/DELTA1.25)

If the experiment fails to optimize or diverge, they are marked with a "-"

[bXoG] W.1 Comparison with VFM-less Methods with

We acknowledge the reviewer's concern regarding the fairness of comparing our method with VFM-less approaches. We recognize that Visual Foundation Model (VFM) features provide a significant advantage due to their extensive pre-training on large datasets, which VFM-less methods lack. To address your concern, we carry out additional experiments using VFM knowledge in VFM-less baselines by replacing the sparse depth input with Eq. 4 of the paper [19].

In this experiment, we found that directly applying the VFM approach, as suggested, sometimes yields unsatisfactory performance compared to the baseline (Table.G). This underperformance can be attributed to optimization issues stemming from the fact that the sparse depth provides complete metric depth information, whereas fitting the relative-scale depth from VFM using Eq. 4 of [19] does not achieve this precision level. The fitting process involves solving ∣Ax−B∣, which performs a linear fit with the available data, i.e., sparse depth. In [19], the authors employed global linear fitting to predict the depth scale using scalar values A and B, initially converting relative depth to metric scale. However, this approach often fits disproportionately to regions with rich information, leading to inaccuracies in areas with sparse depth information. Consequently, using metric sparse depth as input can cause inaccuracies, making optimization difficult and resulting in suboptimal performance.

We agree that there is a significant gap in using VFM directly for depth completion. Instead of directly using relative-scale depth, we chose to leverage intermediate features to indirectly utilize foundation knowledge. This approach allows us to benefit from VFM while avoiding the direct application of relative-scale depth, thereby mitigating some of the challenges observed in this experiment.

[bXoG] W.2&4 Related literature, experiments, and competitors are missing

We appreciate the reviewer's suggestion to include the work of [P1] and [P2], which addresses the UniDC problem in a zero-shot manner. We agree that this is an important piece of the related work, and we will update our literature review and how it compares to our method. Additionally, we will expand our experiments to include their benchmark, as outlined in their work, to better evaluate various sensors with a few labeled data in the other domain.

Our model is not specifically designed for zero-shot generalization; rather, it addresses task-specific domain shifts by integrating relative-scale 3D knowledge learned by the VFM into a sensor fusion model, i.e., depth completion model. Our primary objective is to quickly and effectively adapt to new sensors, leveraging the generalization capability of the VFM model. While cross-domain experiments (source RGBD dataset to target RGBD dataset), as shown by P1 and P2, are important for understanding model generalization, challenges such as catastrophic forgetting or significant covariate shifts (e.g., with new sensors like 4D radar) can render existing models ineffective.

Our solution focuses on fast adaptation with minimal data, enabling strong performance even without dense labeled data, as demonstrated in Table 3 of the paper. Thanks to the reviewer's suggestion, we can showcase the superiority and applicability of our model across various environments by comparing it with advanced methods, demonstrating the feasibility to various sensors and environments.

[Full In-domain Performance]: In response to the request for full in-domain experiments, we carry out an extensive evaluation of the KITTI DC and NYU datasets. To explore our method under various configurations, we develop four variants by adjusting the number of channels, similar to the recent state-of-the-art methods like LRRU and CompletionFormer. As shown in Table.C, the results demonstrate that our method is effective in a full training setting and performs competitively against traditional methods that require extensive labeled data, confirming our approach's robustness and versatility across different training scenarios. Moreover, Table.A shows that our variants achieve competitive performance compared to well-established methods concerning specialization within a single domain. We compare those advanced methods, including their variants (Mini to Base), with ours in few-shot experiments and verify the effectiveness of our proposed method regardless of the number of model parameters.

[Varying-density Performance]: We simulate different LiDAR setups by varying the density of the input data. For the NYU indoor dataset, we randomly sampled 100 and 32 sparse depths, while for the KITTI outdoor dataset, we utilize 16-Line and 4-Line configurations. These experiments test the robustness and adaptability of our method in response to changes in input data quality and quantity. As shown in Table.H, the results show that our method achieves superior performance across different sensor configurations. In contrast, most comparison methods exhibit a decline in performance when adapting to new sensor configurations, as demonstrated in the DepthPrompting [19].

[AS7d] W.1 Experiments on other RGBD datasets

We appreciate your feedback regarding the limited use of datasets in our evaluation. In response to concerns about unseen domain generalization, we have expanded our experimental evaluation to include an additional dataset, SUN-RGBD, which contains RGB-D images from four different sensors, offering a diverse range of scenes and sensors, e.g., Intel RealSense 3D Camera, Asus Xtion LIVE PRO, Microsoft Kinect V1 and V2. The total of 1000 scenes, where each sequence is roughly 20 seconds long and annotated every 0.5 seconds, is officially split into train/val/test set with 700/150/150 scenes. According to the results of Table.B, our method consistently demonstrates improved performance across these datasets, outperforming state-of-the-art methods in new domain scenarios. These results highlight our framework's strong generalization capabilities and will be included in the revised version of this paper.

[AS7d] Q.1 Comparison of qualitative results

The difference in qualitative results results observed between the NYU and KITTI datasets can be primarily attributed to the sensor characteristics. ITTI employs a 64-line Velodyne LiDAR, which provides consistent sparse depth information at fixed locations, creating an inductive bias that simplifies optimization. The model can more easily learn from consistent depth cues provided by the LiDAR. In contrast, NYU relies on synthetic random sampling for sparse depth, leading to varying locations of depth seeds across different samples. This variability increases the difficulty of optimization, particularly in few-shot setups, as the model struggles to learn effectively from constantly changing sparse depth locations.

[Table.F] Ablation of Hyperbolic Operation (RMSE/MAE/DELTA1)

[1] LRRU: Long-short Range Recurrent Updating Networks for Depth Completion (ICCV23)

[3] Improving Depth Completion via Depth Feature Upsampling (CVPR24)

[2] OGNI-DC: Robust Depth Completion with Optimization-Guided Neural Iterations (ECCV24)

[4] Revisiting depth completion from a stereo matching perspective for cross-domain generalization (3DV24)

[5] Hyperbolic Neural Networks (NIPS 2018)

[6] Hyperbolic Neural Networks++ (ICLR 2021)

[7] Rethinking the compositionality of point clouds through regularization in the hyperbolic space (NIPS 2022)

[8] Capturing implicit hierarchical structure in 3D biomedical images with self-supervised hyperbolic representations (NIPS 2021)

[9] HypLiLoc: Towards Effective LiDAR Pose Regression with Hyperbolic Fusion (CVPR 2023)

[10] On Hyperbolic Embeddings in Object Detection (DAGM 2022)

[11] Fully Hyperbolic Convolutional Neural Networks For Computer Vision (ICLR 24)

[12] The Numerical Stability of Hyperbolic Representation Learning (ICML23)

[13] Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifiers (CVPR22)

[14] HypLiLoc: Towards Effective LiDAR Pose Regression with Hyperbolic Fusion (CVPR23)

[15] Hyperbolic Contrastive Learning for Visual Representations beyond Objects (CVPR23)

[g2HF] W/L. Comparison with other methods and experiments about unseen domain generalization.

As requested by the reviewer, we conduct additional experiments on diverse datasets beyond KITTI DC and NYU v2. We include tests on the SUN RGB-D, which show different environmental conditions and various sensor types, e.g., Intel RealSense 3D Camera, Asus Xtion LIVE PRO, Microsoft Kinect V1 and V2. According to Table.B, our method achieves consistent performance improvements for all the datasets, outperforming the state-of-the-art methods in unseen domain scenarios. Plus, we compare with other advanced methods (e.g., LRRU[1], DFU[2], OGNI-DC[3], VPP4DC[4],) to highlight the proposed method's superiority (Please refer the Table.A, [tAUP] W.2 and [bXoG] W.3).

[g2HF] Q.1 Full sequence training

In response to the request for full sequence training experiments, we carry out an extended evaluation of the KITTI DC dataset. The results, shown in Table.C, indicate that our method not only maintains its effectiveness in a full training regime but also achieves competitive performance over traditional methods that rely on extensive labeled data. This supports our claim that the proposed approach is robust and versatile across different training settings.

[g2HF] Q.2 Implementation of hyperbolic operation

As you noted, “Euclidean” refers to a space working for conventional operations like convolution. This means that all computations, including embedding and affinity computations, are performed using typical neural network operations in n-dimensional Euclidean space. Starting with the foundational work on Hyperbolic Neural Networks [5,6], there has been a significant shift towards leveraging hyperbolic operations across various downstream tasks. Typically, tensors processed in neural networks are considered within the Euclidean space. However, by embedding these tensors into hyperbolic space, we can exploit the advantages of hyperbolic geometry. This embedding enables us to perform operations directly in hyperbolic space, capturing hierarchical relationships more effectively than in Euclidean space [5,6]. Here is a pseudo-code implementation illustrating the difference between Euclidean and hyperbolic operations in our method:

Pseudo-Code

Input: image features $E^M_{L} \in \mathbb{R}^{C \times H \times W}$, initial sparse depth $S$, curvature $\kappa$, pixel coordinate $i$, and the neighboring pixel coordinate of $j$.

• $HyperbolicEmbedding$: transform function from hyperbolic to Euclidean space (Eq.4)

• $HyperbolicDistance$ : Distance function with hyperbolic features (Eq.8)

• $HyperbolicConvolution$ : convolution layer with hyperbolic features (Eq.10)

For each $e_i$ in $E_L^M$ do

1: $h_i = HyperbolicEmbedding (e_i, \kappa)$ and $h_j = HyperbolicEmbedding (e_j, \kappa)$

2: $w_{i,j} = HyperbolicDistance (h_i, h_j)$

3: $d_i =$ $\sum_{j}$ $w_{i,j} S_j$

4: $a^{hyp}_i = HyperbolicConvolution(e_i, \kappa)$

End for

Output: Hyperbolic affinity $a^{hyp}_i \in A^{hyp}$ and initial depth $d_i \in D$.

[g2HF] Q.4 Pros/cons/limitations of hyperbolic geometry for depth perception

Hyperbolic geometry offers substantial advantages in modeling hierarchical relationships within sparse data, providing new cues for understanding 3D structures in depth perception tasks[7,8,9,10]. By embedding data into hyperbolic space, we can more effectively capture complex structures, enabling a richer representation of 3D environments. This approach enhances the performance of depth completion tasks, particularly in few-shot settings, and improves generalization and adaptability in unseen domains.

Most depth completion tasks use encoder-decoder structures to extract multi-scale features and propagate sparse depth information through affinity maps. Hyperbolic space naturally accommodates exponentially growing hierarchies and tree-like structures, allowing robust affinity construction with low distortion. This enhances the distinction between unrelated pixel features and reduces bleeding errors by effectively capturing boundary information, even in ambiguous regions.

However, hyperbolic geometry remains certain challenges. One major limitation is the increasing computational complexity, which can impact both training and inference times [11,12,13], even with a similar number of parameters as highlighted in [tAUP] Q.2. Additionally, the performance sometimes depends on a hyperparameter setting for curvature value. Despite these challenges, the overall benefits of using hyperbolic geometry, such as improved generalization and performance in diverse and unseen environments, often outweigh these drawbacks, making it a valuable tool for depth estimation and completion tasks.

Considering these advantages and limitations, a strategy that combines Euclidean and hyperbolic operations could be beneficial as done in Eq.8, and [14,15]. By leveraging the strengths of both geometries, it is possible to achieve a balance between computational efficiency and the ability to capture complex hierarchical relationships in depth perception tasks, which will be mentioned in the revised version of this paper.

[g2HF] Q.3 Few shot comparisons for Euclidean and hyperbolic space

As suggested, we conduct additional experiments under the few-shot regime. The results, presented in Table.F, show a noticeable improvement when using hyperbolic space, with a performance gain of 5% on average, compared to Euclidean space. This validates the effectiveness of hyperbolic geometry in depth completion tasks, especially when dealing with limited data samples.

[tAUP] W.1 Online KITTI depth completion benchmark.

We thank the reviewer for the comments about the outstanding performance and generalization ability of our method. Following your comment, we report the performance of our work in the KITTI benchmark, which is reported in Figure.A of the uploaded PDF file. To analyze our method under various configurations (Table.C), similar to recent state-of-the-art methods such as LRRU and CompletionFormer, we designed four variants by adjusting the number of channels. Notably, both our method and these methods totally follow the scaling laws of deep learning models [1, 2]. Our variants achieve competitive results compared to the state-of-the-art methods, especially in setups with fewer labels, where our approach demonstrates superior performance.

[tAUP] W.2 Comparison with recent SoTA methods

We have provided a detailed comparison with recent state-of-the-art methods in Table.A. This comparison highlights the advantages of our approach across different experimental setups.

In the case of the LRRU family [3], we observe that in the KITTI dataset, smaller models tend to perform better, whereas in the NYU dataset, larger models achieve better performance. This behavior is attributed to the IP-Basic algorithm [4] used for preprocessing, which is biased towards the KITTI dataset. Unlike LRRU, our approach leverages foundation model knowledge, enabling consistent and rapid adaptation to both indoor and outdoor data.

Additionally, the DFU [5] model addresses the issue of sparse decoder features in encoder-decoder networks by using a depth feature upsampling method. While this approach seems to aid adaptation in few-shot setups, it appears vulnerable in indoor scenarios with varying sparse depth configurations.

OGNIDC [6] iteratively refines the depth gradient field (depth differences between neighboring pixels) and integrates this information to produce a dense depth map. This method has shown excellent performance in various zero-shot generalization experiments and satisfactory results in few-shot experiments, ranking second-best among comparing methods. Unlike other models, which highly rely on RGB features, OGNIDC focuses on refining 3D-aware information through the Depth Gradient Field, making it less sensitive to RGB appearance changes, as seen in models like CostDCNet [7] which shows superior performance than other methods in the Table.1 of the manuscript. This results in strong generalization and few-shot adaptation capabilities. However, our methodology demonstrates an ability to learn feature representations effectively in hyperbolic space using smaller datasets, leading to faster adaptation in harsh conditions compared to recent SoTA methods.

[tAUP] Q.1 Advantages of hyperbolic space for calculation of the pixel affinity map.

We appreciate the opportunity to explain why depth completion methods benefit from hyperbolic geometry. Most spatial propagation networks (e.g., CSPN, NLSPN, and DySPN) adopt encoder-decoder structures to extract multi-scale features w.r.t. structure and photometric similarities. Then, initial seeds (i.e., sparse depth) are propagated based on affinity maps computed from the learned features in an iterative manner. Therefore, if the computed affinity map is accurate, capturing boundary information, which is the highly ambiguous region for pixel-wise prediction task, is concomitant. However, object boundary ambiguities, caused by noise or smooth intensity changes, can lead to bleeding errors [8].

To address these issues, traditional studies (i.e. optimization-based approaches) for affinity construction have utilized hierarchical structures [9-13]. Like tree-based structure, non-local propagation has shown superiority over the local propagation that CSPN, NLSPN, and DySPN follow. By embedding pixel features into hyperbolic space [14,15], we formulate these hierarchical relations in a continuous and differentiable manner. The hyperbolic space naturally accommodates exponentially growing hierarchies and tree-like structures, allowing robust affinity construction with low distortion. This enhances the distinction between unrelated pixel features (i.e., low affinity), while semantically-close pixels benefit from the hierarchical structure, reducing bleeding errors.

We conducted a toy example to verify the effectiveness of hyperbolic geometry in various propagation schemes, including CSPN (Convolutional), NLSPN (Non-Local), and DySPN (Dynamic attention). Using the same backbone (ResNet-34) and loss functions (L1 and L2) across all schemes ensures a fair comparison. As shown in the Table.D, hyperbolic operations significantly improve performance in various few-shot setups. Compared to Euclidean methods, hyperbolic structures improve pixel distinction under challenging conditions. We also hope you refer to section [g2HF-Q.3] for an ablation study of hyperbolic operations in our architecture.

[tAUP] Q.2 Probe for computational costs of depth foundation model

As mentioned, depth foundation models are typically large and computationally expensive due to training on extensive datasets. However, recent models offer various variants, allowing flexibility in computational demands. As shown in Table.7 of the manuscript, we conduct ablations on multiple models and observe comparable performance across them. As shown in Table.E, MiDaS[12] and Depth Anything[13] have significantly fewer parameters than other depth completion models, suggesting that leveraging a pre-trained foundation model's knowledge does not necessarily entail high computational costs. The light-weight models can achieve sufficient generalization performance as well. Note that we use the publicly available official codes for MiDaS [16] (v2.1 Small), Depth Anything v1 [17] (ViT-S), and UniDepth [18] (ViT-L). Plus, we will mention it in the revised version of this paper.

[Table.D] SPNs (CSPN, NLSPN, DySPN) with Hyperbolic Geometry

[Table.E] Computational Cost of Depth Foundation Models

References

[1] Scaling Laws for Neural Language Models (Arxiv 2020)

[2] Scaling Vision Transformers (Arxiv 2020)

[3] LRRU: Long-short Range Recurrent Updating Networks for Depth Completion (ICCV 2023)

[4] In defense of classical image processing: Fast depth completion on the cpu (CRV 2018)

[5] Improving Depth Completion via Depth Feature Upsampling (CVPR 2024)

[6] OGNI-DC: Robust Depth Completion with Optimization-Guided Neural Iterations (ECCV 2024)

[7] CostDCNet: Cost Volume based Depth Completion for a Single RGB-D Image (ECCV 2022)

[8] Learning Affinity with Hyperbolic Representation for Spatial Propagation (ICML 2023)

[9] Tree filtering: Efficient structure-preserving smoothing with a minimum spanning tree. (TIP 2013)

[10] Fully connected guided image filtering. (CVPR 2015)

[11] A non-local cost aggregation method for stereo matching.(CVPR 2014)

[12] Stereo matching using tree filtering. (TPAMI 2014)

[13] Real-time salient object detection with a minimum spanning tree. (CVPR 2016)

[14] Representation tradeoffs for hyperbolic embeddings. (ICML 2018)

[15] Poincar´e embeddings for learning hierarchical representations. (NeurIPS 2017)

[16] Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer (TPAMI 2022)

[17] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. Foundation Model for Monocular Depth Estimation (CVPR 2024)

[18] Universal Monocular Metric Depth Estimation (CVPR 2024)

We appreciate your insights and the opportunity to address your concerns.

Regarding the performance on the online KITTI benchmark, we acknowledge that the error is higher compared to the current state-of-the-art methods. “However, the primary objective of our work was to explore the potential of large-model prompts in enhancing adaptation ability to arbitrary sensors even with limited labeled data, rather than solely outperforming previous works only focused on the KITTI dataset." Additionally, due to limited time in the rebuttal period, we couldn't fully engineer the performance for the rebuttal period. We now invite you to review the benchmark performance of our Ours_Base model, as presented in Table A and Table C. This model, listed as "UniDC Base" on the official KITTI benchmark homepage, achieved an RMSE of 736.0mm and an MAE of 202.4. As shown in the table below, the result shows that our method outperforms the relevant works in [1,2,3] which are recently published and have almost the same purpose as ours. While these results may not match the very latest SoTA, we hope you will consider the broader motivation behind our work.

• Results for models [1] and [2] are sourced from their respective papers, while [3] represents the KITTI online leaderboard benchmark. *

[1] Flexible Depth Completion for Sparse and Varying Point Densities (CVPR24)

[2] Masked Spatial Propagation Network for Sparsity-Adaptive Depth Refinement (CPVR24)

[3] Depth Prompting for Sensor-Agnostic Depth Estimation (CVPR24)

We want to emphasize that, over the past decade, numerous depth completion papers have focused on in-domain experiments on NYU and KITTI. However, there has been a recent trend towards addressing out-of-domain challenges in depth completion research [1-9]. This new direction aims to develop models that can handle variations in new sensor configurations [1,2,3,4,5], unseen environmental conditions [6], and training schemes without the need for dense GT [7,8,9], rather than being restricted to specific sensors or environments. This trend is gaining attraction in top-tier conferences (e.g., CVPR [1,2,3,7]) and journals (e.g., TPAMI [5]), highlighting the importance of adaptability and generalization in depth completion models. Our research aligns with this direction and shares similar goals. We kindly note that most of those works do not consider the KITTI benchmark, which is an in-domain experiment with a 64-Line LiDAR sensor. While we agree that top-tier papers should demonstrate a certain level of performance, we also believe that research focusing on generalization and adaptability for arbitrary sensors and environments is valuable and deserves recognition.

In response to the lack of references and discussions on recent SOTA methodologies, during the rebuttal process, we had tried to compare and discuss as many advanced models as possible. Unfortunately, only two of the five suggested papers provide publicly available code, LRRU (ICCV24) and DFU (CVPR24) experiments. Another reviewer (bXoG) also mentioned comparisons with recent models, and we have conducted additional experiments with the VPP4DC (3DV24) and OGNI-DC (ECCV24) papers (see Table.C & Table.I). We will revise the manuscript to include a comprehensive review of the latest high-performance works.

We hope that these clarifications with the updated performance on the online KITTI benchmark address your concerns.

[4] Sparsity Agnostic Depth Completion (WACV23)

[5] G2-MonoDepth: A General Framework of Generalized Depth Inference from Monocular RGB+X Data (TPAMI24)

[6] All-day Depth Completion (Arxiv24)

[7] Test-Time Adpation for Depth Completion (CVPR24)

[8] Monitored Distillation for Positive Congruent Depth Completion (ECCV22)

[9] Unsupervised Depth Completion with Calibrated Backprojection Layers (ICCV21)