RPG360: Robust 360 Depth Estimation with Perspective Foundation Models and Graph Optimization

We appreciate your time and careful evaluation of our work. We are glad that the effectiveness of our training-free approach and the core contribution of our graph-based optimization framework with per-face scale alignment were well recognized. We also appreciate the recognition of our method’s zero-shot generalization and its utility in downstream applications.

W1Q1) Model Specification

In Table 3 in the main paper, we explicitly indicate the backbone or pretrained model used for each method, including Metric3D V2 and Omnidata V2 for RPG360. For the main results in Tables 1 and 2, RPG360 (Ours) consistently uses Metric3D V2 as the foundation model. We will revise the manuscript to explicitly state the foundation model used in Tables 1 and 2 to ensure clarity and reproducibility.

W2Q3) Analysis of Optimization Strategies

To the best of our knowledge, there are no existing methods that apply graph-based optimization specifically for 360 depth estimation. Among prior works, 360MonoDepth [47] and Peng and Zhang [42] incorporate optimization techniques, but their approaches are limited to blending-based depth alignment without explicit consideration of structural consistency or global 3D geometry (Lines 47–51). In contrast, our method introduces a graph optimization framework that leverages both depth and surface normals to enforce local planarity and global structural coherence, resulting in more accurate 3D reconstruction. Moreover, we analyze the effectiveness of our proposed optimization strategy through an ablation study presented in Table 5.

W3) Reproducibility Details for Depth Anywhere [60]

For models that we reimplemented, we followed the original codebase and training configurations as closely as possible to ensure fair comparison. Specifically, for Depth Anywhere, we used its best-performing backbone BiFuse++ (Line 205), and followed the original settings: training for 200 epochs with a batch size of 4, a learning rate of 3e-4, and the Adam optimizer. We selected the best-performing checkpoint based on validation performance for testing. We will include these training details in the supplementary material to enhance clarity and reproducibility.

W4) Lack of Surface Normal Ablation

In our proposed optimization, surface normals are an essential component and not an optional module. Removing normals would effectively mean disabling the core of our method. The performance in this setting, without applying our optimization framework, is reported in Table 5, fourth row (Chamfer = 0.380). Furthermore, while surface normals can be predicted using a separate network, they can also be computed directly from the predicted depth. Thus, incorporating normals does not introduce additional inference overhead, and we do not consider their use a burden to the overall method.

W5) Space Allocation

Our primary contribution lies in proposing a robust 360 depth estimation method using a perspective foundation model. Given this, we believe it is important to demonstrate not only why robustness matters, but also how the proposed method can be meaningfully applied to various 360 vision tasks. These downstream tasks help validate the generalization capability of our method and provide empirical insight into its practical effectiveness. Therefore, we allocated a portion of the main paper to downstream applications, which we believe contribute to a more comprehensive understanding of the methodology.

W6Q3) Limited Comparison

As noted in our response to W2, our work is the first to explicitly incorporate structural correctness and local planar approximation into a 360 depth estimation framework. Instead, we compared our approach with prior optimization-based blending algorithms such as [42, 47]. We believe this comparison supports the validity and advantages of our graph-based optimization approach.

To the best of our knowledge, most planar-based 3D reconstruction techniques fall under the category of multi-view stereo (MVS), which operates under the assumption of overlapping views. In contrast, our graph optimization method is designed to operate on adjacent but non-overlapping cubemap faces in a single 360 image. Therefore, these MVS-based approaches are not directly applicable to our problem setting.

If the reviewer was referring to other specific planar-based techniques beyond MVS, we would be happy to consider them and include a discussion in the supplementary material.

W7Q5) Image Degradation

Our primary focus in this work is to explore how robust 360 depth estimation can be achieved in the absence of large-scale 360 training datasets, across both indoor and outdoor scenes (Line 60). While we agree that evaluating robustness under image degradation such as low-light, noise or motion blur would provide additional insights, acquiring 360 datasets with controlled degradations is challenging.

Also, to ensure fair comparison, we conducted experiments using standard benchmark datasets in 360 depth estimation: Matterport3D [5], Stanford2D3D [4], and recently released 360Loc [26]. Although they may not frequently contain the specific types of degradations suggested, their diversity in scene types allows us to test generalization across varied conditions. Furthermore, because our method builds upon a foundation model trained on large-scale perspective data with strong generalization capabilities, we expect it to remain relatively robust to common image degradations, especially compared to models trained solely on limited 360 data.

W8Q2) Table Formatting

Thank you for the helpful suggestion. Due to differences in caption length and table layout, aligning and visually separating Tables 4 and 5 precisely at the center was challenging in the current submission. We agree that improving visual clarity is important, and we will reformat the tables to enhance readability.

Q4) Inference Time

The time required for depth estimation by the foundation model depends entirely on the specific model used. The subsequent graph optimization stage introduces additional computation, and its runtime varies depending on the number of optimization iterations. We have included a runtime versus performance trade-off Table in our response to Reviewer s1NL’s Q1. With the parameters used in our main experiments, the proposed optimization takes approximately 4.6 seconds per image based on our Python implementation. If real-time performance is desired, further speed-ups can be achieved by reducing the number of iterations.

Q6) Thin Structures and Fine Details

Learning-based methods often preserve edge-level details better because they directly regress depth from image features, although the resulting 3D points may lack geometric accuracy. In contrast, our optimization-based approach focuses on enforcing geometric consistency, but may underperform in preserving very thin structures due to the smoothing nature of graph optimization. This limitation could potentially be mitigated by incorporating additional priors, such as semantic segmentation, to selectively break graph connections near object boundaries. However, this would increase computational complexity and was considered beyond the scope of this work.

Q7) Visualization of Limitation

In some cases, such as thin structures like chair legs that occupy only a small number of pixels, our optimization may smooth out the depth and reduce detail. To help clarify this limitation, we will add qualitative examples of such failure cases in the supplementary material.

We are sincerely grateful for your thoughtful and detailed feedback. We are glad that the proposed optimization method leveraging a perspective foundation model was found to be useful and relevant for 360 depth estimation. We plan to release the code upon acceptance to further support the community.

W1Q1) Main Purpose and Runtime

Our primary goal is not real-time deployment. Instead, we aim to develop a robust 360 depth estimation that can serve offline mapping itself, as well as downstream applications such as visual localization (e.g., SfM), as demonstrated in the main paper. Due to the lack of large-scale training datasets for 360 images, it is particularly challenging to build a generalized model directly for this modality. To address this, we leverage an existing perspective-view foundation model and introduce our proposed optimization technique to enable robust zero-shot 360 depth estimation. The downstream tasks are presented to illustrate the importance and utility of such robustness: even though the model is not designed for real-time use, a robust 360 depth estimation can still benefit many 360 vision tasks, such as feature matching and SfM.

W2Q2) Response to Unfair Comparison Concern

While RPG360 shows slightly lower Chamfer Distance scores than Elite360D on Stanford2D3D and 360Loc, it actually outperforms Elite360D on other metrics such as F-Score and IoU. Moreover, although RPG360 leverages a foundation model pretrained on diverse perspective view datasets, it is not trained on any 360 datasets, unlike Elite360D. We acknowledge that training Elite360D on a larger and more diverse corpus (excluding Matterport) could potentially improve its zero-shot performance. However, such training assumes access to large-scale 360 training datasets, which is currently unavailable due to the scarcity of 360 ground truth depth data. Our goal, instead, is to demonstrate a training-free approach that enables robust 360 depth estimation even under such data-scarce conditions. We believe this setup highlights the practical relevance of RPG360 as a viable approach when 360 training data is unavailable or limited.

W3Q3) Performance on Outdoor Dataset

Our method is based on local plane approximation at the patch level, which allows it to handle various surface geometries, not only flat structures such as walls but also curved objects, by approximating local tangential planes. As a result, it is applicable to both indoor and outdoor scenes, including autonomous driving environments. Most prior works on 360 depth estimation adopt indoor datasets such as Matterport3D or Stanford2D3D as standard benchmarks. For a fair comparison with existing methods, we followed this convention.

We agree with the reviewer that performance in outdoor environments is important. To address this, we additionally evaluated our method on 360Loc [26] (Line 61, Figure 4), a recently released dataset that includes outdoor scenes. Regarding Dur360BEV, we appreciate the reviewer’s suggestion. As this dataset was introduced at ICRA 2025, which took place after our full paper submission, we were not aware of it at the time. Nonetheless, we recognize its significance as the first large-scale autonomous driving dataset featuring spherical imagery and plan to include relevant experiments using Dur360BEV in the final version of the paper.

W4Q4) Ablation Study on Iteration and Surface Normal

We have added a Table summarizing the relationship between the number of iterations and performance in our response to Reviewer s1NL's Q1. Our results show that performing a sufficient number of iterations at lower resolution, followed by a small number of refinement steps at higher resolution, provides the best trade-off between accuracy and runtime. While increasing the number of iterations can further improve performance (type: double), we observe that the gains begin to saturate, making the additional runtime less cost-effective.

In our method, surface normals are an essential component for enforcing the local plane approximation and are therefore not an optional module that can be ablated. Moreover, surface normals can be either estimated by a separate network or directly computed from the predicted depth, meaning that their use does not require additional supervision or compromise the training-free nature of our approach. Therefore, we believe that incorporating normals does not pose a limitation in our framework.

W5Q5) Reimplementation of 360MonoDepth

MiDaS V2 and V3, the original backbones used in 360MonoDepth, predict inverse depth. However, directly converting inverse depth to standard depth is not straightforward. Simply inverting the values does not yield an accurate depth map, as it ignores important factors such as focal length, leading to structural distortions in 3D space. Inverse depth represents only a relative inverse distance; thus, naively inverting it does not recover true relative depth or distance. As a result, the reconstructed 3D geometry (e.g., point clouds) becomes unreliable, and surface normals computed from such depth maps are also inaccurate.

Since our method relies on the structural integrity of both depth and normals through local plane approximation, using MiDaS V2 or V3 would not result in a meaningful or fair comparison. Therefore, we chose to reimplement 360MonoDepth using Metric3D V2, which directly predicts standard depth and preserves structural consistency. While Metric3D V2 may not be optimal, we believe it provides a more appropriate and fairer basis for comparison with our method.

W6Q6) Use of Pseudo Ground Truth from RPG360 for Training Elite360D

We acknowledge that using RPG360 to generate pseudo-ground truth and further train models like Elite360D would be valuable. However, such an approach is inherently limited by the quality of the generated pseudo-ground truth. In other words, the maximum achievable performance is bounded by the accuracy of the pseudo labels themselves.

In contrast to Depth Anywhere [60], which rely on pseudo-label training from perspective foundation models, our goal is to demonstrate that a perspective foundation model itself can directly perform robust 360 depth estimation in a zero-shot manner when combined with our proposed optimization technique. We believe this highlights a distinct and meaningful contribution that goes beyond merely serving as a pseudo-label generator.

W-minor)

To clarify, the sentence means that we included the performance of all relevant methods in our tables. In 360 depth estimation research, the same methods are often reported with slightly different performance values across different papers, which can lead to confusion. To ensure consistency and fairness, we followed the values reported in the most recent and comprehensive benchmark papers [1] and [18] when available. For methods not covered in [1] or [18], we referred to their original papers to obtain the reported performance. Our intention was to avoid inconsistencies in reported metrics, not to imply that any prior works were excluded from our comparison. We will revise the wording in the final version to make this point clearer and will include the full set of reference values in the supplementary material for transparency.

We thank the reviewer for their time and careful review. We appreciate the recognition of the technical soundness of our graph optimization for scale alignment, as well as the thoroughness of our evaluations across models (Table 5), partitioning strategies (Table 3), and datasets.

Q1. Running time

The inference time of our method depends on the input image resolution and the number of optimization iterations. For example, with an image size of $1024 \times 512$ and multi-scale iterations (Line 209) set to $(300, 150, 30)$, the total inference time is approximately 4.6 seconds. We summarize the relationship between the number of iterations and performance in the Table below.

The total number of iterations is $300 + 150 + 30 = 480$. To analyze the impact of iteration distribution, we define several configurations with the same total iterations: allocating more iterations to lower resolutions (coarse), allocating more to higher resolutions (fine), and distributing iterations evenly across all scales (equal). We also include configurations with double and half the number of iterations for comparison.

We tested RPG360 with Metric3D v2 on the Matterport3D dataset. Due to rebuttal time constraints, we sampled 100 images from Matterport3D to show performance trends. We will include full evaluation metrics in the supplementary material of the final version.

Q2. Visualization of Scale Map

We originally believed the ablation in Table 5 and Figure 6 sufficiently demonstrated the scale map’s impact since it was simply used as an intermediate representation of scale parameters for optimization. We will add the estimated scale map to help readers better understand the effect of the global scaling.

Q3. Concerns about Interpolation

When merging the cubemap predictions back into the ERP space, we use nearest-neighbor interpolation for both depth and normal maps, which is a standard practice. While this method may not fully address interpolation deviations caused by projection distortions, the subsequent joint graph optimization helps compensate for such issues. In our experiments, this interpolation strategy did not result in noticeable performance degradation.

We sincerely appreciate your thoughtful and constructive feedback. We are glad that our method was found to be solid and effective for 360 depth estimation without retraining, and we appreciate the recognition of our graph optimization with per-face scale alignment as the core contribution.

Q1. Time and Memory Usage

The Table showing inference time with respect to iteration count is included in our response to Reviewer s1NL’s Q1. However, allocating more iterations to high-resolution scales does not always yield better performance, as seen in the fine configuration. Our default setting (original), which follows a coarse-to-fine strategy (more iterations at lower scales, then refinement at higher scales), achieves the best balance between performance and efficiency. Depending on the application, one may reduce the total number of iterations (half) for faster inference with reasonable performance, or slightly improve performance by increasing iterations (double), though performance gains begin to saturate.

Regarding memory usage, our method requires approximately 1680 MiB on an RTX A5000 GPU with an 8-core CPU when processing a $1024 \times 512$ panorama.

Q2. The role of $\lambda_c$

The overall optimization process can be interpreted as performing affine transformation of the form $D^* = \lambda D + \tau ,$ where $D$ is the initial input depth and $D^*$ is the final consistent depth (Line 141). In this formulation, the per-face scale parameter $\lambda_c$ in Eq. 8 serves as the global scaling factor $\lambda$, which aligns the depth across cubemap faces. In contrast, the local smoothness constraint Eq. 6 effectively corrects the pixel-wise offsets, acting to estimate the bias term $\tau$.

As a result, rather than conflicting, these two components work in a complementary fashion. $\lambda_c$ enforces global consistency across faces, while the local constraints refine local geometric coherence. Furthermore, Eq. 6 serves as the main loss function, whereas Eq. 8 acts as a regularization term that prevents the optimization from diverging too far from the input or collapsing to trivial solutions.

Q3. Hyperparameters

Following Rossi et al. [50], we set $\sigma_{int} = 0.07$ and $\sigma_{spa} = 3.0$.

We conducted a grid search to empirically determine the hyperparameters $\eta_p = 50$, $\eta_d = 0.5$, and $\eta_n = 10$. The table below shows the performance with different hyperparameter values. While there are some variations, the overall performance distribution remains relatively stable.

Here, $\eta_p$ controls the weight of the local planar approximation, while $\eta_d$ and $\eta_n$ constrain the optimized depth from deviating too far from the original input. Among these, we observed that the optimization process is generally robust to $\eta_p$ and $\eta_n$, but slightly more sensitive to $\eta_d$.

If $\eta_d$ is set too high, the optimization resists deviation from the original input, resulting in almost no difference between the input depth and the optimized depth. Consequently, the discrepancies between cubemap faces persist.

Conversely, if $\eta_d$ is too small, the influence of the per-face parameters diminishes, weakening the constraint from the original input. This causes over-smoothing across cubemap faces and structural collapse near seams (as illustrated in Fig. 6, "without $\lambda$").

Ultimately, we selected the hyperparameter values reported in the main paper because we believe that qualitative results are as important as numerical performance in 3D vision tasks, and the chosen values produced strong visual results. To ensure fair and consistent evaluation, we used the same set of parameters across all datasets and found that these values led to stable performance. While we cannot include figures in the rebuttal, we plan to add additional visualizations in the supplementary material, similar to Figure 6, to illustrate the effects of different hyperparameter settings.

Thank you for your detailed rebuttal. I still have one more question and hope to get your answer.

As far as I know, some current depth estimation models will produce relatively large distortions in the depth estimation of the upper and lower surfaces (i.e., the ceiling and the floor) when dealing with panoramic depth estimation. This is specifically manifested as the depth of the ground or the sky being curved. Does the method you proposed have any optimizations in this regard, or can it solve some of the problems in this area? I haven't used Metric3Dv2, but I have shown distortion in this regard when using Depth Anything v2 and Omnidata v2. And could you please explain this aspect?