ST$^2$360D: Spatial-to-Temporal Consistency for Training-free 360 Monocular Depth Estimation

We sincerely thank you for your positive review and thoughtful questions. "W/Q" numbers the weakness or question, followed by our response.

W1/Q1: About the qualitative results.

Thanks for your valuable advice. We will include additional qualitative results for the compared methods, including error maps, to more clearly illustrate the performance differences between our $\text{ST}^2\text{360D}$ and the baselines.

Q2: About the depth quality at edge regions.

Thanks for the suggestion. To evaluate the sharpness of estimated depth maps at edge regions, we employ the "Boundary F1 score" proposed in Depth Pro [1]. We conduct experiments on the Replica360-2K dataset.

Our method is training-free and achieves accurate boundaries by leveraging temporal consistency and Poisson blending. We compare our method with PanDA, which fine-tunes Depth Anything v2 on synthetic 360 depth datasets (Structured3D[2] and Deep360[3]). Note that our method does not support direct fine-tuning as the implementation of Poisson blending is not differentiable.

Given the significant influence of input resolution on boundary accuracy, we compare our method with PanDA at two input resolutions (training resolution of PanDA is 504x1008). Our method takes perspective frames with 518x518 resolution. The results, shown below, demonstrate that our $\text{ST}^2\text{360D}$ with mean blending outperforms PanDA even when using a higher input resolution of 1008×2016. Employing Poisson Blending can further improve the boundary accuracy of our $\text{ST}^2\text{360D}$. We ascribe it as our training-free pipeline is capable of preserving the high depth precision inherent in video depth estimation models, which have been trained on a diverse set of high-quality perspective depth datasets.

Q3: About the degraded performance with more input views.

Sorry for the confusion. As shown in Fig. 6, even when applying the SUS and LGS strategies, increasing the number of viewpoints—from 160 to 628—can lead to degraded performance. There are three potential reasons to explain it.

[1. Limited Visual Transition.] As the subdivision level increases, the visual difference between adjacent viewpoints becomes minimal. The Video Depth Anything (VDA) model processes 32 frames at a time, and its temporal attention calculation relies on effective motions across frames. When visual transitions are insignificant, it becomes challenging for the model to effectively capture temporal cues.

[2. Oversampling at Polar Regions.] Although the SUS strategy has proven effective in reducing oversampling in polar areas, higher subdivision levels inevitably introduce more viewpoints in those regions. This increased sampling density in polar regions may degrade the overall performance.

[3. Temporal Distance in Long Sequences.] While the VDA model can handle super-long video sequences, using 628 viewpoints results in a significant temporal gap between the first and last views. This temporal distance may make it challenging to construct coherent spatial relationships between temporally distant viewpoints.

[1] Bochkovskii, Aleksei, et al. "Depth pro: Sharp monocular metric depth in less than a second." ICLR 2025.

[2] Zheng, Jia, et al. "Structured3d: A large photo-realistic dataset for structured 3d modeling." European Conference on Computer Vision. Cham: Springer International Publishing, 2020.

[3] Li, Ming, et al. "MODE: Multi-view omnidirectional depth estimation with 360 cameras." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

We sincerely thank you for your positive review and thoughtful questions. "W/Q" numbers the weakness or question, followed by our response.

W1: About the novelty of the SUS module.

Thank you for your valuable suggestion. We acknowledge the significance of prior work in this domain. Regarding spherical uniform sampling (SUS), please note that it is a sampling mechanism used to represent a sphere. For the previous methods, SpherePHD projects a 360 image onto a spherical polyhedron with triangular faces to minimize spherical distortion; Elite360D maps a 360 image to an icosahedron point set to extract global spherical features; PanoFormer uniformly samples spherical neighbors for pixel-wise self-attention calculation, thereby mitigating distortion.

Sharing a different intuition, the novelty of incorporating SUS in our $\text{ST}^2\text{360D}$ stems from the following:

1. The SUS strategy is specifically employed to obtain viewpoints for perspective patch projection, according to the vertices of icosahedral grids at different subdivision levels. This approach reduces redundant sampling in polar regions, which typically contain limited structural information.

2. It constructs the spherical relationships among spherical viewpoints, facilitating the spherical neighbor searching in the proposed LGS strategy.

3. As demonstrated in Fig. 6 of the main paper, SUS significantly outperforms ERP-plane uniform sampling, showing its effectiveness in aligning 360 images with existing video depth estimation models.

W2: About the Comparison methods.

For the mentioned recent 360 depth estimation methods, HRDFuse and Elite360D have not released their model checkpoints. Therefore, we report the performance of PanDA (including its Small, Base, and Large versions) on three high-resolution datasets, as shown in the table below. Following a similar pre-processing strategy of 360MonoDepth, we first downsample input 360 images to a resolution of 504×1008—the training resolution of PanDA. The resulting depth predictions are then upsampled to the original image resolution using bilinear interpolation. Our $\text{ST}^2\text{360D}$ outperforms PanDA in most metrics.

Q1: About depth-controlled generation experiments.

Sorry for the missing explanations. For the depth-controlled generation, we utilize the ControlNet branch of the FLUX model, which takes a depth map to guide the synthesis process. Specifically, we adopt the black-forest-labs/FLUX.1-Depth-dev variant, which supports the use of a referenced depth map as input for conditioning. As illustrated in Fig. 7, the generated images preserve consistent spatial relationships across varying text prompts. Specifically, in the third column of Fig. 7, we input the text prompt "cyberpunk style", while in the fourth column, we input the empty text prompt "".

Q2: About the dead-end cases in the viewpoint searching process.

We take the case of 160 sampled viewpoints as an example. Each viewpoint is associated with 5 or 6 spherical neighbors. At each stage, the scanning procedure traverses specific horizontal slices of the sphere. For a given viewpoint, if all of its spherical neighbors within the current slice have already been visited, the procedure may encounter a dead-end. In such cases, we expand the neighbor set of the current viewpoint to include all unexplored viewpoints, allowing the scanning procedure to resume. In Fig. 1 of the supplementary material, we visualize the actual scanning paths with viewpoint indices. Several dead-end examples can be found in Fig. 1, such as viewpoint transitions 24 $\rightarrow$ 25, 42 $\rightarrow$ 43, and 129 $\rightarrow$ 130. We will clarify the dead-end cases in the revision.

We sincerely thank you for your positive review and thoughtful questions. "W/Q" numbers the weakness or question, followed by our response.

W1: About the illustration of latitude-aware traversing.

Thanks for your suggestion. To make the latitude-aware traversing clear enough, we present it with an algorithm block as follows:

Algorithm 1: Latitude-aware Traversing

1. Input: A set of $N$ Viewpoints $\mathbf{v} = \\{\mathbf{v}\_{1},..., \mathbf{v}\_{N}\\}$

                   $K$ key latitudes $0 < \theta_1 < \dots < \theta_K \le 90^{\circ}$

2. Latitude computation: For each viewpoint $\mathbf{v}_i$, obtain the $\Theta(\mathbf{v}_i)$

3. Slice partitioning: Partition viewpoints into latitude slices based on $K$ key latitudes:

         $\mathbf{v}\_{\text{slice}\_{0}} = \\{\mathbf{v}\_i \mid \mathbf{v}\_i \in \mathbf{v}, |\Theta(\mathbf{v}\_i)| \leq \theta\_{1}\\}$

         $\mathbf{v}\_{\text{slice}\_{k}} = \\{\mathbf{v}\_i \mid \mathbf{v}\_i \in \mathbf{v}, \theta\_{k} < |\Theta(\mathbf{v}\_i)| \leq \theta\_{k+1}\\}$, for $k = 1,\ldots,K-1$

         $\mathbf{v}\_{\text{slice}\_{K}} = \\{\mathbf{v}\_i \mid \mathbf{v}\_i \in \mathbf{v}, \theta\_{K} < |\Theta(\mathbf{v}\_i)| \leq 90^{\circ}\\}$

4. Slice ordering: Arrange the slices in ascending latitude to form the list $\mathbf{v}\_{\text{slices}} \leftarrow [\mathbf{v}\_{\text{slice}\_{0}},\ldots,\mathbf{v}\_{\text{slice}\_{K}}]$

5. Output: The ordered slices $\mathbf{v}_{\text{slices}}$

Based on the latitude slices, we utilize the spherical neighbor viewpoint searching to generate the video frame sequence $\\{\mathbf{F}_1, \dots, \mathbf{F}_N\\}$.

W2/Q2: About the inference time and GPU consumption.

Thanks for your suggestion. We have extended our evaluation to include inference time and GPU memory consumption across additional baseline methods. All experiments are conducted on a single A40 GPU. The input resolution of UniFuse and ACDNet is 512x1024, while the input resolution of PanDA (Small, Base, and Large variants) is 504x1008. We report the results of our $\text{ST}^2\text{360D}$ with 160 input views and perspective resolution 378x378.

Q1: About more views of 360MonoDepth.

As suggested, we conduct an extended comparison with 360MonoDepth using increased numbers of input views. In 360MonoDepth, viewpoints are derived from the triangular faces of an icosahedron. A triangular face can be divided into four sub-faces, thus increasing the input views accordingly. However, the original codebase supports only an unsubdivided icosahedron, and this limitation applies to its input processing, depth estimation, post-processing, and output processing.

To enable additional input views, we incorporate subdivision functions into the input projection and output re-projection processes by adapting code from SpherePHD [1]. Due to the time constraints of the rebuttal period, we omit the depth alignment module in the post-processing for this evaluation.

We test 360MonoDepth under various subdivision levels: level 0 (20 views), level 1 (80 views), and level 2 (320 views). We utilize the Mean blending, and test on the Replica360-2K dataset. All hyperparameters are set according to the recommended configurations. Our results show that increasing the number of input views degrades the overall performance. Since each input view is processed independently using an image-based depth estimation model, simply increasing the number of views does not enrich structural information but instead introduces depth inconsistency across views.

Moreover, in Fig. 6 of the main paper, we have presented the results of our $\text{ST}^2\text{360D}$ in different input views, ranging from 12 views to 628 views. The results demonstrate the effectiveness of our proposed SUS and LGS strategy, especially when increasing input views from 40 to 160.

[1] Lee, Yeonkun, et al. "Spherephd: Applying cnns on a spherical polyhedron representation of 360deg images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

We sincerely thank you for your positive review and thoughtful questions. "W/Q" numbers the weakness or question, followed by our response.

W1/Q1: About the robustness of the LGS strategy in outdoor datasets.

Thank you for your valuable suggestions. Based on additional experiments, we confirm that the proposed LGS strategy demonstrates robustness in outdoor scenarios. Below we describe the details of these additional experiments.

[Outdoor datasets:] We utilize two datasets containing outdoor scenes. The first one is the Deep360 dataset [1], including outdoor driving scenes generated via the CARLA simulator. Its testing set contains 600 samples. The second dataset is derived from the Matterport3D dataset. We employ SegFormer [2] to detect sky regions in each sample of the Matterport3D dataset and collect samples where the detected sky regions constitute more than 10% of the entire ERP images. Subsequently, we manually select 100 samples by further filtering out wrongly detected samples.

[Structural Information Across Latitudes in Outdoor Scenes:] We quantitatively assess the distribution of structural information across latitudes. Specifically, we utilize the Sobel operator to detect edges of 360 ground-truth depth maps, and normalize edge maps to the range [0, 255]. Subsequently, we average all edge maps within each dataset and summarize the edge values within four distinct latitude slices: $[-90^\circ, -45^\circ)$, $[-45^\circ, 0^\circ)$, $[0^\circ, 45^\circ)$ , and $[45^\circ, 90^\circ]$. The summarized edge values are normalized and presented in the following table. The results indicate that equator regions consistently exhibit rich structural information across both outdoor datasets. Furthermore, high-latitude regions can also contain significant structural information, particularly in the Deep360 dataset.

[Ablation studies of LGS strategy in outdoor scenes:] To evaluate the robustness of the proposed LGS strategy, we conduct ablation studies on the two outdoor datasets. The results are presented in the table below. These results demonstrate that the LGS strategy consistently improves performance in outdoor scenes.

[Conclusion:] (1) From the results of structural information, it can be found that equator regions contain rich structural information in most cases. Thus, starting from equator viewpoints is generally suitable. (2) Furthermore, our proposed LGS strategy is still effective in outdoor scenes. Notably, the LGS strategy rearranges perspective views rather than discarding high-latitude views entirely. The high-latitude views can also be aligned with the previous equator views. (3) Finally, a promising direction for future work involves designing an adaptive LGS strategy that automatically identifies the most salient region as the starting point.

W2/Q2: About the randomness of the scanning strategy.

Sorry for the confusion. In the proposed LGS strategy, given (1) the viewpoints determined by the SUS strategy, (2) the starting viewpoint, and (3) key latitudes, the scanning path becomes entirely deterministic. Notably, we have presented the scanning path using viewpoint indices in Fig. 1 of the supplementary material, including different subdivision levels. A viewpoint might encounter the dead-end case, where no unexplored spherical neighbors remain within the current latitude slice. In this case, all remaining unexplored viewpoints in this latitude slice are collected as the neighbor set for the current viewpoint, allowing the scanning process to continue. From this expanded neighbor set, we select the viewpoint with the minimum latitude as the subsequent viewpoint.

Although such dead-end cases could potentially introduce variability, they consistently arise at specific viewpoints, and the selection of the next viewpoints remains uniform across implementations. Moreover, we have verified our experimental results multiple times and across diverse GPU environments to confirm reproducibility and robustness of the scanning process.

Q3: About the efficiency compared to 360MonoDepth.

The primary reason for the improved efficiency of our method compared to 360MonoDepth lies in the avoidance of time-consuming depth map alignment in the post-processing. In 360MonoDepth, depth alignment involves multi-scale operations and incorporates several optimization terms, such as smoothness and scale consistency, which significantly increase time consumption.

In contrast, our pipeline does not require such alignment, as it leverages the temporal consistency inherent in video depth estimation models during inference. Moreover, our SUS and LGS strategies efficiently organize the perspective views into a coherent video sequence.

Furthermore, The Video Depth Anything (VDA) model used in our pipeline can process 32 frames simultaneously within each sequential step. As the number of input views increases from 12 to 160, the number of sequential processing steps increases from 1 to 8, with a 12-frame overlap between adjacent sequences. This design ensures efficiency even with large number of input views.

[1] Li, Ming, et al. "MODE: Multi-view omnidirectional depth estimation with 360 cameras." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[2] Xie, Enze, et al. "SegFormer: Simple and efficient design for semantic segmentation with transformers." Advances in neural information processing systems 34 (2021): 12077-12090.