SC-OmniGS: Self-Calibrating Omnidirectional Gaussian Splatting

W1: Brief captions in Figure 2 and 3.

We tried to elaborate figure details in the captions while the space exceeded the limitation. Considering these two figures are self-explanatory and their corresponding contents are on the same page, we therefore keep captions of Figure 2 and 3 concise.

W2: Lack some geometric information, such as depth visualizations and point cloud reconstructions.

Thank you for your suggestions. We have added depth visualizations to the Appendix of the revised version. Please refer to Figure 8 of the revision.

Q1: About the benchmark dataset.

Yes, the dataset 360Roam and Omniblender are only composed of 360-degree images and commonly used to evaluate omnidirectional radiance field methods performance.

W1: Lack comparisons of camera poses.

Thank you for your suggestion. We have added the evaluation results of the camera pose into the revised version, including Table 7. Our method indeed achieves highest pose estimation accuracy leading to high-fidelity radiance field reconstruction.

W2: Lack comparison with poes-prior-free methods, e.g. COLMAP-free 3D GS, NoPE-NeRF.

Comparisons with calibration methods with no pose prior. Note that either Nope-NeRF or CF-3DGS requires depth prior during camera calibration, but our SC-OmniGS calibrates camera without any depth prior during optimization.* was reported in the paper.

Although some self-calibrating methods emphasize they are pose-prior-free in their papers, they in fact only support object-centric scenes or using sequential data as input.

NoPE-NeRF and COLMAP-free 3DGS heavily rely on depth supervision during the training process which make them sensitive to depth prior obtained from monocular depth estimation models. By contrast, our method only uses coarse point clouds for 3D Gaussians initialization without relying on dense depth supervision. Even though using random point cloud initialization, our methods still have dominant performance, demonstrating our robustness and flexibility.

Moreover, A1 COLMAP-free 3DGS and A3 [Meuleman A et al 2023] should be closely relevant to radiance-field-based SLAM methods which require videos as input and make use of sequential relationships to progressively recover radiance fields. The incoming frames can be roughly initialized by the motion model. However, the benchmark dataset for omnidirectional radiance field evaluation is composed of sparse and discrete frames. Therefore, we did not use them as baselines in the paper.

W3: Lack ablation study of the anisotropy regularizer loss.

Since anisotropy regularizer has become a practical use and is not counted as our contribution, we did not conduct an ablation study to further verify its effectiveness in our paper.

Q1: It would be better to provide the experimental results mentioned in the weaknesses.

Please refer to the responses in W1-3.

Q2: Table 1 lacks results of the proposed methods when the input camera is perturbed while 3D gaussians are initialized randomly or from estimated depth.

Thank you for your thoughtful suggestion. We have reported these results in Table 1 of the revised version.

Q3: Detailed analysis and results of the failure cases on real-world multi-room scenes to better understand the reasons behind this limitation.

As we discussed in the “Limitations” of the paper (Line535-539), all self-calibration methods fail to learn radiance fields without any pose priors in challenging multi-room-level scenes. This is because the tolerance level of the self-calibrating radiance field method is capped. We have analyzed our method robustness against varying levels of camera perturbation in Sec. 5.4. With the noise increasing to some levels, the reconstruction performance drops obviously. Still, our SC-OmniGS consistently outperforms baselines.

When training from scratch without pose prior, the initial camera poses are identical, at the origin of world coordinates. Therefore, in challenging cases, i.e, sparse and discrete views on multi-room-scale scenes, there is no doubt that the noise of initial value has exceeded tolerance level.

Dear Reviewer,

Thank you very much for dedicating your time and effort to reviewing our paper. We are grateful for your constructive feedback, which has significantly enhanced the quality of our work.

If you have any further concerns or suggestions, please do not hesitate to share them with us. We look forward to the opportunity for further discussion and paper refinement. And we hope that our work can make a valuable contribution to the community.

Best regards,

The Authors

Thank you for the precise and insightful comments, please find our responses below:

W1: About title preciseness of self-calibrating.

This point is worth discussion. As defined, calibration is to determine or adjust the accuracy and quality of measurements. Given SfM estimation without perturbation, SC-OmniGS can continue to refine camera models and poses which increases reconstruction performance, as evidenced in Table 2. In our experiments, we have also studied various situations of radiance field self-calibration involving varying levels of initialization noise to verify our method's robustness. Training from scratch is just a special situation where all camera poses are initialized at the origin of world coordinates. In essence, our paper title, "SC-OmniGS: Self-Calibrating Omnidirectional Gaussian Splatting," accurately encapsulates the core content of our research.

Q1 Evaluation of optimized camera poses.

We appreciate your suggestion and have incorporated the evaluation results of optimized camera poses in the revised version. Our method has indeed achieved the highest accuracy in pose estimation, a fundamental aspect crucial for the reconstruction of high-fidelity radiance fields. For details, please refer to Table 7 in the revised paper.

Q2: Showcase of improvements in the intrinsic parameters.

In some self-calibrating radiance field papers, they utilize Colmap results as ground truth to evaluate their optimized intrinsic parameters. However, as we mentioned in the introduction section (i.e., Line 43-46), existing SfM methods rely on an idealized omnidirectional camera model assumption and overlook the adverse effects of omnidirectional camera distortion in real-world scenarios. We are the first to tackle this issue but cannot obtain pseudo ground truth to conduct quantitative evaluation of intrinsic parameters. As a radiance field method, we instead relied on reconstruction quality, i.e. novel view rendering quality to reflect the improvement of camera parameters. We also conducted an ablation study (Table 3) to solely evaluate camera model efficacy in terms of reconstruction quality. We believe the experiment is comprehensive.

Q3: Are the optimizations of focal length and distortion parameters shared across all views?

Yes, we only optimized a single omnidirectional camera model and used it to tackle all views on each scene. We have made it clearer in the implementation details of the revised version (Line 335-336).

W1: Limited technical improvement.
The scientific motivation for combining Self-calibration and omnidirectional GS is limited.

The proposed SC-OmniGS is not an incremental work that simply combines existing solutions. Although the pose optimization along Gaussian splatting process has been studied in some recent GS-based SLAM methods, they only support perspective images without distortion. Their theoretical analysis and implementation of camera pose derivatives in 3D Gaussian Splatting cannot be directly reused to achieve self-calibrating omnidirectional radiance fields. Additionally, joint optimization of intrinsic camera models and GS is still underexplored.

Our SC-OmniGS systematically analyzes and achieves omnidirectional camera pose optimization within the omnidirectional Gaussian splatting procedure. We are the first to tackle the complex distortion pattern contained in omnidirectional cameras via introducing a novel differentiable omnidirectional camera model. Furthermore, we proposed a weighted spherical photometric loss to enhance omnidirectional radiance field reconstruction quality.

We believe our work will attract good attention and make a positive contribution to the omnidirectional vision community.

W2: Gradient derivation is one of key technical parts of the paper but it has been addressed.
The derivation of gradients on spherical camera poses (Eqs. (13--14)) is the key technical part. However it is rather straightforward since it would be naturally extended from perspective cases. Moreover, some SfM softwares supporting the spherical camera already has gradient computing similar to Eqs. (13--14).

The gradient computings in SC-OmniGS and the mentioned softwares (Metashape, OpenMVS) are theoretically different. The mentioned methods optimize camera pose by minimizing 2D-to-3D reprojection residual of corresponding points. The optimization problem is formulated as a factor graph and solved by the Levenberg–Marquardt (LM) algorithm, i.e., the first-order approximation of the error function. By contrast, our optimization objective in SC-OmniGS is to minimize weighted spherical photometric loss between rendering and reference images. The rendering process is differentiable, a key departure from traditional methodologies. The gradients of omnidirectional camera pose are then derived and back-propagated along the GS process.

Q1: Emphasizing technical novelty of the proposed method again.

The primary technical contributions of our work include:

1. Gradient Derivation for Omnidirectional Camera Poses. SC-OmniGS stands as a pioneering effort dedicated to the precise calibration of omnidirectional radiance fields, showcasing cutting-edge performance levels. These advancements can further facilitate applications such as GS-based omnidirectional SLAM.
2. Addressing Complex Distortion Patterns with a Generic Omnidirectional Camera Model. Due to complex distortion patterns inherent in omnidirectional cameras, current 3D omnidirectional vision methods rely on ideal spherical camera model assumption, resulting in suboptimal performance, as we discussed in the main paper Line 43-50. To the best of our knowledge, we are the first to effectively handle this issue by proposing a generic camera model tailored for the 360-degree camera.
3. Enhanced Reconstruction Quality through Weighted Spherical Photometric Loss. To promote spatially consistent optimization and elevate the overall quality of omnidirectional radiance field reconstruction, we introduce a novel weighted spherical photometric loss function.