Graph-Guided Scene Reconstruction from Images with 3D Gaussian Splatting

Thank you for recognizing our ideas, presentation, and experiments.

Q1: Spelling Errors.

A: Thank you for pointing them out; we will carefully make the corrections.

Q2: Details on acquiring initial camera poses.

A: We use only Dust3R’s pairwise pose estimation module without the Global Alignment module to obtain rough initial poses. Each image pair takes approximately 0.01 seconds on the NVIDIA A6000, with 599 pairs from 600 images in the Waymo dataset and 1999 pairs from 2000 images in the Mill 19 dataset. Given that the dataset’s initial poses are less accurate than our results, we compare them against COLMAP benchmarks and our final poses in Waymo, as the table below:

Q3: Symbol for concentric nearest neighbor pairing.

A: Thanks for pointing this out, we will rename S_1, S_2, and S_3. S_1 is related to the nearest neighbor cameras, so we rename it to S_neighbor; S_2 are the cameras with isometric interval, so we rename it to S_concentric; S_3 is designed for connection of camera graph, so we rename it to S_connection.

Q4: Quadrant filter details.

A: The orientation is a 3D vector which is equivalent to the 3-rd column of the extrinsic matrix. This provides convenience for the following cross-product and inner-product.

Q5: Details of adaptive sampling.

A: Degree centrality focuses on the activity level of nodes, while betweenness centrality is concerned with the control a node exerts or its role as a bridge. In our experiments, we found that using only betweenness centrality yielded better results, so we only use betweenness centrality to design node weights.

Q6: Details on how to obtain the initial pose.

A: The Details as below:

1. Time and Hardware: Each pair is estimated in 0.01 seconds using an NVIDIA RTX3090 with 24GB of memory. There is no need for Dust3R’s global alignment, as it relies solely on pairwise pose estimation.
2. Datasets: For the Waymo dataset, we use 599 pairs (600 images) for evaluation; For the Mill 19 dataset, we use 1999 pairs (2000 images) to do evaluation.

Please be clarified that Dust3R is not mandatory. Coarse poses from the datasets can also be used directly in our proposed framework. For initial pose accuracy evaluations, please refer to Q2.

Q7: The minimum pose quality, the error tolerance, and the reasons for comparison with COLMAP.

Q7.1: What quality of poses does the framework require?

A: Our goal is to develop a framework for reconstructing outdoor scenes from uncalibrated images. We use coarse pose prior to speed up the Structure-from-Motion (SfM) process and build a graph-guided 3DGS reconstruction. The Dust3R for obtaining rough poses is optional; any coarse pose is suitable, but we choose it for its rapid and widely recognized capabilities. As long as the initial results with correct camera layout on the xy-plane can be used for successful bundle adjustment (BA), we regard them to meet the minimum quality for initial poses. For initial pose quality quantitative comparison, please see Q2.

Q7.2: Are there measures to handle large pose errors?

A: Our matching strategy requires only coarse information about the camera’s position and orientation, specifically the third and fourth columns of the extrinsic matrix. This provides a higher error tolerance; as long as the initial results do not produce severe camera layout errors on the xy-plane, BA can be completed. If BA proceeds successfully, it will not affect subsequent accuracy estimations. However, how to handle BA failures is beyond the scope of this paper. In the test datasets we used, BA is successfully completed.

Q7.3: Why does it compete with COLMAP?

A: The comparison with COLMAP is due to its wide recognition as an SfM benchmark in the community. The latest SfM methods like VGGSfM [R1], GLOMAP [R2], and ACEZero [R3] all use COLMAP as their primary comparative baseline. We have extensively evaluated these recent methods on the Waymo dataset.

As shown in the table, VGGSfM encounters an out-of-memory (OOM) issue when processing 600 images, ACEZero is fast but fails to converge entirely, and other methods also underperform in pose performance for novel view synthesis compared to our method. Therefore, COLMAP remains the most competitive benchmark for comparison.

Reference

[R1] J. Wang et al., “Visual Geometry Grounded Deep Structure from Motion,” arXiv preprint arXiv:2312.04563, 2023. [Online]. Available: https://arxiv.org/abs/2312.04563

[R2] L. Pan et al., “Global Structure-from-Motion Revisited,” arXiv preprint arXiv:2407.20219, 2024. [Online]. Available: https://arxiv.org/abs/2407.20219

[R3] E. Brachmann et al., “Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer,” arXiv preprint arXiv:2404.14351, 2024. [Online]. Available: https://arxiv.org/abs/2404.14351

[R4] P. Lindenberger et al., “Pixel-Perfect Structure-from-Motion with Featuremetric Refinement,” arXiv preprint arXiv:2108.08291, 2021. [Online]. Available: https://arxiv.org/abs/2108.08291

Thank you for your detailed feedback. We appreciate your recognition of the practicality of our method. To our best knowledge, we are the first to propose a graph-guided 3DGS optimization method for outdoor scenes, which can effectively accelerate and enhance the performance of 3DGS methods such as 3DGS, OctreeGS, ScaffoldGS, 2DGS, etc.

Q1: Implementation Details of Dust3R.

A: The Details as below:

1. Time and Hardware: Each pair is estimated in 0.01 seconds using an NVIDIA RTX3090 with 24GB of memory. There is no need for Dust3R’s global alignment, as it relies solely on pairwise pose estimation.
2. Datasets: For the Waymo dataset, we use 599 pairs (600 images) for evaluation; For Mill 19 dataset, we use 1999 pairs (2000 images) to do evaluation. Please be clarified that Dust3R is not mandatory. Coarse poses from the datasets can also be used directly in our proposed framework.

Q2: Experiment Details for COLMAP.

Q2.1: Which dataset is used to compute the vocab-tree？

A: Dataset Settings: We use the Waymo dataset, which contains 600 images, selecting three views (left, middle, right) for each. We also use the Building dataset from Mill-19, which contains 2000 images. The setting of vocab-tree: The vocab tree is pre-build on the Flickr100k dataset, which is provided officially on https://demuc.de/colmap/ . We use the item "Vocabulary tree with 256K visual words" for evaluation.

Q2.2: How many nearest neighbors are retrieved, and how many pairs are evaluated in total?

A: The setting of CNNP is "r=5, h=20, w=1", which is an empirical setting with a lower failure rate of BA. Under this setting, ~ 18k / $C_{600}^2$ pairs will be selected for 0.6k images; ~ 200k / $C_{2000}^2$ pairs will be selected for 2k images. Explanation of parameters of CNNP:

1. "r=5" represents for every camera c_i, the nearest 5 cameras to c_i will be selected as matching pairs
2. "h=20, w=1": For camera c_i, we select 1 camera from every 21 cameras as matching pairs. This selection is based on distance order to c_i too. As illustrated in the main paper Fig 2. (left)

Q2.3: Computing Resources.

A: On GPU NVIDIA RTX3090 and CPU AMD EPYC 7542, COLMAP’s SfM time is 104 minutes for the Waymo dataset and exceeds 24 hours for the Mill 19 dataset. In contrast, our method’s SfM time is 23 minutes for the Waymo and 206 minutes for the Mill 19.

Q3: Comparison with Other Existing Efficient SfM Methods.

A: Despite numerous efficient SfM methods being proposed, COLMAP remains widely recognized by the community and is considered a crucial benchmark for comparison. As mentioned in GLOMAP [R2], “GLOMAP achieves a similar level of robustness and accuracy as state-of-the-art incremental SfM systems (COLMAP, Schönberger & Frahm) while maintaining the efficiency of global SfM pipelines.” Moreover, the latest SfM methods like VGGSfM [R1], GLOMAP [R2], and ACEZero [R3] also use COLMAP as their primary comparative baseline. We have extensively evaluated these recent methods on the Waymo dataset.

As shown in the table, VGGSfM encounters an out-of-memory (OOM) issue when processing 600 images, ACEZero is fast but fails to converge entirely, and other methods also underperform in pose performance for novel view synthesis compared to our method. Due to the earlier publication dates of OpenMVG, OpenSfM, and TheiaSfM that the reviewer mentioned, along with the time constraints of the rebuttal, we focus our comparison on the latest GLOMAP and other recent methods for this stage. More results will be added in a future version later.

Q6: Provide results on the MipNeRF360 dataset.

A: We consider the comparison with the original 3DGS not as a direct comparison, but as a type of ablation study. Our framework can be seamlessly integrated into methods such as 3DGS, 2DGS, Scaffold, OctreeGS, etc. Due to time constraints, all other results come from those published in OctreeGS. We provide the results with the original 3DGS method on the Mip360 dataset as follows:

Q7: Contributions are dispersed and not fully evaluated with SoTA methods.

A: Thank you for your positive feedback on the improvements highlighted in our paper. Please be clarified that our research is dedicated to reconstructing outdoor 3DGS scenes from unaligned images, targeting the complexities of 3DGS and outdoor settings. As shown in Q5, our graph-guided optimization module has a significant improvement over various SOTA methods. To address your concerns, we conducted additional experiments and will include these comparisons in future versions.

Q8: What does "w/o structure estimation" mean in Table 4.

A: This means we only use Waymo’s original poses, not the estimated pose by our structure estimation module, for 3DGS reconstruction.

Q9: Which datasets are used for results in Table 4, Table 5, and Table 7.

A: Tables 4, 5, and 7 all use the Waymo datasets for evaluation, with each segment containing 600 images.

Q10: Dust3r Details.

A: We solely use Dust3R’s pairwise pose estimation.

Q11: Definition of S_3^i in Eqn 1.

A: In the CNNP part, we traverse all of the cameras (from 1 to N, supposing current camera as c_i) and calculate which camera (c_j) should form a matching pair with c_i. S_3 means for c_i, we always add c_(i-1) in the traversal loop to form a matching pair. This can guarantee the camera graph must be a connected graph as illustrated in Remark.

Q12: Definition of a Large-Scale Scene.

A: To avoid confusion, we will remove it.

Q13: Clarification of Formula 8.

A: Thank you for pointing that out; we will make the corrections.

Reference

[R1] J. Wang et al., “Visual Geometry Grounded Deep Structure from Motion,” arXiv preprint arXiv:2312.04563, 2023. [Online]. Available: https://arxiv.org/abs/2312.04563

[R2] L. Pan et al., “Global Structure-from-Motion Revisited,” arXiv preprint arXiv:2407.20219, 2024. [Online]. Available: https://arxiv.org/abs/2407.20219

[R3] E. Brachmann et al., “Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer,” arXiv preprint arXiv:2404.14351, 2024. [Online]. Available: https://arxiv.org/abs/2404.14351

[R4] P. Lindenberger et al., “Pixel-Perfect Structure-from-Motion with Featuremetric Refinement,” arXiv preprint arXiv:2108.08291, 2021. [Online]. Available: https://arxiv.org/abs/2108.08291

Thank you for recognizing the clarity of our paper and the effectiveness of our methods.

Q1: A detailed explanation of Equation (1).

A: Thanks for your suggestion, we will add more detailed explanation in the revision. In Equation 1, we traverse all of the cameras (from 1 to N, supposing the current camera as c_i) and calculate which camera c_j should form a matching pair with c_i. We have the following steps:

1. For c_i, we sort other cameras based on their distance to c_i
2. We add c_i 's nearest r cameras, as S_1
3. We add w cameras from every h+w camera based on the same sorted order, as S_2.
4. We add c_(i-1) in the main loop to match c_i, as S_3

Q2: Compare the accuracy of the initial values.

A: We provide initial pose accuracy evaluations in Waymo in the table below:

Our previous experiments indicate that the true poses provided by the dataset perform poorly. Therefore, we compare both the initial poses and the final results optimized by our pose estimation module against the COLMAP results as a benchmark. This comparison validated our method’s ability to significantly enhance accuracy.

Q3: Tolerance of Spatial prior-based structure and graph construction.

A: Sure, inaccurate initial results do lead to poorly constructed graph. However, our matching strategy requires only coarse information about the camera’s position and orientation, specifically the third and fourth columns of the extrinsic matrix. This offers a higher tolerance for errors; as long as the initial results with correct camera layout on the xy-plane can be used for successful bundle adjustment (BA), we regard them to meet the minimum quality for initial poses.

If BA succeeds, it means the camera graph has already passed the BA test, and the graph topology has no big problem. However, how to handle BA failures is beyond the scope of this paper.

Q4: Comparison of FPS with 3DGS.

A: In our experiments, FPS refers solely to the rendering speed after training completion, with 3DGS and our results remaining within a reasonable fluctuation range. The pose estimation method you mentioned affects the overall training time, which should be compared with the 3DGS + Colmap pipeline. As shown in the table below, we significantly lead in both efficiency and quality in Waymo:

Dear Reviewer,

Thank you for providing valuable feedback. We are pleased to know that our responses have addressed most of your concerns and appreciate your recognition of our pipeline improvements.

We would like to take this opportunity to further clarify and emphasize the technical contributions and novelty of our work:

1. We propose a novel spatial prior-based structure estimation method, where the proposed Quadrant Filter rapidly computes the 6-bit encoded relative position and orientation. It significantly improves estimation speed and robustness in outdoor settings.

2. We are the first to involve graph information into the 3DGS optimization process, such that the overfitting caused by limited constraints in outdoor scenes is mitigated. This strategy significantly enhances reconstruction quality and is applicable to various 3DGS-based methods.

We present extensive experiments and convincing mathematical proof. Our method can also effectively process uncalibrated images from the internet to reconstruct outdoor scenes.

Thank you for your attention and feedback. We hope this clarifies concerns regarding our method’s novelty and highlights our contributions to the field. We appreciate your reconsideration of our work based on these points.

We thank you for your valuable comments and feedback. We are pleased to learn that you recognize the contributions of our proposed method in enhancing the quality and efficiency of GS, and we appreciate your affirmation of our new experimental results.

We greatly appreciate your valuable suggestion to focus on the framework and premises of our paper. We understand the importance of clearly presenting the motivation and overarching approach of our method to our readers. To this end, we will refine our paper in the following areas:

Q1: Detailed description of the experimental setup.

We will incorporate the following experimental details into our paper. Experiments were conducted using an NVIDIA RTX 3090 GPU and an AMD EPYC 7542 CPU. Dust3R was utilized for initial pose estimation across both the Waymo, Kitti, and Mill 19 datasets, achieving pose estimation within 0.01 seconds per pair. The Waymo includes 600 images from three viewpoints (left, center, right), covering 599 image pairs. The Kitti includes 100 images with 99 image pairs. The Mill 19 comprises 2000 images with 1999 pairs included.

For the CNNP configuration, settings of “r=5, h=20, w=1” were employed. “r=5” designates the five nearest cameras as matching candidates for each camera c_i . The “h=20, w=1” configuration means that one camera is chosen as a match for c_i out of every 20 cameras based on proximity. To optimize multi-view consistency and balance the coefficients, we set the coefficient \lambda to 0.07, which was empirically found to be optimal in our experiments. Additionally, we adopted a graph-guided optimization setup, where the sampling probability is determined by the weights of the graph nodes. We set the minimum sampling probability at 0.5 to ensure that nodes with lower weights are not overlooked.

For comparative experiments with Colmap, the Colmap setup included a vocabulary tree containing 256K visual words, pre-built using the Flickr100k dataset (available on the COLMAP project page).

Q2: Enhanced Presentation of Empirical Evidence.

We have added more experimental results, including comparisons with other SFM baseline methods and the outcomes of applying our method to various 3DGS methods. These data further confirm the effectiveness and practicality of our approach, which will be included in a future version of the paper to enhance its persuasiveness.

Q3: Discussion of limitations and challenges.

As you suggested, we will add a section discussing the limitations and challenges of our method.

Our research focuses on 3DGS reconstruction of outdoor, unbounded scenes, primarily facing two challenges: 1) Accurate pose estimation outdoors is difficult due to the unpredictability and complexity of outdoor environments; 2）Outdoor scenes generally have sparser camera coverage and less overlap between images compared to indoor settings, resulting in insufficient constraints during training. To address these challenges, we introduced two key modules: Spatial Prior-Based Structure Estimation and a graph-guided Gaussian optimization strategy. These modules are designed to efficiently and accurately complete the reconstruction of outdoor scenes.

However, our method shows limited improvement in 3DGS reconstruction of objects. This is primarily because datasets in this category usually have precise ground truth (GT) poses, and the camera setups often rotate 360 degrees around the object, providing sufficient image overlap and constraints to aid convergence. This results in nearly equal importance weights for graph nodes, which diminishes the impact of our optimization.

Moreover, our approach relies on the accuracy of rough spatial priors. If the initial camera pose distribution in the xy plane is inaccurate, it may lead to Bundle Adjustment (BA) failure. In such cases, our method might not effectively handle pose estimation errors.

We value the insights your feedback has provided and are committed to continually refining our work. By conducting additional experiments and evaluations, we believe we can demonstrate that our contributions are substantial and highly relevant to practical 3DGS reconstruction tasks. We look forward to further discussions and appreciate the opportunity to strengthen our submission.

Thank you for your positive feedback and constructive comments. We appreciate your recognition of our clarifications and proposed ideas. In response to your suggestions, we will enhance the clarity and precision of our method descriptions, provide detailed methodological and experimental information for reproducibility, and reinforce the presentation of our results to emphasize the efficacy and practical utility of our approach. We value your guidance and are committed to presenting an improved version.