Deep Gaussian from Motion: Exploring 3D Geometric Foundation Models for Gaussian Splatting

Thank you for your detailed and thoughtful feedback. We greatly value your recognition of the strengths of our work, including its writing quality, novel point-to-ray consistency loss, and promising experimental results. Below, we address the concerns you raised and provide clarifications based on additional analysis and updated results.

1. Why COLMAP/GLOMAP shouldn't be used?

We would like to clarify that our goal is not to replace COLMAP/GLOMAP pipelines entirely. Instead, our motivation is to leverage and fine-tune existing 3D foundation models for high-fidelity novel view synthesis in real-world applications. While COLMAP and GLOMAP are robust and fast, they are not sufficiently accurate for certain rendering tasks, particularly where high-quality novel view synthesis is crucial.

Our method is designed to bridge this gap by enabling 3D foundation models to achieve:

(1) Comparable Pose Accuracy: Using our progressive registration and point-to-ray consistency loss, 3D foundation models can reach pose accuracy levels close to those of COLMAP/GLOMAP. (2) Improved Rendering Quality: Our framework enhances the rendering fidelity of 3D foundation models, making them suitable for applications where visual quality is essential.

By combining the robustness and speed of existing 3D foundation models with the accuracy and rendering quality demonstrated in COLMAP/GLOMAP-based pipelines, our approach significantly improves their versatility for real-world applications. This hybrid approach focuses on optimizing foundational systems to work efficiently without discarding their inherent strengths.

2. Lack of Evaluations on Different 3R Backbones and Evaluation on DL3DV dataset

We provided the results of different 3R backbones (DUSt3R, MASt3R, Splatt3R, MV-DUSt3R+, Flare, etc.) and our method on the DL3DV dataset to Reviewer JX9E. For convenience, we copy and paste the results and analysis here for your reference.

We provide a more thorough comparison to SOTA methods on sparse view settings (2-views and 8-views). We conducted the experiments on the RealEstate10K dataset and the DL3DV dataset. For these datasets, we follow the protocol in Flare and NopoSplat, which samples 2/8 views for the testing video sequence as input. Note that Splatt3R and CopoNeRF only support inputs with 2-views, we did not provide the results on the 8-view setting.

• Comparison on two-view settings | Method | Dataset | PSNR ↑ | SSIM ↑ | LPIPS ↓ | |------------------ |----------------|------------|-----------|-----------| | DUSt3R | DL3DV10k | 13.47 | 0.390 | 0.590 | | MASt3R | DL3DV10k | 13.88 | 0.405 | 0.513 | | Splatt3R | DL3DV10k | 15.23 | 0.432 | 0.491 | | CoPoNeRF | DL3DV10k | 16.06 | 0.472 | 0.474 | | Flare | DL3DV10k | 23.04 | 0.725 | 0.182 | | NoPo-Splat | DL3DV10k | 24.19 | 0.745 | 0.173 | | MV-DUSt3R+ | DL3DV10k | 19.95 | 0.698 | 0.195 | | DeepGfM | DL3DV10k | 20.33 | 0.702 | 0.190 |

• Comparison on eight-view settings | Method | Dataset | PSNR ↑ | SSIM ↑ | LPIPS ↓ | |------------------ |----------------|------------|-----------|-----------| | DUSt3R | DL3DV10k | 17.37 | 0.601 | 0.390 | | MASt3R | DL3DV10k | 18.41 | 0.675 | 0.427 | | Splatt3R | DL3DV10k | - | - | - | | CoPoNeRF | DL3DV10k | - | - | - | | Flare | DL3DV10k | 23.33 | 0.746 | 0.237 | | NoPo-Splat | DL3DV10k | 22.77 | 0.722 | 0.240 | | MV-DUSt3R+ | DL3DV10k | 21.33 | 0.707 | 0.188 | | DeepGfM | DL3DV10k | 22.90 | 0.728 | 0.176 |

• Comparison with AnySplat

We would like to emphasize that the AnySplat paper was published on arXiv after the paper submission deadline (29 May 2025) for the conference. As AnySplat was released later than our submission, it is not reasonable to expect a paper to compare its methods to those published without peer review and after the submission deadline.

Importantly, scientific evaluations aim to compare methodologies that are accessible at the time of research development and submission. While AnySplat introduces interesting advancements, it was not publicly available during the preparation of our work, and therefore, could not be included in our study or baseline comparisons.

That said, we acknowledge the relevance of AnySplat for future explorations, and we are excited about the possibility of extending our work to compare and integrate complementary ideas introduced by such methods. The addition of results compared to AnySplat would further strengthen the impact of our proposed framework, but this remains work for future research efforts.

• Experiments summarization on sparse-view settings

(1) Comparison on Two-View Settings Though DeepGfM is not specifically designed for sparse-view settings, it demonstrates competitive performance, surpassing several sparse-view methods.

• DL3DV10k Dataset: DeepGfM achieves a PSNR of 20.33, significantly outperforming DUSt3R, MASt3R, and Splatt3R. SSIM (0.702) and LPIPS (0.190) are also competitive against MV-DUSt3R+, while slightly behind Flare and NoPo-Splat, which are optimized for sparse-view tasks.

• RealEstate10K Dataset: DeepGfM records PSNR of 21.715, ranking higher than DUSt3R, MASt3R, and Splatt3R, while trailing Flare and NoPo-Splat. SSIM (0.753) is higher than MV-DUSt3R+ and comparable to NoPo-Splat. LPIPS (0.229) is better than both DUSt3R and MASt3R, showing robustness in perceptual quality.

(2) Comparison on Eight-View Settings

DeepGfM shows strong capabilities in handling sparse-view tasks even with more input views.

• DL3DV10k Dataset: With a PSNR of 22.90, DeepGfM outperforms MV-DUSt3R+ while approaching Flare and NoPo-Splat. SSIM (0.728) and LPIPS (0.176) are the best among all methods, highlighting DeepGfM’s ability to preserve structural and perceptual quality even with sparse-view conditions.

• RealEstate10K Dataset: DeepGfM achieves a PSNR of 23.370, outperforming MV-DUSt3R+ and NoPo-Splat, and remaining competitive with Flare (23.813). SSIM (0.815) and LPIPS (0.190) are also highly competitive, showing improvements over MV-DUSt3R+ and comparable values with Flare.

• Highlight: It is important to note that these models are optimized for sparse-view settings, whereas DeepGfM addresses dense-view workflows with hundreds of images. These results underline that, despite being designed for dense-view settings, DeepGfM performs competitively in sparse-view tasks, demonstrating robust generalization and adaptability across both two-view and eight-view configurations. This flexibility is beneficial for tasks requiring high-quality synthesis in diverse input-view scenarios.

3. Why Spann3R rather than CUT3R?

We fully agree that CUT3R is a strong alternative to Spann3R and a more competitive backbone. However, it is important to highlight that the CUT3R code was publicly released only two months before our paper submission deadline. Due to this timeline constraint, we were unable to integrate CUT3R into our codebase and perform further validation experiments within the limited preparation period.

That said, CUT3R offers several advantages as a backbone:

• Training Time Reduction: CUT3R is designed to converge faster, reducing the overall training time compared to Spann3R.
• Handling Sparse Inputs: CUT3R performs better in settings with fewer input images due to its efficient coordinate prediction capabilities.

We are confident that substituting Spann3R with CUT3R as the backbone would yield even better results, especially in reducing training time and improving performance with a lower number of input images. This strengthens the robustness and generalizability of our framework.

Even though we did not have sufficient time to integrate CUT3R during this study, we consider future explorations with CUT3R to be an exciting direction that could further validate the flexibility and scalability of our method.

We sincerely thank you for pointing out this potential improvement and for your constructive feedback overall. We firmly believe our approach remains strong irrespective of the backbone used, and CUT3R would likely offer additional benefits that augment the results presented in this study.

The authors provide a comprehensive rebuttal and reply to most of my questions, which is very much appreciated.

I also appreciate the responsiveness of the authors and verify major weakness 4) and minor weakness 1) 2). I will update my rating after the discussion.

There seems to be a slight misunderstanding regarding Major Weakness 2). My point was to suggest evaluating DeepGfM with alternative *3R backbones in place of Spann3r to better assess the method’s generalizability.

Some additional suggestions: In the title and early parts of this paper, I initially thought this was a feed-forward 3DGS reconstruction method. This could be somewhat misleading. I encourage that the authors clarify in the final version, particularly in the title, abstract and introduction that the proposed method is based on per-scene optimization, to avoid unnecessary confusion for readers.

Thank you for your thorough review and valuable insights. We address each weakness raised point-by-point below:

1. Reusing Existing Components (Spann3R, PnP, Gaussian Rasterizer)

We acknowledge that some components of our approach, such as Spann3R and PnP, are not entirely novel. However, the novelty of our method lies in the unique combination of these components within a unified pipeline and its ability to finetune a feed-forward 3D foundation model for high-fidelity, pose-free novel view synthesis—a capability that prior methods have not achieved.

Specifically, our contributions include:

• Integration of Spann3R with Scene-specific Fine-tuning: While Spann3R itself is pre-trained, our method enables it to predict Gaussian-based geometry and photometric texture for each specific scene efficiently. This is achieved by introducing a tailored refinement mechanism geared toward high-quality view synthesis tasks, distinct from Spann3R's typical sparse reconstruction use case.
• Point-to-Ray Iterative Refinement: We designed an iterative refinement loop incorporating ray-consistent optimization and photometric adaptations to correct the inherent inaccuracies and biases often present in geometric foundation models during runtime. This separates our framework from standard pipeline reuse.
• Unified Progressive Pipeline: The combination of these components forms a cohesive pipeline that addresses foundational challenges in view synthesis, such as adapting to scene-specific conditions and maintaining scalability with high-resolution inputs, which remains a challenge for existing foundation pipeline methods.

While PnP and Gaussian rasterization are standard operations, their combined use with foundation model fine-tuning and our iterative, scene-adaptive approach is novel in the context of high-fidelity novel view synthesis.

2. Details on Spann3R Fine-tuning

We appreciate the request for additional clarification on how Spann3R is finetuned within our pipeline. Below, we summarize the process:

• Geometry Initialization: Spann3R predictions act as coarse geometric foundations and texture priors for our pipeline, providing initial states for Gaussian point clouds and precomputed rasterizations.
• Scene-specific Gaussian Prediction: Spann3R's outputs are refined dynamically using extensive fine-tuning (with photometric losses and gradient consistency) to predict high-fidelity, scene-specific Gaussian distributions. This step enables scene-adaptive representations without requiring explicit pose annotations.
• Photometric and Geometric Consistency: During each stage of refinement, Spann3R adapts moment-based Gaussian parameters by minimizing multi-view photometric errors and ensuring ray-aligned measurements across input images (addressing alignment inaccuracies in pre-trained foundation models).
• On-the-fly Optimization: The system iteratively updates Spann3R predictions for each incoming scene, producing output consistent with progressive geometric improvements for novel views. Gaussian scene distributions evolve naturally alongside these refinements, resulting in high-quality, view-consistent reconstructions.

We will expand this section in the revised manuscript and provide illustrative breakdowns of Spann3R’s fine-tuning process across different stages of the pipeline for greater clarity.

3. Fragility of the Progressive Neural Reconstruction Pipeline

We appreciate the concern regarding the pipeline’s perceived fragility due to its multi-step design. We acknowledge that complex pipelines often face challenges related to robustness and ease of deployment. However, we believe the following points address this concern:

• Step Modularity: While our pipeline leverages multiple distinct steps, each module (geometry foundation, Gaussian prediction, photometric refinement) is designed to operate independently, allowing for flexibility during implementation and scaling. For example, Spann3R can easily be replaced with alternative backbones without impacting the pipeline's overall functionality.
• Ease of Fine-tuning: Unlike traditional SfM methods, which rely on pose annotations and explicit camera predictions, our pipeline operates pose-free, reducing dependency on components prone to error. This makes the system less susceptible to breakdowns caused by pose inaccuracies or upstream variability.
• Empirical Robustness: Our results across diverse datasets (LLFF, Tanks-and-Temples, 7Scenes, etc) demonstrate stable synthesis quality despite inherent challenges (e.g., dense inputs, viewpoint diversity, appearance changes). This underscores the resilience of our design even under variable and complex input configurations.

While we understand the concern regarding multi-step pipelines, we argue that our modular design ensures flexibility and robustness during training and deployment.

4.Comparison to VGGT and MegaSAM We appreciate the insightful comparison drawn between DeepGfM and VGGT/MegaSAM and would like to elaborate on the differences and complementary nature of these approaches.

(1) Feed-forward Nature of VGGT and MegaSAM

Both VGGT and MegaSAM operate in feed-forward pipelines where foundation model predictions are applied directly to dense image reconstruction workflows. This design inherently offers computational speed and efficiency but comes with certain limitations:

• i. GPU Memory Constraints: Feed-forward methods often encounter severe GPU memory limitations during dense scene handling, constraining their scalability for large-scale datasets or high-resolution inputs. In our approach, the modular nature of the progressive pipeline allows decomposition of scene data into iterative levels of detail, effectively reducing the memory burden. This design better accommodates larger-scale reconstructions.

• ii. Pose Accuracy for High-Fidelity Synthesis: Foundation model-derived camera poses, as utilized by VGGT and MegaSAM, often lack the precision required for high-quality novel view synthesis. While VGGT and MegaSAM aim to improve reconstruction alignment, their reliance on pose estimation can introduce inaccuracies in high-fidelity outputs under challenging dense image set scenarios. DeepGfM circumvents this issue by adopting a pose-free progressive refinement approach, leveraging Gaussian geometry prediction to achieve superior ray-consistent novel views without explicit reliance on pose annotations.

• iii. Scene-specific Adaptation: VGGT and MegaSAM pipelines are not explicitly optimized for fine-tuning to adapt to specific scenes. In contrast, DeepGfM integrates scene-specific geometry prediction (via Gaussian rasterization and photometric corrections), producing fine-grained outputs tailored to the input set's particular conditions. This customization empowers a more detailed and view-consistent synthesis compared to feed-forward methods.

(2) Design Focus of MegaSAM:

MegaSAM, while valuable, is specifically suited for dynamic scene analysis and reconstruction (e.g., leveraging Droid-SLAM for motion-based scenarios). Our approach, instead, targets dense and static scene reconstruction workflows that demand ultra-high fidelity outputs without pose reliance. These complementary focuses highlight the distinct goals of MegaSAM and DeepGfM, with each addressing different application spaces.

(3) Clarification of Problem Statement and Novel Contributions

The problem DeepGfM addresses is the reconstruction of dense image capture workflows to enable high-fidelity novel view synthesis without reliance on pose annotations or large-scale memory-intensive methods. While feed-forward pipelines like VGGT and MegaSAM prioritize speed, DeepGfM emphasizes depth and quality of reconstruction, introducing the following key novelties:

• Pose-free Foundation Model Adaptation: Unlike VGGT/MegaSAM, which rely on pre-computed poses (potentially affected by inaccuracies), our pipeline operates without pose annotations. This is achieved by refining Gaussian geometries dynamically to align photometric appearance with ray-consistent novel view synthesis.

• Progressive and Modular Framework Design: The progressive design enables iterative scalability, addressing GPU bottlenecks evident in VGGT-like pipelines. Modularity ensures robustness against scene diversity, allowing refinement of components independent of memory constraints imposed by dense image sets.

• Scene-specific Gaussian Prediction: Our method dynamically predicts Gaussian geometries for each input scene, adapting to its unique photometric and geometric characteristics for high-quality synthesis -- a flexibility less evident in feed-forward methods like VGGT/MegaSAM, which process inputs less adaptively.

We emphasize that these methods complement our work, as MegaSAM excels in dynamic scene reconstruction, and VGGT provides strong speed-oriented solutions for dense workflows. In promoting each other, they strengthen the broader research direction of foundation model-driven image set reconstruction. We will revise our manuscript to clarify this relationship and explicitly highlight how DeepGfM complements these methods and contributes novel alternatives to high-fidelity workflows.

We are grateful for your constructive feedback and suggestions. Each point raised will be carefully addressed in our revised manuscript with additional details and clarifications to further substantiate the strengths and novelty of our approach.

Thank you for your thoughtful review and valuable suggestions. We address your comments point-by-point as follows:

1. State-of-the-art Approaches Not Covered (COGS, Limited Inclusion of VGGT)

We appreciate the suggestion to compare with additional state-of-the-art methods. However, we wish to clarify the reasons for not including VGGT and COGS in our evaluation:

• (1) VGGT: The released model is limited to 3D point and camera pose reconstruction, and the finetuned model for the novel view synthesis task was not made available. Additionally, VGGT requires substantial memory (hundreds of input images), which exceeds the capacity of our experiment settings conducted on a 24GB GPU. Thus, a direct comparison was not feasible within our experimental constraints.
• (2) COGS: This method is designed specifically for sparse view settings (at most 12 views, as outlined in the original paper), while our framework scales to hundreds of images without performance degradation. Comparing our approach to COGS would introduce an imbalance in the evaluation, as our method was designed for dense and highly complex datasets.

It is important to emphasize that we have already compared with multiple well-established state-of-the-art methods, such as NeRF, BARF, NeRFmm, Nope-NeRF, DBARF, COLMAP+3DGS, and ACE0+3DGS. These comparisons were performed under identical experimental settings, which we believe are sufficient to demonstrate the superiority of our framework over existing approaches.

2. LLFF Results Appear Worse

Thank you for raising this concern. We would like to clarify that, in our experiments, we strictly adhered to the standard settings and pipeline provided in the official repositories for all baseline methods, including CF-3DGS, COLMAP+3DGS, ACE0+3DGS, and others. No deviations or modifications were made to the training parameters or implementations.

Additionally, the LLFF results presented in Table 3 (main paper) and Table 11 (supplementary) align closely with the averaged and per-scene breakdown results reported in previous works such as NeRF (Table 5), BARF (Table 3), NeRFmm (Table 1), Nope-NeRF (Table 9), and DBARF (Table 1). These consistency checks confirm the fidelity of our evaluation methodology. Furthermore, all experiments follow the default settings and parameters established in the official 3DGS repository.

We believe this concern stems from a misconception, as our results objectively mirror the behavior demonstrated by baseline methods in prior papers. If the reviewer would like additional clarification or supplementary evidence, we would be happy to provide detailed breakdowns across more metrics or include additional visualizations for the LLFF dataset.

3. GPU Memory and Runtime Evaluation

Your feedback highlights an essential gap in our analysis. While GPU memory reduction is a primary focus of our work (Table 1), runtime metrics are equally critical for practical applications. We will evaluate total run-time for both individual substeps (COLMAP pose estimation and subsequent rendering for COLMAP+3DGS) in the revised manuscript.

In terms of runtime, our progressive training strategy optimizes complex neural architectures over multi-frame inputs, which indeed demands additional computational time compared to COLMAP. Nevertheless, our focus is on scaling high-fidelity reconstructions to larger datasets, for which existing pose-free COLMAP/3DGS-based methods' preprocessing remains insufficient. Our method demonstrates significant advantages over CF-3DGS, which is also a progressive method but requires days for training on datasets with hundreds of images and still suffers from out-of-memory issues. In contrast, our approach efficiently scales to hundreds of input images while remaining well within the memory constraints of a 24GB GPU. This efficiency is a key strength of our method and highlights its suitability for large-scale and high-resolution datasets.

Detailed runtime comparisons with failure cases for other approaches (where our method succeeds) will be included in the revisions.

4. Perceived Limited Novelty

We respectfully disagree with the notion that our method lacks sufficient novelty. Our method introduces several key innovations and contributions:

• Novel Training Pipeline: This is the first approach capable of finetuning an existing feed-forward 3D foundation model for high-fidelity novel view synthesis, a task that prior feed-forward methods could not achieve for dense inputs and fine-level reconstruction details.
• General Framework Design: While Spann3R is selected as a backbone for our experiments, it is not integral to the pipeline itself and can be replaced with other foundation models. This flexibility highlights the robustness and generalizability of our method, which enables a unified path for scaling different backbone approaches for novel view synthesis tasks.
• Iterative Refinement Pipeline: Our iterative refinement mechanism addresses geometric inaccuracies present in foundation models and introduces point-to-ray consistency alongside fine-tuned photometric losses. Such refinement has not been previously explored in pose-free pipelines, further demonstrating the uniqueness of our approach.

We sincerely appreciate your insights and constructive feedback. We are committed to addressing all concerns comprehensively in the revised manuscript.

We thank Reviewer wnHi for the insightful suggestion and for pushing the comparison to VGGT and COGS.

(1) For VGGT, we emphasize that it does not release a finetuned model for the novel view synthesis task. Therefore, we compare its camera pose accuracy with ours and other methods. And for a fair comparison on the novel view synthesis task, we use the camera poses and point clouds from VGGT for per-scene 3D Gaussian Splatting training. Regarding the memory limitations, the comparison is conducted on the LLFF dataset.

From the table, we can see that our method outperforms VGGT and COGS in camera pose accuracy and novel view synthesis quality. We also provide results compared with more baseline methods. We kindly suggest Reviewer wnHi refer to the section "4. Missing Comparisons for Splat Quality recommend" for Reviewer JX9E.

(2) For the LLFF results, it is true that "explicit 3DGS does a much better job of encoding texture details", and we did not see any issue or misalignment with this observation of the baseline results on the LLFF dataset. From Table 3 in our main paper, we can see that the baseline 3DGS method (COLMAP and COLMAP, which use COLMAP poses to train 3DGS) outperforms the baseline NeRF methods by a large margin both quantitatively and qualitatively in texture details.* For example, the SSIM of BARF is 0.626 while the baseline COLMAP is 0.794; the LPIPS of BARF is 0.238 while the baseline COLMAP is 0.132. In Figure 5 of the main paper and Figure 14 of the appendix, we can observe that COLMAP+3DGS is better than BARF in texture details; we can also observe the floater artifacts in the COLMAP+3DGS and CF-3DGS, which is a common issue with the 3DGS methods (root causes can be inaccurate camera poses, insufficient initialization point clouds, appearance changes, etc.), but it does not mean that the 3DGS baselines are worse than the NeRF baselines in texture details on the LLFF datasets. We also agree that with parameter tuning, the 3DGS baseline methods may achieve better results. However, we used the same parameters for training the 3DGS baselines, which provide sufficient and fair comparisons to all baseline methods. And since the results on the LLFF datasets of 3DGS baseline methods align with the observation that "explicit 3DGS does a much better job of encoding texture details", we do not think there is any necessity in retraining baseline 3DGS methods with more parameter tuning.

(3) "The benefit of feed-forward model would be fast inference/optimization. Using a feed-forward model by itself is not a contribution unless backed by clear motivation and results (like run-time eval above)." We did not claim that using a feed-forward model is a contribution of this paper. Instead, we want to emphasize that our method is designed to bridge this gap by enabling 3D foundation models to achieve:

• Comparable Pose Accuracy: Using our progressive registration and point-to-ray consistency loss, 3D foundation models can reach pose accuracy levels close to those of COLMAP/GLOMAP.
• Improved Rendering Quality: Our framework enhances the rendering fidelity of 3D foundation models, making them suitable for applications where visual quality is essential.

By combining the robustness and speed of existing 3D foundation models with the accuracy and rendering quality, our approach significantly improves their versatility for real-world applications. This hybrid approach focuses on optimizing foundational systems to work efficiently without discarding their inherent strengths.

Thank you for your thoughtful review and valuable feedback highlighting the strengths and weaknesses of our work. We appreciate your recognition of DeepGfM's improved quality over Spann3r and its strong performance in novel view synthesis and camera pose estimation. Below, we address your concerns point by point.

1. Choice of Pretrained Model (Spann3r)

We acknowledge the reviewer’s concern about the choice of Spann3r as the pretrained model, especially given its weaker performance compared to Dust3r, Mast3r/Mast3r-SFM. However, our decision to use Spann3r stemmed from several considerations:

(1) Balancing Memory Constraints with Scalability: Spann3r enables scalability and model initialization for large-scale image sets due to its lightweight memory retrieval architecture. While Dust3r and Mast3r provide stronger pretrained baselines, their memory limitations (outlined in their respective publications) often result in out-of-memory (OOM) issues for dense or extensive image sequences. Spann3r’s architecture allows foundational initialization for unordered collections without the prohibitive memory costs of Dust3r/Mast3r.

(2) Two-Stage Fine-Tuning Strategy: By designing DeepGfM’s incremental camera optimization over Spann3r’s pretrained framework, our method significantly improves camera pose estimation even when initialized with weaker models like Spann3r. This setup reduces dependency on the pretrained model’s inherent accuracy and allows DeepGfM to overcome challenging sequences effectively.

(3) Online Scene Reconstruction Background: We agree that Spann3r’s retrieval structure is indeed tailored for sequential rather than unordered data collections. However, our two-stage fine-tuning specifically compensates for unordered collections by progressively aligning photometric and geometric features during camera pose refinement. As demonstrated in the manuscript results (Figures 4-5), this approach consistently improves Spann3r’s outputs, capitalizing on its resource-efficient initialization while mitigating its design limitations.

2. Splat Prediction Head Training

We appreciate the question regarding the benefit of introducing a splat prediction head in our pipeline and its equivalence with optimizing vanilla 3DGS parameters. The splat prediction head offers several advantages:

(1) Scene-Specific Fine-Tuning: Unlike generic 3DGS parameter optimization, the splat prediction head is dynamically fine-tuned per input scene to capture Gaussian parameters specific to the photometric and geometric properties of the given set of images. This adaptive mechanism enhances splat quality and ray-consistent synthesis at the scene level, which would be harder to achieve with static optimization of global 3DGS parameters.

(2) Cost and Quality Efficiency: The splat prediction head reduces computational overhead associated with full geometric optimization by focusing on localized Gaussian prediction within the progressive framework. This modular refinement results in noticeable improvements in novel view synthesis (as highlighted by qualitative and quantitative results in Tables 3-4), with reduced memory demands compared to full-scale 3DGS optimization.

(3) Fidelity Improvements: Compared to vanilla 3DGS, the splat prediction head provides greater visual coherence and depth consistency in regions with high occlusion or sparsity. This benefit is particularly evident in challenging scenes, as demonstrated in comparisons to Spann3r and COLMAP-trained 3DGS.

3. Dust3r comparisons missing from Table 2

We acknowledge the absence of direct Dust3r results in Table 2. Since Dust3r has the OOM issue on a 24GB GPU with hundreds of images as input, we only report the results on the LLFF dataset (each scene contains only 20-50+ images): $\Delta \mathbf{R}: 0.79$, $\Delta \mathbf{t}: 0.173$. We will include these numbers in the revised manuscript. Preliminary evaluations indicate that DeepGfM demonstrates competitive pose estimation accuracy and improved scalability compared to Dust3r for large dense image sets.

4. Missing Comparisons for Splat Quality

We appreciate the reviewer’s suggestion to include comparisons with NoPo-Splat, Splatt3r, and MV-Dust3r+ for assessing splat quality. Since these methods are designed for sparse view rendering, we provide a more thorough comparison to SOTA methods on sparse view settings (2-views and 8-views). We conducted the experiments on the RealEstate10K dataset and the DL3DV dataset. For these datasets, we follow the protocol in Flare and NopoSplat, which samples 2/8 views for the testing video sequence as input. Note that Splatt3R and CopoNeRF only support inputs with 2-views, we did not provide the results on the 8-view setting.

• Comparison on two-view settings | Method | Dataset | PSNR ↑ | SSIM ↑ | LPIPS ↓ | |------------------ |----------------|------------|-----------|-----------| | DUSt3R | DL3DV10k | 13.47 | 0.390 | 0.590 | | MASt3R | DL3DV10k | 13.88 | 0.405 | 0.513 | | Splatt3R | DL3DV10k | 15.23 | 0.432 | 0.491 | | CoPoNeRF | DL3DV10k | 16.06 | 0.472 | 0.474 | | Flare | DL3DV10k | 23.04 | 0.725 | 0.182 | | NoPo-Splat | DL3DV10k | 24.19 | 0.745 | 0.173 | | MV-DUSt3R+ | DL3DV10k | 19.95 | 0.698 | 0.195 | | DeepGfM | DL3DV10k | 20.33 | 0.702 | 0.190 |

• Comparison on eight-view settings | Method | Dataset | PSNR ↑ | SSIM ↑ | LPIPS ↓ | |------------------ |----------------|------------|-----------|-----------| | DUSt3R | DL3DV10k | 17.37 | 0.601 | 0.390 | | MASt3R | DL3DV10k | 18.41 | 0.675 | 0.427 | | Splatt3R | DL3DV10k | - | - | - | | CoPoNeRF | DL3DV10k | - | - | - | | Flare | DL3DV10k | 23.33 | 0.746 | 0.237 | | NoPo-Splat | DL3DV10k | 22.77 | 0.722 | 0.240 | | MV-DUSt3R+ | DL3DV10k | 21.33 | 0.707 | 0.188 | | DeepGfM | DL3DV10k | 22.90 | 0.728 | 0.176 |

• Experiments summarization on sparse-view settings

(1) Comparison on Two-View Settings. Though DeepGfM is not specifically designed for sparse-view settings, it demonstrates competitive performance, surpassing several sparse-view methods.

• DL3DV10k Dataset: DeepGfM achieves a PSNR of 20.33, significantly outperforming DUSt3R, MASt3R, and Splatt3R. SSIM (0.702) and LPIPS (0.190) are also competitive against MV-DUSt3R+, while slightly behind Flare and NoPo-Splat, which are optimized for sparse-view tasks.

• RealEstate10K Dataset: DeepGfM records PSNR of 21.715, ranking higher than DUSt3R, MASt3R, and Splatt3R, while trailing Flare and NoPo-Splat. SSIM (0.753) is higher than MV-DUSt3R+ and comparable to NoPo-Splat. LPIPS (0.229) is better than both DUSt3R and MASt3R, showing robustness in perceptual quality.

(2) Comparison on Eight-View Settings. DeepGfM shows strong capabilities in handling sparse-view tasks even with more input views.

• DL3DV10k Dataset: With a PSNR of 22.90, DeepGfM outperforms MV-DUSt3R+ while approaching Flare and NoPo-Splat. SSIM (0.728) and LPIPS (0.176) are the best among all methods, highlighting DeepGfM’s ability to preserve structural and perceptual quality even with sparse-view conditions.

• RealEstate10K Dataset: DeepGfM achieves a PSNR of 23.370, outperforming MV-DUSt3R+ and NoPo-Splat, and remaining competitive with Flare (23.813). SSIM (0.815) and LPIPS (0.190) are also highly competitive, showing improvements over MV-DUSt3R+ and comparable values with Flare.

• Highlight: It is important to note that these models are optimized for sparse-view settings, whereas DeepGfM addresses dense-view workflows with hundreds of images. These results underline that, despite being designed for dense-view settings, DeepGfM performs competitively in sparse-view tasks, demonstrating robust generalization and adaptability across both two-view and eight-view configurations. This flexibility is beneficial for tasks requiring high-quality synthesis in diverse input-view scenarios.

We hope this addresses the reviewer’s concern, and we will include explicit clarifications regarding sparse-view vs. dense-view settings within the revised manuscript for better context.

Dear Reviewer JX9E,

Did we satisfactorily answer your questions? Would you like us to clarify anything further? Feel free to let us know. Many thanks.

Best regards,

Authors of #15711