PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting for Novel View Synthesis

Why not use MVSplat with MASt3D, given that this method falls behind MASt3R in camera pose estimation?

We opted not to use MVSplat with MASt3R because MASt3R's runtime of around 10 seconds conflicts with our goal of achieving a fast, feed-forward process. Moreover, our experiments include a variant, MASt3R*, which employs PnP+RANSAC for pose estimation similar to our approach, yet MASt3R* performs worse than both our baseline and final model, as shown in Tab. 2. We believe these reasons justify our design choice for the baseline.

Additionally, we wish to highlight that we have shown comparisons that outperform MASt3R. We kindly refer the reviewer to Tab. 6, where RoMa outperforms MASt3R, and our variant that uses RoMa enables further improvements in both image and pose estimation quality. Through this experiment, we highlight the flexibility of model choice for coarse alignment stage and a potential for performance improvements.

This paper assumes known camera intrinsics, whereas DUSt3r does not have this requirement. We thank the reviewer for highlighting this point. As discussed in Sec. A.6, we acknowledge that assuming known camera intrinsics is a limitation common to many NVS methods. Nevertheless, we conducted an additional experiment using UniDepth's intrinsic predictions instead of ground truth, and the results indicate that our method still performs well without GT intrinsics.

From the results, we highlight that even without GT intrinsics, ours outperforms CoPoNeRF and also performs on par with ours with GT intrinsics. This suggests a potentially feasible approach to alleviate the limitation.

Moreover, our primary focus is on generalized novel view synthesis from unposed images—where pose, depth, and correspondence estimation serve as intermediate tasks to achieve high-quality novel view rendering. In contrast to SfM-based methods such as DUSt3R, VGGSfM, and SP+SG, our approach differs in terms of data, objectives, and training setups. Notably, methods like DUSt3R and MASt3R require additional steps (e.g., training a radiance field via NeRF or 3D Gaussian Splatting), incur long optimization times per scene.

The improvement over InstantSplat is limited. We thank the reviewer for the comment. As shown in Tab. 5, our method outperforms InstantSplat by a large margin, and our inference speed is 100x faster, demonstrating a significant practical advantage. Moreover, when a more advanced correspondence network is adopted or test-time optimization is performed (like InstantSplat) for coarse alignment, training process or inference-phase optimization, our method further broadens the gap, as shown below:

Questions about “feed-forward” approach, because there is a module for a test-time iterative optimization for initial pose.

We wish to refer to reviewer k81v’s appreciation that “feed-forward” in our context emphasizes efficiency and speed (See comparison to InstantSplat). Although our coarse initialization stage employs iterative optimization, specifically, PnP+RANSAC, which is recognized as fast and efficient that takes only milliseconds (as also noted in NoPoSplat), the subsequent fine alignment module is fully feed-forward and represents a novel contribution. This fine alignment refines the coarse pose to correct 3D Gaussian misalignments, leading to significant performance gains (see Tab. 4). Leveraging off-the-shelf models for coarse alignment allowed us to focus on this key challenge (Our novelty in this aspect of coarse alignment is also recognized by the reviewer TAPU and EyLp), and we believe that our framework is novel and addresses a very important task in computer vision, which are appreciated by reviewer K81V and EyLp. We hope that this minor terminology point should not impact the overall evaluation of our work.

The authors assume that camera intrinsics are generally available, which is not always the case for generic video analysis tasks.

While we acknowledge that there are some corner cases where intrinsics might not be available, this limitation is common to many pose-free feed-forward NVS methods, as discussed in Sec. A.6. Nonetheless, when using intrinsic estimates from UniDepth at inference, our method still outperforms CoPoNeRF as shown below. We wish to stress that in practical applications, intrinsics are generally available, and even in cases where they are not, our approach maintains competitive performance relative to other methods facing the same limitation.

From the results, we highlight that even without GT intrinsics, ours outperforms CoPoNeRF and also performs on par with ours with GT intrinsics. This suggests a potentially feasible approach to alleviate the limitation.

The study overlooks some state-of-the-art baselines, such as NoPoSplat and MASt3R-SfM, both of which had publicly available code well before the ICML submission deadline.

We thank the reviewer for highlighting NoPoSplat and MASt3R-SfM. As noted by reviewer EyLp and the reviewer guideline explicitly states, NoPoSplat is a concurrent work—released on arXiv less than three months before our ICML submission and accepted by ICLR three days prior—so we are not obliged to compare against it, though it is acknowledged in the related works (Nonetheless, we provide additional comparison below, as the reviewer kindly suggests). Moreover, MASt3R-SfM is an arXiv paper and, while it shares similar objectives in 3D reconstruction, it addresses a different task than our pose-free feed-forward NVS (its sparse view variants (MASt3R) are already compared with our method in Tab. 2).

Below includes the comparison:

While NoPoSplat performs well, as indicated in the table, we stress that our method clearly differentiates in the setting that the 3D geometry data, such as GT camera pose, is not utilized during training. Moreover, we show in the following that our method can be naturally extended to N-view setting, where NoPoSplat struggles.

To summarize, compared to NoPoSplat, our approach offers key advantages compared to NoPoSplat: it is a conceptual contribution proposing a general pipeline for pose-free view synthesis that addresses 3D Gaussian misalignment using off-the-shelf depth and correspondence networks and fine alignment modules; it naturally extends to N-view inference without retraining; and it is designed for scenarios without 3D geometry data (GT camera pose) during training. We will additionally include this discussion in related works.

FreeSplatter not discussed in the paper.

We thank the reviewer for highlighting FreeSplatter[1]. FreeSplatter addresses the same task using an LRM-based architecture that directly maps images to 3D Gaussians. To tackle coarse alignment, it employs staged training with early supervision from 3D geometry data (ground-truth pointmaps constructed using pose and depth), which is essential for convergence. This supports our claim that misaligned 3D Gaussians hinder learning, and contrasts with our approach, which incorporates a dedicated coarse alignment module. We will include this discussion in the related works section.

The contribution mainly relies on the refinement modules. While it is effective, much of the work builds on existing works or ideas.

While our approach leverages existing methods for coarse alignment (while recognized as creative by reviewer TAPU), our main technical contributions lie in the novel fine-alignment modules. In addition to our findings to identify a unique challenge in pose-free feed-forward 3DGS, we address the challenge by proposing new losses, fine alignment modules, and a confidence estimation module. As demonstrated in Table 4, each component significantly contributes to performance improvements, underscoring the technical innovation of our work.

NoPoSplat [4] is acknowledged but not evaluated and compared.

As mentioned by the reviewer EyLp, NoPoSplat is a concurrent work released on arXiv less than three months ago and accepted only three days before our submission. Moreover, as shown in Table 2, fair comparison is challenging since other works (CoPoNeRF and NoPoSplat) use additional supervisory data (GT camera pose) during training. Nonetheless, we agree that including NoPoSplat as a baseline would further enhance our work. Below includes the comparison:

While NoPoSplat performs well, as indicated in the table, we stress that our method clearly differentiates as they use GT camera pose during training. Moreover, below shows our method can be naturally extended to n-view setting, where NoPoSplat struggles.

More qualitative analysis for the ablation study.

We thank the reviewer for the valuable suggestion. Although it has been challenging to identify samples that clearly distinguish all the variants in Table 4 during rebuttal period, we agree that including such examples would further strengthen our work. We will incorporate additional qualitative results in the updated version of the paper.

In Table 4, Please explain (I-II) Scale/Shift Tuning Depth Network in the paper.

Variant (I-II) in Table 4 refers to the approach where we fine-tune the scale and shift parameters used to convert relative depth estimates into UniDepth predictions. We observed that directly fine-tuning the UniDepth parameters resulted in unstable training, indicating that a more sophisticated training strategy may be required.

Around line 211 and 212, does the inputs to 3D Gaussian parameter prediction also include the refined pose? I assume that gradient can also be flowed from the refined depth input.

Yes, that's correct. The refined pose is used to construct the standard MVS cost volume, which is then aggregated with the guidance cost volume. Consequently, gradients do flow from the refined depth input as well, contributing to the overall optimization process.

Intuition of confidence estimation and what exactly does it help with the reconstruction overall?

This confidence score is used to modulate the regression of 3D Gaussian parameters, such as opacities and covariances, ensuring that more reliable matches have a greater influence on the reconstruction. Empirically, this approach improves both rendering quality and pose estimation accuracy, leading to an overall improved reconstruction, as shown in the Tab. 4.

How did you determine the lambda for 3D-3D loss?

We set the lambda for the 3D-3D loss to 0.05 by balancing its scale with the other losses in our framework. This value was chosen empirically, and a more extensive hyperparameter search might yield performance improvements.

Given that NoPoSplat was accepted to ICLR 2025 only few days before the ICML submission deadline, its omission is understandable. However, incorporating it as a baseline in future revisions would further strengthen the claims of state-of-the-art performance.

We appreciate the reviewer’s valuable suggestion. As noted, NoPoSplat is a concurrent work released on arXiv less than three months ago and accepted only three days before our submission. Moreover, as indicated in Table 2, fair comparison is challenging since other works (CoPoNeRF and NoPoSplat) use additional supervisory data (GT Camera pose) during training. In contrast, our approach can be learned without such data, clearly differentiating our approach to them. Nonetheless, we agree that including NoPoSplat as a baseline along with detailed comparison would further enhance our work. Below includes the comparison:

While NoPoSplat performs well, as indicated in the table, we stress that our method clearly differentiates in the setting that the 3D geometry data, such as GT camera pose, is not utilized during training. Moreover, we show in the following that our method can be naturally extended to N-view setting, where NoPoSplat struggles.

To summarize, compared to NoPoSplat, our approach offers key advantages compared to NoPoSplat: it is a conceptual contribution proposing a general pipeline for pose-free view synthesis that addresses 3D Gaussian misalignment using off-the-shelf depth and correspondence networks and fine alignment modules; it naturally extends to N-view inference without retraining; and it is designed for scenarios without 3D geometry data (GT camera pose) during training. We will additionally include this discussion in related works.

Did you consider or experiment with other parameterization strategies or approaches to mitigate the impact of inaccuracies in 3D Gaussian center localization on noisy and sparse gradients?

We appreciate the reviewer’s suggestion. While we have not yet experimented with alternative strategies like PixelSplat, we agree that mitigating inaccuracies in 3D Gaussian center localization is promising. Future work could explore leveraging NeRF representations, as in RADSplat, or formulating a probabilistic approach similar to PixelSplat. We will include a discussion of these alternatives in the supplementary material.

Could you provide further analysis on why inaccuracies in 3D Gaussian center localization result in noisy and sparse gradients? Specifically, what factors contribute to this phenomenon? Including citations that discuss this phenomenon would also improve credibility.

We thank the reviewer for the valuable suggestion. In our framework, we claim that inaccuracies in 3D Gaussian center localization lead to noisy and sparse gradients because only the Gaussians within the rasterization window receive gradients. When centers are mislocalized, few Gaussians get effective updates, resulting in weak guidance, especially when initialized far from the optimal positions [1]. For this, we stressed the importance of coarse alignment. Moreover, providing photometric only showed limited effectiveness, in which we proposed 2D-3D consistency and regularization losses for addressing this. Additionally, [2] and NoPoSplat highlight that when the network is initialized without 3D priors and reliable 3D geometry data during early training (e.g., GT pointmaps) to compensate for such priors is absent, convergence is difficult (Early training stage for FreeSplatter and CroCo initialization for NoPoSplat). We will include these discussions and citations in the final version of the paper.

[1] Jung, J., Han, J., An, H., Kang, J., Park, S. and Kim, S., 2024. Relaxing accurate initialization constraint for 3d gaussian splatting. arXiv preprint arXiv:2403.09413.

[2] Xu, Jiale, Shenghua Gao, and Ying Shan. "FreeSplatter: Pose-free Gaussian Splatting for Sparse-view 3D Reconstruction." arXiv preprint arXiv:2412.09573 (2024).

The paper states that correspondence models like LightGlue struggle in low-texture regions. It is unclear how PF3plat can improve in these areas.

We enhance performance in low-texture regions through two complementary approaches: As explained in line 233, our proposed 2D-3D consistency loss encourages corresponding points to lie on the same surface, drawing from principles of multi-view geometry. The effectiveness is shown in Table 4 (I-III).

Second, the flexibility of the design choice at coarse alignment stage allows us to use other methods beyond LightGlue. For example, replacing LightGlue with networks like RoMA has led to noticeable improvements, as demonstrated in Table 6.

Lack of qualitative results for pose estimation.

Please refer to Figure 7, which provides a visual comparison for pose estimation. Additionally, we have included the baseline pose estimates in this figure to clearly demonstrate the effectiveness of our fine alignment module. Please refer to the following anonymous link: https://anonymous.4open.science/r/7048/7048_camvis.pdf

However, the method still requires GT camera intrinsics, which may not always be available in real-world scenarios.

Camera intrinsics are commonly required in previous pose-free methods, such as DBARF, CoPoNeRF and NoPoSplat, as discussed in Sec. A.6. While this is a common limitation, for this rebuttal, we have conducted an additional experiment using camera intrinsics from UniDepth prediction:

From the results, we highlight that even without GT intrinsics, ours outperforms CoPoNeRF and performs on par with the model with intrinsics. This suggests a potentially feasible approach to alleviate the limitation.

Mast3R for pose estimation + MVSplat for rendering.

We opted not to use MVSplat with MASt3R because MASt3R's runtime of around 10 seconds conflicts with our goal of achieving a fast, feed-forward process. Despite our superior performance in in RealEstate10K, thanks to the relatively larger-scale outdoor training of MASt3R, it is true that MASt3R performs slightly better than ours in pose estimation in ACID. Nevertheless, this is compensated when compared to MAST3R* in Tab. 2. We also highlight that with more advanced correspondence network, RoMa, ours further broadens the gap, as shown in Tab. 6.

Nonetheless, for this rebuttal, we provide an additional comparison below:

From the results, we find that ours outperforms the baselines, thanks to our fine alignment modules and confidence estimation that contribute to further improvement.

ACID dataset results indicate weaknesses in large-scale outdoor scenes

We acknowledge that our performance in large-scale outdoor scenes (dynamic coastline environments), is not optimal, as discussed in Sec. A.6. We attribute these challenges primarily to UniDepth's training dataset, which, unlike DUSt3R or MASt3R, was built on a smaller and less diverse collection of outdoor scenes. A straightforward approach would be to leverage depth models that are trained on large-scale outdoor scenes or advanced correspondence models, to yield more accurate initial 3D Gaussian locations.

The explanation of how the refined poses are computed is unclear

In Section 3.2.3, the refined pose is computed using three distinct inputs: Plücker Coordinates, Feature Maps and Pose Token. Each of these inputs is processed through a series of attention layers, which help propagate information about previous camera estimates, multi-view geometry, and the current camera state. After fusing this information, the pose token is passed through a simple MLP that predicts residual rotation and translation parameters. These residuals are then added to the coarse camera parameters to yield the final refined pose.

Cost Volume Construction and Aggregation, how does this differ from prior multi-view stereo cost volumes?

Our approach differs from traditional MVS cost volumes by introducing a guidance cost volume. Since the cost volume built from our estimated camera poses is inherently noisy, we supplement it with a guidance cost volume derived from a monocular depth estimate. By aggregating these two volumes, we construct a fina cost volume better tailored for a pose-free setting.