Missing baseline.

We would like to thank the reviewer for the suggestion on baselines. We include the comparison between the proposed approach and SPARF in General Response. In additions, we also compare our method with UpFusion on novel view synthesis:

UpFusion performs slightly better than LEAP, while our approach constantly outperforms both pose-free baselines with two different pose initialization and view numbers. Our method improves consistently by leveraging more precise initial camera poses and more observed input images. However, LEAP and UpFusion show little performance increase when more views are available due to their geometry unawareness. We also include a qualitative comparison between our approach and UpFusion in Fig. 4 of the rebuttal PDF file.

Unknown inference speed.

We apologize for the ambiguity in our earlier version. Our shape-pose optimization is fast, benefiting from the efficient Gaussian Splatting compared to NeRF baselines, and on average our inference time is about 9 minutes, whereas SPARF may take more than 10 hours to train a full model. We discuss inference time more thoroughly in our General Response and will revise our paper according to the review's feedback.

Related work.

Thanks for mentioning related work! We will incorporate our discussion into our updated version.

• iFusion leverages diffusion priors from the diffusion model (i.e., Zero-123) for pose estimation in a similar manner to ID-Pose where relative camera poses are parameterized as conditioning to the diffusion model and are refined via gradient descent from diffusion noise prediction. Compared to iFusion, we also leverage diffusion priors from Zero-123 but we optimize camera poses directly via photometric error from differentiable rendering. In this process, we explicitly form a 3D representation (3DGS) for optimization.
• FORGE (which is earlier work from the authors of LEAP) uses the synergy of pose estimation and 3D reconstruction by inferring 3D features that are shared by both tasks. Our work shares a similar high-level idea in that we make use of the estimated camera poses for 3D reconstruction, and the reconstructed 3D, in turn, is exploited to help refine camera poses and identify outliers. By doing this, we also hope to make both sub-tasks to benefit from each other.

In our final version, we will add detailed discussions about pose-free approaches (e.g., LEAP, UpFusion) in our Related Work section.

Ablation study.

We thank the reviewer for suggesting an ablation study on DUSt3R, and present the results below:

The introduction spends a lot space discussing the chicken-and-egg problem of pose estimation and reconstruction... Why the authors want to emphasize this?

We agree with the reviewer that classical SfM methods did indeed tackle this chicken-and-egg problem. Our introduction was instead motivated by ignorance of the chicken-and-egg nature in the more recent learning-based sparse-view reconstruction methods. For example, approaches like LEAP/UpFusion seek to side-step the pose estimation task altogether. On the other hand, methods that explore generative-prior-based reconstruction (e.g. ReconFusion/SparseFusion) assume perfect poses. Even methods that jointly optimize 3D and pose (e.g. BARF/SPARF/FvOR) are designed to use noisy ground-truth poses or near-perfect initial poses (as opposed to ones output by real systems with possibly large errors).

We agree this emphasis maybe obvious to any reader familiar with the rich history of the field, but believe this is useful for readers only familiar with a recent history. That said, we would be happy to revise the introduction if the reviewer suggests.

Statistics of Outlier Removal.

We again thank the reviewer for this advice. With the same setting as Table 4 in our main paper, we investigated the relationship between the identified outlier numbers and the corresponding sequence numbers in our dataset (174 in total) and their initial average rotation error.

In general, sequences with higher rotation errors tend to be identified with more outliers. In this setup, our method identified ~0.94 outliers per sequence. We also computed the rotation error at an image level for outliers and inliers with the average over all samples as reference:

All: 14.92° Outlier: 19.75° Inlier: 13.53°

The results indicate that the outlier we found indeed has a higher error than others, verifying the effectiveness of our outlier identification approach.

The proposed method is compared with SPARF only in the setting of using pose from different pose estimation baselines. However, it would be more convincing to also present the results using the same setting of SPARF, which adds noise into the GT camera pose. This will be a direct comparison with SPARF’s original results reported in their paper.

Thanks for the advice! We experimented with the suggested setting using eight images, where Gaussian noise is added to ground truth camera poses (so the initial poses have a rotation error of about 9.71°). A comparison between the proposed method and SPARF on pose accuracy is presented below:

While our method yields consistent improvements over all metrics, we observed that SPARF can sometimes degrade performance compared to the initialization. As SPARF leverages correspondence, its pose optimization is ineffective (or even diverges) when no reliable correspondence (or false correspondence) is found. The results are consistent with our experiments on DUSt3R poses, which we include in our General Response. We refer the reviewer to the top-level response for more details. Please note that the experiments in the original SPARF paper were done using image sets with significant overlap (e.g. DTU dataset where all images observe the same aspect of an object).

The proposed method is compared with LEAP for 3D reconstruction results. However, the comparison is a bit unfair since LEAP does not require any initial camera poses.

For an additional baseline, please see our general response where we additionally compare to SPARF on novel-view synthesis. We would also be happy to report any additional comparisons the reviewer suggests.

The description of how to effectively detect the outliers (line 212 - line 214) is not very clear. Similarly, the procedure of how to correct the outlier poses (line 223 - line 225) is not very clear either. How the MSE and LPIPS are computed and compared since there is no correspondence?

We apologize for being unclear in our submission and will update the text based on the reviewer's feedback. In outlier correction, we do render-and-compare for each identified outlier, where we resue the reconstructed 3D from only the inliers (obtained in Iterative Outlier Identification). By sampling pose candidates, we can render images that can be compared with the target image (of the outlier) with metrics such as MSE and LPIPS. Therefore, correspondence is not required in the process. We included detailed explanations for outlier removal and correction in our General Response, and we hope that this will resolve the reviewer's concern.

The proposed method is evaluated on the NAVI dataset. It seems that the dataset is quite simple as shown in Fig. 3 and Fig. 4. The reviewer is wondering about the performance of the proposed method on more complex scenes?

We primarily tested on NAVI because it provides ground truth 3D meshes that are unavailable in other datasets. This enables the evaluation of 3D metrics such as F1 for our 3D reconstruction. In addition, NAVI offers highly accurate object masks and high-quality sparse-view image sequences. However, we include qualitative results of our approach on more challenging scenarios in Fig. 1 of our rebuttal PDF, where we show two self-captures and four challenging instances from the held-out set of CO3D. These instances have more complex textures and shapes than the NAVI samples, which verifies the proposed approach's generalization ability.

The reviewer is wondering about the separate ablation results on the outlier removal and correction.

We would like to thank the reviewer for mentioning this ablation. We followed our setup in Table 4 of the main paper, but we only do Iterative Outlier Identification without the following correction stage.

Compared to the baseline Ray Diffusion numbers, this experiment setup benefits from iterative pose-3D co-optimization, improving the base pose accuracy by a notable margin. Moreover, the results verify that an additional outlier correction is necessary for higher pose accuracy and high-fidelity novel views.

This optimization-based method requires more time compared to a feed-forward model, taking about 5-10 minutes. Additionally, the writing discussing this aspect is somewhat unclear: the paper states, “with increased inference time depending on the number of outliers.” Could this statement be more specific? How much does the time increase with the number of outliers? The correction of outliers may be time-consuming as it requires dense searches of initial camera poses.

We apologize for the lack of clarity and address the inference time in our general response above, and will update the final version to include these details. We agree that our system is slower than feed-forward methods (e.g LEAP), but allows significantly more accurate generations. Compared to prior NeRF-based optimization methods however, our gaussian-splatting based system is far more efficient e.g. SPARF takes 10hrs per instance whereas our systems takes ~9min.

The authors do not sufficiently discuss experiments in a more “standard” sparse-view setting, such as using 3 or 4 views. The reported experiments use at least 6 views, which is not a particularly small number.

We chose 6-8 images as a setting representative of online marketplaces, but agree with the reviewer that it is important to also study even fewer views and report an experiment analyzing N=4 below.

While our system does lead to consistently improved poses, the gains are less prominent compared to settings with more views. We believe this is because our approach does rely on multiple images 'co-observing' common 3D regions (to guide pose correction), but these are not common if we randomly sample a small set of views around an object, thus leading to diminishing benefits with very few images.

A related work is lacking in discussion: Sun, Yujing, et al. "Extreme Two-View Geometry From Object Poses with Diffusion Models." arXiv preprint arXiv:2402.02800 (2024).

We thank the reviewer for sharing relevant work, which we will incorporate in our final version. The process of matching the generated images with the target image is similar to our outlier correction procedure at a high level, but we leverage a reconstructed 3D representation to render novel views instead of purely using diffusion models to generate them. Regarding this perspective, our novel views may be more detailed and faithful as we leverage multiple input views.

Is the testing data included in the training set for fine-tuning the 6-DoF diffusion model?

No, we did not include NAVI in our data to fine-tune Zero-123 with 6-DoF camera conditions.

The core idea is to use a pose-conditioned diffusion model (Zero-123) as a prior, impose the SDS loss, and jointly optimize the poses and objects, similar to the approach in ID-pose.

We would like to clarify that our method is not similar to ID-Pose. ID-Pose estimates camera poses by optimizing the relative pose conditions in Zero-123 for image pairs while it does not form any 3D representation. In fact, we adopted ID-Pose as initialization for our experiments on three synthetic datasets in the supplementary and show that our systems leads to significant gains over it.

This paper presents very limited novelty in the reconstruction part with a trivial extension to DreamGaussian to use multi-view images, which is already implemented in a public repository stable-dreamfusion.

We respectfully disagree with the reviewer's assessment that our extension to DreamGaussian is trivial. While using multiple input images for DreamGaussian has been implemented in the stable-dreamfusion GitHub repository (as mentioned in L173-174 of our main paper), our work goes further by enabling the handling of 6-DoF camera poses and gradient-based pose optimization. To achieve this, we fine-tuned Zero-123 with a novel 6-DoF camera parameterization and implemented customized CUDA kernels, as the official Gaussian Splatting codebase did not support this feature.

We appreciate that the reviewer recognizes the novelty of our pose refinement strategy. However, we kindly ask the reviewer to also consider the broader contributions of our proposed system and the specific problem we aim to address. Specifically, we integrate gradient-based pose optimization with generative priors to tackle sparse-view pose refinement, a scenario prone to overfitting. Additionally, we introduced an outlier identification and correction system to manage significant initial pose errors so that our method can work in real-world scenarios where only estimated camera poses from off-the-shelf methods are available. To the best of our knowledge, no existing work effectively addresses this challenging yet practical setup.

The major weakness of the paper is the lack of fair comparisons in terms of the 3D reconstruction.

Please refer to our general response, where we include an evaluation of SPARF regarding novel view synthesis. We apologize for not including these in our earlier version -- SPARF training is slow (the overall training takes more than 10 hours per instance on a single GPU such as V100) and due to limited resources, we only compared to SPARF's pose optimization stage (which takes 30% iterations). We also note that the suggested baseline DMV3D is not applicable in our setup -- it generates multiple views as output but does not allow multiple unposed views as input! Moreover, their implementation is not open-sourced.

I'm quite curious what the reconstruction quality the method can achieve without 3D generative prior but with the proposed refinement. Namely the combination of (1) and (4) in Table 4.

We thank the reviewer for suggesting this experiment. We conducted the ablation study under the same settings as Table 4 and compared the results with our full system:

In this setup, the quality of the 3D reconstructions is worse as there are more artifacts (significant roughness and holes) in the geometry. However, we found that this setup does achieve high pose accuracy (even being more precise on Rot@5° than our full system). Analyzing further, we found the no-SDS setup makes it easier to remove outliers (removing one input image more significantly reduces the reprojection errors). Quantitatively, 97.1% of sequences in this setup involve at least one identified outlier, compared to 67.2% in the full system.

Perhaps more surprisingly, we found that this setup also yielded slightly better view synthesis results (PSNR and LPIPS), but this did not correspond to the qualitative results (see Fig. 2 in our rebuttal PDF file). We hypothesize that this discrepancy is because we compute a single global alignment between our optimized camera poses and GT poses before evaluating NVS, and this alignment maybe slightly more precise for the baseline leading to improved NVS metrics even though the 3D/view-synthesis quality is worse. To mitigate the effects of alignment in NVS evaluation, we also report PSNR*, and LPIPS* -- where we locally optimize camera pose for each novel view for a few steps using photometric error while keeping our 3D representation fixed. We see that our full system does yield better predictions, highlighting that the use of SDS does indeed improve the novel-view generations.

How are the poses used for generative prior sampled in addition to the input views?

We preprocess the initial camera poses provided by off-the-shelf methods to align the world origin with the approximate intersection of their optical axes. Next, we create a sphere centered at the origin, with a radius equal to the average distance of all cameras from the origin. When rendering novel views for SDS, we sample points on the sphere for camera translation using random azimuth and elevation angles. For rotation, we orient the sampled camera towards the origin.

How are the thresholds for pose outlier removal tuned?

During the early stages of our work, we conducted initial experiments on a few samples from the CO3D dataset and found that a threshold of 0.05 for LPIPS generally worked well. This setting was maintained for all our in-the-wild evaluations on NAVI reported in the paper. We did not tune the threshold further for these results, though a fine-grained search might yield improved performance.