UpFusion: Novel View Diffusion from Unposed Sparse View Observations

We thank the reviewer for their valuable feedback. In this section, we elaborate on concerns that are not covered in the common rebuttal.

There exists a plethora of related papers [1,2,3, 4]. However, pivotal experimental comparisons with [1,2,3,5] are missing.

We have updated the related works section to include discussion about more single-view methods, and will incorporate the reviewer feedback to emphasize the ‘object-centric’ of our presentation. Moreover, we have included comparisons against some state-of-the-art single-view baselines, details of which are elaborated in the common rebuttal section.

What are the specifics regarding the training time for each instance?

The neural mode distillation stage currently takes a little more than an hour to optimize per instance. We have included these details in the supplementary material.

The authors omit the background modeling in the manuscript (possibly using the masks in CO3D)

We do indeed use the masks provided in CO3D and sidestep modeling the background. We have clarified this point in the main text.

it would be useful if the authors could provide diverse rendered video of different objects, as well as producing metrics for your claim (of 3D consistency)

We have included GIFs that showcase objects estimated by UpFusion 3D in this anonymized website. Moreover, UpFusion 3D is 3D consistent by construction since we optimize a NeRF to represent an object (and NeRFs are 3D consistent).

Further, comparisons excluding the SRT/Diffusion model or contrasting it against [4] would be insightful

As recommended by the reviewer, we include quantitative and qualitative comparisons against FORGE (see table 3 and figure 6; and also the common rebuttal). We note that our model significantly outperforms FORGE when using 6 input views, and we believe this is because our approach allows bypassing explicit pose prediction which can lead to inaccurate predictions.

We thank the reviewer for their valuable feedback. We have added the references pointed out to our discussion.

I believe the contributions of UpFusion can be better justified when comparing to existing single view novel view synthesis works

We have included comparisons against some state-of-the-art single-view baselines, details of which are elaborated in the common rebuttal section. In particular, we find that our approach is competitive with the SoTA single-view methods given 1 input image, but can scale with more views to outperform the single-view baselines given additional (unposed) images.

We thank the reviewer for their valuable feedback. In this section, we elaborate on concerns that are not covered in the common rebuttal.

The following works could be discussed in the paper: [1,2,3,4,5] .... and can be compared with the proposed method experimentally.

We have updated the related works section to include discussion about more single-view methods. Moreover, we have included comparisons against some state-of-the-art single-view baselines, details of which are elaborated in the common rebuttal section. We hope this experiment illustrates the benefit of our approach which is competitive with the single-view methods given one view, but can benefit from additional (unposed) images unlike the single-view methods and outperform them in this scenario.

Quantitatively, the UpFusion 3D model has much better numbers than the 2D model, but visually it loses a lot of geometric details compared to the 2D results. ‘

While we agree that individual images from Upfusion2D are more detailed, these are often spurious details which might actually detract from the performance. In contrast, UpFusion3D recovered a consistent 3D mode whose renderings are ‘likely’ from all viewpoints, and thus maybe more likely to match the input.

but why is the LPIPS also better? -- LPIPS is not a pixel-aligned metric, while visually the 2D results look cleaner and have much more details than the 3D ones

LPIPS is technically pixel-aligned on a feature-grid, and does still care about alignment (although perhaps less than other metrics). We conjecture that even LPIPS for Upfusion3D is better because predicting spurious details is penalized more than predicting less detail. We would also point out that a similar trend is also observed in prior work (e.g. in SparseFusion, their full approach performs better than the underlying 2D diffusion model even in LPIPS).

It seems that the generated views sometimes have inconsistent colors as the input view (e.g. the blue bench and the blue umbrella in Fig. 8, Appendix A)

This is an interesting observation! As of now we believe that these colors could perhaps be close in latent space and hence the neural mode distillation process gets stuck on a mode that represents a faithful rendering of an object albeit with colors that are slightly off. We also see similar artifacts (perhaps more pronounced) when distilling 3D representations from Zero-123 with the SJC.

I wonder how the proposed method compares to this baseline: first running COLMAP to estimate the relative poses of the input views, and then running a pose-dependent 3D synthesis method?

In table 1 of the paper, we had already included a baseline which uses RelPose++ to estimate poses (which is shown to be superior to COLMAP in the sparse-view setting) and then use an independent pose-dependent 3D synthesis method, SparseFusion. Both, this experiment and the added comparisons to FORGE highlight that accurately estimating this pose explicitly is a hard task, and that methods relying on this are not robust.

We thank the reviewer for their timely feedback. In this section, we elaborate on concerns that are not covered in the common rebuttal.

Since this paper asserts novel view generation from sparse views using diffusion, it would also be equitable to compare with state-of-the-art single-image-to-3D baselines such as Zero-123.

We have now included comparisons with state-of-the-art single-image-to-3D baselines. We have included details in the common rebuttal. We hope this experiment illustrates the benefit of our approach which is competitive with the single-view methods given one view, but can benefit from additional (unposed) images unlike the single-view methods and outperform them in this scenario. We hope these results on GSO also address the concern regarding evaluation on additional domains.

Without specifying poses from images, it becomes ambiguous to define the pose of a target view unless canonical poses are provided

Following prior conventions e.g. UpSRT, we use the first input image as an anchor to define the orientation of the coordinate system. The pose of a target view is then defined w.r.t. this coordinate frame, thus making the task well-posed. While there is still a scale ambiguity in the reconstruction, we normalize our coordinate system during training to minimize this, and also allow the alignment during evaluation to not penalize methods for such ambiguity.

We mention this choice of the coordinate frame in Sec 3.2, and would be happy to expand on this further in the text.

For the task of novel view synthesis, it is advisable to include comparisons with end-to-end pose optimization baselines, such as BARF [2] and NoPe-NeRF [3] … The paper lacks a discussion and comparison with some relevant prior work, particularly with references [6] and [7]

Thank you for these references. We have edited the related work to discuss these further. While these methods can estimate pose, they are not applicable in our setting as they operate on densely-sampled images e.g. NoPe-NeRF presents results on images from dense trajectories, and BARF requires ~100 images for a 360-degree inference. As perhaps a more directly applicable baseline, we have now added a comparison to FORGE (please see common response above) which explicitly infers sparse-view poses followed by reconstruction, and we find that our system is more robust.

To enhance the evaluation methodology, it is recommended that the authors consider implementing a global alignment across all views collectively, rather than performing alignment on a per-frame basis.

We fully agree with the reviewer’s comment. In fact, we initially attempted gradient based optimization techniques to calculate a global alignment. However, in practice we found that it was not possible to obtain gradients to perform this computation across all baselines (e.g. differentiably transforming SparseFusion’s InstantNGP reconstructions was not feasible) and hence we settled for a per-frame alignment method which readily works across all baselines.

However, in the additional results presented on GSO, we forego the aligned metrics and report the ‘regular’ metrics as the training data on Objaverse has origin-centered and normalized meshes, thus removing the ambiguity (unlike CO3D, where each object is in arbitrary coordinate frame).

Could the authors provide insights into the motivation behind making the specific modifications to UpSRT

We use a pre-trained DINOv2 backbone (instead of a CNN) because DINO features have been shown to be informative for correspondences (which we assumed might help our downstream application).