MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views

Response to Reviewer 3TXW (R4)

 

Q1: Limited technical contributions.

A1: Kindly refer to the global response to all reviewers for more discussions regarding our contributions.

 

Q2: The scores of quantitative results on DL3DV are smaller than expected.

A2: These values are reasonable since the latest generative 3D synthesis is different to the traditional 3D reconstruction. The PSNR, SSIM, and LPIPS are all pixel-aligned metrics, which are not fully correct for the generated new content. Our MVSplat360 is able to generate multiple and diverse results for the occluded and invisible regions, which are consistent with high quality, but may not match the original ground truth. Note that our generated 360 video is consistent (supplementary video), and the results match well with the inputs’ distribution (FID score improves around 13.4)

For better reference, we refer to the scores of Reconfusion (CVPR24) and CAT3D (arXiv24) tested on a similar dataset mip-NeRF 360. As reported in their paper, Reconfusion scores (PSNR=16.93, SSIM=0.401, LPIPS=0.544) and CAT3D scores (PSNR=17.72, SSIM=0.425, LPIPS=0.482). Note that even with per-scene optimisation, their scores are not impressive by using 6 observed views and averaging over only 9 scenes, while our scores are reported with a more challenging feed-forward manner, using only 5 observed views and averaging over 140 scenes.

Nonetheless, our MVSplat360 outperforms all existing state-of-the-art models, as reported in Tab. 1. The extensive visual results in both the paper and the supplementary video also verify its superiority.

 

Q3: The rendering speed is slow.

A3: Unlike the latest related works, such as ZeroNVS, ReconFusion and CAT3D, that trained additional 3D NeRF for each scene, our MVSplat360 can directly generate consistent novel view video in a feed-forward architecture, which is much faster than these related models (see further discussions in the global response). As discussed in the limitation section (L517-L520), we are aware that integrating with a diffusion model will lead to slower rendering speed. This is caused by the multi-step denoising process, while it can be sped up by those improved one-step SVD models such as SF-V. Since this is beyond the main focus of our work, we leave it for future improvement. Besides, although latentSplat runs faster, its rendering quality on complex scenes is far from satisfactory, as demonstrated in the supplementary video, and latentSplat’s GAN-based decoder is unstable and can cause the model to collapse on complex scenes (L248-L249).

• SF-V: Zhang, Zhixing, et al. "SF-V: Single Forward Video Generation Model." arXiv preprint arXiv:2406.04324 (2024).

 

Q4: Comparisons with “MVSplat Encoder + latentSplat Decoder”.

A4: We replace latentSplat’s encoder with the MVSplat’s encoder, and its performance is reported below.

As reported, replacing latentSplat’s encoder with the MVSplat’s slightly improves the performance since its backbone features are enhanced. However, latentSplat’s GAN-based decoder still limits its performance in modelling complex scenes. Our MVSplat360 again outperforms this upgraded latentSplat across all metrics, assuring the superiority of our design.

 

Q5: Testing with more than 7 views.

A5: This work mainly targets the sparse view scenario, which is more challenging yet practical for casual users. Testing the feed-forward model (refer to the difference between feed-forward and per-scene tuning models in the global response) with a large number of views is not our primary focus, which is also quite challenging. The main reasons are: 1) Our backbone model MVSplat, similar to pixelSplat, predicts 3D Gaussians in a pixel-align manner, it becomes more difficult to correctly align Gaussians from different views as the view number increases, leading to more artifacts. The unreliable reconstruction results will then impact our refinement module, potentially leading to unexpectedly worse quality. 2) Our model is trained on 5 views, testing it with a significantly different number of views might lead to less effective performance. Hence, we only verify our model within a reasonable range (from 3 to 7 views) to ensure the correctness of our conclusions.

 

Q6: Inter- and extrapolation experiments on RE10K.

A6: We train two separate models for the RE10K experiments, one for interpolation and one for extrapolation. To maintain a fairer comparison, we retrain latentSplat’s model using the interpolation-based training strategy, and its performance is reported in the table below. Our MVSplat360 remains the best. We will update it to Tab. 2 in the updated version.

 

Q7: The PSNR improvement on RE10K is smaller than expected.

A7: As replied in A2, the generative 3D synthesis is different from the traditional “accurate” 3D reconstruction. MVSplat360 has a more obvious improvement in the feature-level metrics (L274-L277), although the gain is smaller in the pixel-aligned metric. This might be caused by 1) the oversaturated issues of the SVD model, as detailed in the limitation section (L509-L511); 2) the generative model only provides a reasonable guess for the unseen regions, which is not guaranteed for a pixel-align match with the original ground truth.

Dear Reviewer 3TXW,

Did we satisfactorily answer your questions? Would you like us to clarify anything further? Feel free to let us know, many thanks.

Best regards,
Authors of #2348

We thank the reviewer for the follow-up comments. We are very grateful for your recognition of our key contribution: good visual quality in an under-explored setting of 360 scene synthesis.

Below, we summarise information to justify the running efficiency, in case you, other reviewers and AC want to refer to it during the internal discussion.

 

• The task of feed-forward 360 scene synthesis from sparse views is very challenging.

  • Sparse views (e.g., 5 views in our experiments) provide only limited coverage of a scene, resulting in many unseen or occluded regions when rendering from 360-degree viewpoints. This causes significant challenges to previous feed-forward methods like pixelSplat, MVSplat and latentSplat. Prominent failures can be observed in all these methods, as demonstrated in our supplementary video.
  • Another line of research, like ReconFusion and CAT3D, relies on additional per-scene optimization to achieve plausible results, which is an order of magnitude slower than our method (minutes vs. seconds (Ours)).
  • In this paper, we demonstrate that high-quality feed-forward 360 scene synthesis can be achieved from sparse views, without any additional per-scene optimization.

• The efficiency of video diffusion models can be potentially improved with the latest techniques since video diffusion models are actively developed and advanced.

  • The main efficiency bottleneck of our method lies in the video diffusion model, which is slow since it requires the multi-step denoising process.
  • However, we note that speeding up video diffusion models is an active topic and advanced techniques are emerging. For example, the recent SF-V model speeds up SVD with improved one-step denoising. Our model can benefit from such advancements.

 

Overall, we provide a viable solution to the under-explored task of feed-forward 360 scene synthesis from sparse views, we believe our efficiency can be further improved with more advanced video diffusion models.

Feel free to let us know if you would like us to clarify anything further. Many thanks.

Response to Reviewer pb1U (R3)

 

Q1: Unsatisfactory multi-view consistency in complex scenes.

A1:

1. The setting is extremely challenging: as verified by all provided visual results, MVSplat360 is the only approach that can provide reasonably good results on occluded and invisible regions compared to other state-of-the-art feed-forward models. Although concurrent work CAT3D might have a similar demo, it differs from our setting as it relies on per-scene optimization. Please refer to the global response for more discussions between feed-forward and per-scene tuning settings.

2. Long-sequence multi-view consistency for the diffusion model is still under exploration: built on top of the video diffusion model SVD, MVSplat360 maintains good multi-view consistency in typical sequence lengths such as 14 frames. It only becomes less satisfactory when the sequence gets significantly longer such as 56 frames in our testing. This shortcoming originates from our component SVD, and it can be addressed by replacing SVD with other released more powerful video diffusion models in the near future.

3. Our rendered novel views maintain reasonable multi-view consistency as verified by the SoTA reconstruction model: we run structure-from-motion on our rendered novel views, and the results confirm that they are multi-view consistent (see Fig. III in the one-page PDF).

 

Q2: Ablation of the post-processing operation.

A2: Below is the comparison on DL3DV regarding with and without (w/o) post-processing operation, using the same setting as Tab.3.

As reported, the post-processing (histogram matching) helps align the output color space with the input one, hence relieving the oversaturated issues caused by SVD and improving the quantitative performance. The improvements are more obvious in those image space metrics (PSNR and SSIM) since other metrics are measured in the feature domain and are more robust to image space distortion.

 

Q3: More details of multi-view conditions

A3:

1. The CLIP embedding extracted from the context views (observed viewpoints) mainly aims to provide a high-level semantic understanding of the scene (L193-L195). But more importantly, the diffusion module is also conditioned with the 3DGS rendered features of the novel viewpoints (L197-L199). These features contain pose and texture information, guaranteeing a successful denoising process.

2. For invisible parts, the 3DGS rendered features are filled with background values since GS is a regression model with no generative power. Based on the visible parts, information in these invisible regions can be hallucinated by the diffusion model, which has strong prior knowledge gained from large-scale data. The multi-view consistency in the in-painted/out-painted areas is ensured by the strong prior of the video diffusion model. Related novel views containing invisible regions are provided in Fig. II of the one-page PDF.

 

Q4: The impact of the reconstructed geometry on SVD.

A4: The SVD refinement module is influenced by the rendered features, whose rendering/rasterization process takes as input all Gaussian attributes, including both geometry and texture factors, not solely the SH coefficients. In case the geometry is not accurate, the SVD model can help refine to get better visual results using its powerful prior knowledge.

 

Q5: Unclear illustration of Fig. 5

A5: In the updated version, we will indicate (a) and (b) in Fig. 5.

Dear Reviewer pb1U,

Did we satisfactorily answer your questions? Would you like us to clarify anything further? Feel free to let us know, many thanks.

Best regards,
Authors of #2348

Response to Reviewer RhQM (R2)

 

Q1: Limited contributions: engineering via adding SVD

A1: Please refer to the global response to all reviewers for more detailed discussions of our main contributions.

Dear Reviewer RhQM,

Did we satisfactorily answer your questions? Would you like us to clarify anything further? Feel free to let us know, many thanks.

Best regards,
Authors of #2348

Response to Reviewer 4RiL (R1)

 

Q1: “Benchmarking” in the paper title is inaccurate.

A1: To make the title reflect our contributions more accurately, we decided to change the title to “MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views”.
Our initial intention is to emphasise that MVSplat360 is the first to explore a new setting: 360-degree feed-forward NVS from sparse views for large-scale scenes, and we demonstrate how to benchmark the models’ effectiveness in this setting. Although it differs from the existing two dominated settings (360-degree NVS for objects or limited NVS for scenes) only in source and target view selection, none of the existing approaches can achieve plausible outcomes in this new setting in a feed-forward way (Fig.3, Fig. 6 and the supplementary video).

 

Q2: Discussions of related “concurrent” work

A2: We acknowledge that MVSplat360 is built on top of existing components, similar to several related works (L120, L191-192), but the key focus of this work is to verify that MVSplat360 renders remarkably higher-quality images in an unexplored new setting: 360-degree feed-forward NVS from sparse views for large-scale scenes.
Existing approaches, such as ZeroNVS, Reconfusion and CAT3D, still need to train a NeRF with the per-scene optimization approach (L61). GeNVS is tailored for object-centric scenes. LatentSplat fails to render plausible views in the new setting (Fig. 3 and Fig. 6), and its GAN-based decoder easily leads to model collapsing (L248). We will add detailed discussions in the updated version and remove the term “concurrent work” to avoid unintentional misunderstandings.

 

Q3: More details of MVSplat360

A3:

• Training objectives: Similar to latentSplat, our MVSplat360 contains two objective functions: reconstruction loss and diffusion loss. We modify the MVSplat backbone to render 1) images and 2) features. 1) The images are supervised by the reconstruction loss (L1 and LPIPS loss) against the ground truth RGB images. 2) The features are used as conditions in the following SVD module supervised by the diffusion loss (MSE in latent space as detailed in L185-188). We will detail the training objective functions in the updated version, and will also release our code to ensure better understanding.

• SD v.s. SVD: In our experiments, we observe that refining using the single-view diffusion model leads to inconsistency among different novel views, which is similar to the findings of Reconfusion (see Fig. 4 in Reconfusion’s paper). On the other hand, we do not want to train additional 3D NeRF/GS for each scene (like Reconfusion did), which is expensive and fussy. To achieve consistent novel view synthesis without additional per-scene training, we therefore opt for the video diffusion model. The integration is achieved by channel-wise concatenating a sequence of the rendered features with the sampled noise (L196-L198).

 

Q4: The approach is incremental.

A4: Kindly refer to the global response to all reviewers for more discussions regarding our contributions.

 

Q5: Conclusion of “benchmarking”: ablations on reconstruction and refinement modules

A5: Regarding the unintentional misuse of the term “benchmarking,” kindly refer to A1.

1. We provide comparisons with the “MVSplat Encoder + latentSplat GAN Decoder” as suggested by R4Q4, which further confirms the superiority of our design.
2. We refer the readers to MVSplat’s paper for the ablations of backbone feature extractors and depth estimation. The ablations of the refinement module are provided in Tab. 3.

 

Q6: Training objectives of the 3DGS reconstruction module

A6: Kindly refer to "Training objectives" in A3.

 

Q7: More details of the refinement module SVD.

A7:

1. The SVD module can, by default, generate an arbitrary number of views. In our experiments, during training, we fixed the target rendering view number to 14 to better align with the SVD pre-trained model (L502-L503). At inference, we feed 56 views to the SVD module and change all related temporal attention blocks to local attention with a window size of 14 to better align with our training.
2. Our trained refinement module is agnostic to temporal permutations. The main reason is that we condition the diffusion model with features rendered by the 3DGS from arbitrary viewpoints, while related work such as SV3D and CAT3D conditions on camera trajectory, leading to their requirements of meticulously designed order (L135-L140).
3. 3D consistency implies that a) our model is 3D-aware due to the usage of 3DGS; b) the rendered novel views are multi-view consistent thanks to the video diffusion module. In addition, we run structure-from-motion on our refined novel views, whose results further confirmed that our outputs are 3D consistent (see Fig. III in the one-page PDF).

 

Q8: Inter- and extrapolation experiments on RE10K.

A8: Kindly refer to R4Q6.

Dear Reviewer 4RiL,

Did we satisfactorily answer your questions? Would you like us to clarify anything further? Feel free to let us know, many thanks.

Best regards,
Authors of #2348

(see previous comments for more response, thanks.)

   

A7: More details of the refinement module SVD

Temporal convolution

We do not apply similar operations to temporal convolution. We observe that applying temporal attention to all 56 frames leads to oversmoothness, so we change it to local attention with a window size of 14. In contrast, convolution is operated in local regions and has no such issues.

Discussions with CAT3D

Sorry for the unintentional confusion regarding “permutation.” Our initial intention was to emphasize that our refinement module takes unconstrained camera trajectories. We assume that the trajectory is plausible, as detailed in the above clarification regarding A4 (Test time trajectories), but we do not require it to be a meticulous design such as an orbital trajectory.

In contrast, CAT3D does assume that its camera trajectory should meet several strict requirements, as detailed in its paper Sec. 3.2 and Appendix C.

 

Feel free to let us know if you would like us to clarify anything further. Many thanks.

We are grateful to see the follow-up comments. We address the additional concerns below. Feel free to let us know if you would like us to clarify anything further.

A2: Discussions of related “concurrent” work

Description of GeNVS

We agree that the Matterport3D dataset used by GeNVS is similar to RealEstate10K, and we will correct its description to “GeNVS mainly works on 360-degree object-centric scenes or nearby viewpoint scene-level datasets”. Nonetheless, it does not alter the previous conclusion as it still belongs to the two existing scenarios summarized in L28-L30 and Fig. I of the one-page PDF. The key contribution of MVSplat360 is that it focuses on an unexplored setting: feed-forward 360-degree scene synthesis. The majority of experiments are conducted on DL3DV (rather than RealEstate10K), which significantly verifies MVSplat360’s effectiveness in handling 360-degree NVS from sparse views.

Discussions with ReconFusion

• We refer the reviewer to the project page of ReconFusion for the results from the diffusion model. In particular, as shown in the last video entitled “ReconFusion distils a consistent 3D model from inconsistent samples” on ReconFusion’s project page, views sampled purely from its diffusion module are far from consistent, showing obvious jittering from frame to frame. Our supplementary video shows that MVSplat360 renders multi-view novel views with much higher consistency.
• Note that those inconsistent demos provided by ReconFusion mainly consist of forward-facing or constraint orbital camera trajectories, while our consistent demo contains way more challenging rendering trajectories, including different types of unconstrained trajectories.
• It is necessary for ReconFusion to apply per-scene optimization in order to get satisfactory consistent novel views, while our MVSplat360 renders 3D consistent novel views in a feed-forward manner thanks to our effective design summarised in “Contributions of MVSplat360 (Method)” in the global response to all reviewers.

 

A3: Training objectives

We will update this potentially ambiguous claim regarding "enhancing geometry" in the paper.

• It is correct that the reconstruction loss is the only source of gradients for the structural parameters of the Gaussians.
• Our initial intention is that joint training can help “enhance the backbone features”. Since the features and other Gaussian parameters are from different heads but share the same backbone, we assume that enhancing the backbone feature will lead to a better reconstruction module and, hence better reconstructed coarse geometry. To avoid unintentional overclaim, we will tone down L168 to “the SVD loss can further optimize the Gaussian features, further enhancing the reconstruction backbone”.

 

A4: SD vs SVD

Ablation of SD

In the updated version, we will make it clearer that we denoise all novel views jointly. We will also add visual comparisons with the SD-based ablation model, which shows inconsistent novel views compared to our default design, as observed in our experiments.

When novel views are far away from each other

We did not observe obvious limitations regarding this issue. This is probably because although the viewpoints range from 180 to 360 degrees for each scene in DL3DV, it still belongs to one scene and might not contain significant “far away” novel views.

Test time trajectories

• In our experiments, the set of novel views is always a plausible trajectory. In particular, for quantitative evaluation, we use the camera trajectories captured by the initial video. For the video demo, we apply a Gaussian filter to the initial captured camera path to obtain 6DoF stabilization results.
• Each trajectory contains 56 frames, which are uniformly sampled from the initial video that contains around 300 frames.

Cross-view attention vs. concatenation

In typical single-view or multi-view diffusion-based NVS models, their input features come from the source/observed viewpoints. In this case, they are required to align/correct those source viewpoint features to match with the novel view cameras, hence it might be better to achieve via cross-view attention. In contrast, in MVSplat360, the features provided to the SVD are those rendered from target/novel viewpoints, containing coarse but correct geometry information for the target viewpoint, hence it is reasonable to achieve via concatenation.

It might be helpful to add additional source/observed view features to the SVD refinement module via cross-view attention, keeping the current concatenation for target/novel views. We will explore this strategy in our further experiments. Thanks for the insightful suggestion!

   

(see next comments for more response, thanks.)