3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors

S1. Will this dataset be open sourced?

Our 3DGS Enhancement dataset is generated based on the publicly available DL3DV dataset. We will provide the complete dataset generation code to ensure the community can quickly reproduce our results and explore new research opportunities.

W1. Related work appears to be thin and some of these paragraphs can be more descriptive.

Thank you for the constructive feedback. We will refine our Related Work section in the camera ready version. Specifically, NVS methods dealing with sparse-view inputs will be categorized under "Few-shot novel view synthesis", while more general enhancement efforts, such as super-resolution and artifact removal, will be categorized under "Radiance fields enhancement". More literature under these two categories will be added. These revisions would make the Related Work more descriptive and organized.

W2. Method relies on adjacent views for continuous interpolation.

Our method performs well when there is a significant overlap between two views. Since we use images rendered from a low-quality 3D Gaussian model as conditions for video diffusion model, even if two views are not close to each other, our method can still achieve decent results when these conditional images contain certain structure or texture information. On the other hand, our method can generate free-trajectory conditional images for continuous interpolation, offering more flexibility in addition to linear interpolation between adjacent views.

W3. Choice for STD should be quantitatively ablated. Individual parts in STD like color correction should be ablated as well.

Thank you for the valuable suggestion, we conducted an ablation study on two STD components, the color correction and the temporal layers, on the DL3DV dataset (9 views). Results are detailed below. Both two STD components contribute to performance improvement, while temporal layers contribute more as they ensure STD produces consistent image outputs to improve final reconstruction performance. We will include the ablation study on STD components in the supplementary material.

Q1. Authors describe in L184-186 "Despite this significant enhancement in the quality of the rendered views, we propose to rely more on the reference views rather than the restored novel views when fine-tuning the 3DGS model, since the 3DGS model is highly sensitive to slight inaccuracies in the restored views." I would appreciate a more thorough discussion on this.

In the 3DGS training process, we observed that the 3DGS model can fit the training images very well, even if the training images contain noises. Unfortunately, images generated by the diffusion model inevitably introduce various noises and inconsistencies, resulting in a performance decrease.

To mitigate the negative impact of these generated images for the 3DGS model, a straightforward approach is assigning larger weights to real images and generated images of higher quality. From our observations, the generated images that are closer to the reference views tend to have higher quality. Motivated by this, we introduce an image-level weighting scheme. Additionally, we observed that smaller 3D Gaussian volumes are more likely to be well-reconstructed. To prevent well-reconstructed regions from being adversely affected by the generated images, we introduce pixel-level confidence scores which are calculated based on the volume of 3D Gaussians. From Table 2 of the paper, we can see the improvement after incorporating image-level confidence and pixel-level confidence. Thank you for the insightful feedback, we will add more discussions in the camera ready version.

Q2. "modified baselines denoted with ∗ taken from [36]". However, no baseline seems to be marked with * in table 1. This is possibly an oversight and the authors should correct it.

Thank you for the helpful comment. This is an oversight, and we will remove it in the camera ready version.

Q3. What's the adversarial loss described in L179 for STD?

The adversarial loss employed in Eq. 6 is the standard adversarial loss, which trains a discriminator $D$ to discriminate between real image $\hat{I^g}$ and fake image $I^g$, as $L_{GAN} = E_{\hat{I^g}} \left[\log D(\hat{I^g})\right] + E_{I^g} \left[1 - \log D(I^g)\right].$ Thank you for the valuable comment, we will add it into the camera ready version.

Q4. On what data is the Temporal Denosing U-Net fine tuned on?

Our Temporal Denosing U-Net is fine-tuned on the 3DGS Enhancement Dataset created in this work, which includes a large number of image pairs generated from corresponding high-quality and low-quality 3DGS models. More details of the dataset can be found in Section 5.1 of the paper.

Suggestions: Figure 2 can be more interpretable if the right part shows the output of the enhanced views (input from left part); currently there is no 1:1 parity

Thank you for the helpful suggestion, we will revise Figure 2 of the paper to ensure its right part shows output images corresponding to the input images.

W1. They can provide more comparisons on ablating the temporal consistency.

As shown in Fig. 1 of the paper, the images rendered by 3DGS model trained on sparse view inputs often contain significant artifacts or blank areas. We observed that a image diffusion model trained solely on these images is unable to restore high quality images without additional information, and the synthesized multi-view images are often inconsistent even for adjacent views. In addition, for the temporal layers in STD, we conducted an ablation study to verify their effectiveness on DL3DV dataset with 9 views, and the result is detailed below. Temporal layers help STD produce more consistent image outputs to improve final reconstruction performance. Thank you for the helpful suggestion, we will provide more ablation studies on the temporal consistency in the supplementary material.

Q1. Video Diffusion prior may have motion information of moving objects in the scene. How do we eliminate that effect in enhancing 3DGS?

We took two steps to mitigate this effect of moving objects in video diffusion prior. First, we created a paired novel view synthesis dataset from DL3DV to fine-tune the video diffusion model from an image-to-video model to a video-to-video model, ensuring that it outputs images more akin to the novel view synthesis of static scenes. Second, we introduced images rendered from a low-quality 3D Gaussian model as the conditional inputs for video diffusion. Using these images as guidance helps the video diffusion model generate images suitable for 3D static scene reconstruction.

L1. The limitation is definitely from the pre-trained video diffusion model whether the data is more like novel view synthesis data (static scene) or a sequence of video frames (moving objects).

Your statement of the limitation of this work is correct. Since our approach is based on video diffusion, it is indeed limited by the performance of the pre-trained video diffusion model (e.g., SVD). However, our method will also benefit from advances in video diffusion models, since better pre-trained video diffusion models or larger datasets can enhance the potential of our approach.

1.The writing of this paper is not very clear, and some typo errors exist in the submission, for instance, line147, it should be ${ I^{ref}_{i-1}, I_1, I_2, ..., I_T, I^{ref}_i }$; line216: it should be $I_c$, consistent to the equation (8).

Thank you for the helpful comment. You are correct, line147 should be ${ I^{ref}_{i-1}, I_1, I_2, ..., I_T, I^{ref}_i }$. In line216, $I_w$ should be $I_c$. We will carefully revise the writing and typos in the camera ready version.

2. Training details are missing in the paper, more details about the video diffusion model and fine-tuning process should be provided.

Our video diffusion model includes a pre-trained VAE to encode an image sequence into a latent sequence and decode the latent sequence back into the image sequence. It also includes a U-Net with learnable temporal layers, which employs cross-frame attention modules and 3D CNN modules to ensure frame-consistent outputs. The input of video diffusion model is a image sequence segment that includes 25 images with different sample steps from the image sequences rendered from the low-quality 3DGS model. The first and the last frames in this segment are replaced with images rendered from the high-quality 3DGS model. During fine-tuning, our video diffusion model is conditioned on these image sequence segments and trained to synthesize the corresponding segments rendered from the high-quality 3DGS model.

Our video diffusion model is fine-tuned with a learning rate of 0.0001, incorporating 500 steps for warm-up, followed by a total of 80,000 training steps. The batch size is set to 1 in each GPU, where each batch consisted of 25 images at 512x512 resolution. To optimize the training process, the Adam optimizer is employed. Additionally, a dropout rate of 0.1 is applied to the conditions between the first and last frames and the training process utilize CFG (classifier-free guidance) to train the diffusion model. The training is conducted on 2 NVIDIA A100-80G GPUs over 3 days. The STD is fine-tuned with a learning rate of 0.0005 and 50,000 training steps. The batch size is set to 1 in each GPU, where each batch consists of 5 images at 512x512 resolution. The fine-tuning process is conducted on 2 NVIDIA A100-80G GPUs in 2 days. Thank you for the valuable feedback, more details of the models and fine-tuning process will be included in the supplementary material.

3. More datasets, such as MipNerf360, should be evaluated to verify the effectiveness and robustness of the proposed method.

Thank you for the constructive feedback, we conducted experiments on the Mip-NeRF 360 dataset, and the results are detailed below and in the attached PDF file in Author Rebuttal section. Our model is trained on the DL3DV dataset, and evaluated on all 9 scenes of Mip-NeRF 360 dataset. Our method significantly outperforms the baselines, although it has not been trained on this complex dataset. This indicates that our method has remarkable cross-dataset generalization capability for unbounded environments, thanks to the powerful 2D video diffusion priors.

W1. Runtime is not discussed in this paper. I’m wondering how long it will take for one scene compared to the other baselines.

As shown in the below table, we estimate the per-scene runtime and rendering FPS of different methods on the DL3DV test set (3 views) with one NVIDIA A100 GPU. Our method takes 24.5 minutes for one scene, including 10.5 minutes for low-quality 3DGS training, 2.0 minutes for SVD inference, and 12.0 minutes for high-quality 3DGS training. Thank you for the helpful suggestion, we will include the efficiency analysis in the supplementary material.

Q1. The experiments are conducted almost on the self-introduced dataset. Can 3DGS-Enhancer also surpasses the other baselines on more commonly used outdoor unbounded datasets (e.g. Mip-NeRF 360 dataset)?

Thank you for the constructive feedback, we conducted experiments on the Mip-NeRF 360 dataset, and the results are detailed below and in the attached PDF file in Author Rebuttal section. Our model is trained on the DL3DV dataset, and evaluated on all 9 scenes of Mip-NeRF 360 dataset. Our method significantly outperforms the baselines, although it has not been trained on this complex dataset. This indicates that our method has remarkable cross-dataset generalization capability for unbounded environments, thanks to the powerful 2D video diffusion priors.