4Diffusion: Multi-view Video Diffusion Model for 4D Generation

We sincerely thank you for providing insightful comments. Below, we address your constructive comments individually.

W1: The paper's novelty wouldn't be its biggest strength, but training a multiview-video module is a good direction so, this point is moderately pass the bar of NeurIPS.

Thank you for your recognition. Our paper primarily focuses on generating 4D content using our multi-view video diffusion model. Although the multi-view video diffusion model is an important contribution of our method, it can only generate four orthogonal viewpoints at a time. This restriction makes it challenging to produce consistent videos from any novel viewpoint, which is essential for 4D generation. Therefore, the 4D-aware SDS loss and anchor loss are also important for our model, which can enhance the spatial and temporal consistency of the generated content, allowing for effective rendering from any novel viewpoint across the temporal dimension.

W2: The results look temporally inconsistent (color flickering), see the frog man's eye and wolf (w/ rider) 's tail. This might due to insufficient training samples.

While we acknowledge that there is room for improvement in our results, our method still outperforms the baselines, demonstrating its effectiveness. Additionally, we have conducted further experiments to assess our method more thoroughly. We select 5 multi-view videos that are not included in the training data as test data. The following table presents the results, where metrics such as FVD, LPIPS, and PSNR clearly demonstrate that our method significantly outperforms the baselines.

Here, we use ground truth videos for novel viewpoints to compute these metrics. As illustrated in the first two rows of Fig.R3 in the attached PDF, Consistent4D and DreamGaussian4D encounter the multi-face problem while our method generates spatial-temporally consistent contents.

We believe that curating more high-quality multi-view video datasets for training our multi-view video diffusion model could further enhance performance. However, there were very few high-quality multi-view video datasets available at the time of our project. We made every effort to curate as many datasets as possible to train our model.

W3: As another nerf based model, the results are not much better than consistent4D.

This weakness is related to W2. We have conducted further experiments to evaluate our method. As shown in the table above and Fig. R3 in the attached PDF, our method significantly outperforms the Consistent4D and other baselines. As illustrated in the first two rows of Fig.R3 in the attached PDF, Consistent4D and DreamGaussian4D encounter the multi-face problem. Consequently, when computing metrics between the input video and synthesized videos, as shown in Tab.1 of the main paper, the performance gap between our method and the baselines may not be apparent.

Q1: The improvement from this multi-view generation model is limited, the colors are still flickering, especially from the generated multview frames.

This question is related to W2. Please refer to W2 for detailed response.

Q2: I wonder if the limited consistency is due to insufficient amount of training data. The paper hasn't show the full potential of the multi-view video for 4D generation, sourcing more synthetic data is important to improve the results.

We believe that curating more high-quality multi-view video datasets for training our multi-view video diffusion model could indeed enhance performance. However, there were very few high-quality multi-view video datasets available at the time of our project. We made every effort to curate as many datasets as possible to train our model. Moreover, we believe that leveraging more robust and powerful 4D representations could further improve results. We plan to explore these enhancements in future work.

We sincerely thank you for providing insightful suggestions. Below, we address your constructive comments individually.

W1: The proposed method requires 12 hours on A100, which is significantly longer than baselines.

Our method focuses on generating high-quality, spatial-temporally consistent 4D content. Therefore, we use dynamic NeRF, which is compact and high-capacity. However, it involves time-consuming volume rendering that extends the optimization time compared to other methods. Although 4DGS can significantly reduce optimization time, it may produce blurred appearances and inaccurate geometries due to the explicit characteristics of Gaussians, as illustrated in Fig.4.

W2: The quantitative evaluation design.

Sorry for the confusion. Here, we follow the setting of Consistent4D. For each generated 4D content, it is challenging to accurately assess video quality when computing FVD with ground truth videos rendered from Objaverse due to the potential distribution gap. Therefore, we use the input video as the reference video to compute the FVD, which provides a more precise evaluation for each test case.

W3: The proposed anchor loss seems very similar to the ones proposed in 4DGen[62] and is not properly discussed in L200.

Thanks for your important comments. Our anchor loss differs from 4DGen in several key aspects:

• Model and Consistency: 4DGen employs a 3D-aware diffusion model, SyncDreamer, to generate multi-view images for each frame of the input video. This approach may lead to temporal inconsistencies. In contrast, our method uses a multi-view video diffusion model to produce videos with improved spatial-temporal consistency, resulting in better overall performance for 4D generation.
• Viewpoint Selection: 4DGen utilizes all viewpoints generated by SyncDreamer to supervise the optimization process. However, viewpoints that are far from the input video may have lower quality, potentially degrading performance. To mitigate this issue, we select the viewpoint closest to the input video as the anchor. This strategy ensures that the anchor video maintains high quality and consistency as illustrated in Fig.R2 in the attached PDF.

We will discuss this in our revised version.

W4: What might be the reason for these artifacts in Fig.5? Is this because of SDS loss?

Figure 5 presents the multi-view video generation results from 4DM, where DDIM was used for sampling. It is important to note that SDS loss was not applied during the sampling process, so the observed artifacts are not due to SDS loss. Despite these artifacts, our approach achieves results that are comparable to, and even better than, those obtained with ImageDream as shown in Tab.2. We believe these artifacts may be attributable to limitations in the base model, which also produces results with transparent edges. Using a stronger base model could potentially enhance performance.

W5: Main concern: Test set construction.

Sorry for making these unclear. During the training of our multi-view video diffusion model, we ensure that our training data does not overlap with the test data used for evaluating our method.

Q1: How many objects were there before filtering out the 966 objects?

Thank you for your appreciation. Before filtering, the dataset contained a total of 44,000 objects, which represents the number of animated shapes from Objaverse 1.0.

Q2: How are the input videos for evaluation constructed? Are they ground truth videos from the Objaverse dataset?

Specifically, for experiments in Section 4.1, We use 3 real-world videos and 3 synthetic videos from the Consistent4D dataset, as well as 3 images from the ImageDream project page. As discussed in Section A.1 of the supplementary materials, for text-image pairs from ImageDream, we utilize SVD to generate monocular videos for 4D generation.

For the qualitative experiments in Sec.4.2, we use the same input videos as in Sec.4.1. For the quantitative experiments in Sec.4.2, we utilize six monocular videos from the Consistent4D test dataset, which provide ground truth for novel viewpoints. While some of these input videos for evaluation may come from the Objaverse dataset, none of them were used in training our multi-view video diffusion model.

Q3: What are the reference videos for calculating the FVD metrics? Are they ground truth videos from the Objaverse dataset? For table 2, are the reference videos the same?

In Tab.1, the reference videos used for calculating the FVD metric are the input videos, as there is no ground truth available for novel viewpoints. In Tab.2, the reference videos for the FVD metric are the ground truth videos that share the same viewpoints as the rendered videos. Although these input videos for evaluation may come from Objaverse dataset, none of them are used for training our multi-view video diffusion model. The reference videos are the same for each method in Tab.2.

Thank you for recognizing and valuing our work. We address your constructive comments as follows:

W1: Difficult to obtain training data.

We have manually curated a high-quality subset of Objaverse, which will released to support community development. Despite data limitations, our method has achieved promising results by leveraging a pre-trained 3D-aware diffusion model. Additionally, as the 4D field continues to advance, we anticipate the emergence of more multi-view video datasets, which have been used in L4GM[1] and Diffusion4D[2].

W2: The evaluation is limited to synthetic, object-centric toy scenes without backgrounds.

Concurrently, most research in 4D generation focuses on creating object-centric scenes without backgrounds. Because maintaining spatial-temporal consistency is challenging. We address this challenge by designing a multi-view video diffusion model for 4D generation, which achieves superior performance compared to other methods. Additionally, our evaluation extends beyond synthetic data. We also evaluated on real-world videos from the Consistent4D dataset, including the squirrel in Fig. 1, the egret and robot in Fig. 4, and the jay in Fig. 8.

W3: Description of test dataset for experiments in Section 4.1.

We conduct experiments in Sec.4.1 by using 3 real-world videos and 3 synthetic videos from the Consistent4D dataset, as well as 3 images from the ImageDream. As discussed in Sec.A.1 of the supplementary materials, for text-image pairs from ImageDream, we utilize SVD to generate input videos. We will add the description in our revised version.

W4: Whether assets in test-videos are unseen during the training of all models.

Sorry for making this unclear. During the training of 4DM, we ensure that our training data does not overlap with the test data. Regarding the pre-trained ImageDream model, the training data has not been released. However, it is known that the model was trained exclusively on multi-view images from Objaverse, which means it has not seen the test videos either.

W5: Evaluation in Sec 4.1 is limited to CLIP and FVD metrics.

Thanks for your constructive comments. As detailed in W3, we utilize monocular videos from Consistent4D and ImageDream to evaluate our method. However, these datasets do not provide ground truth videos for novel viewpoints, preventing us from calculating LPIPS and PSNR metrics. To further evaluate our model, we select 5 multi-view videos that are not included in the training data and conduct experiments. The following table presents the results, where metrics such as FVD, LPIPS, and PSNR clearly demonstrate that our method significantly outperforms the baselines.

Here, we use ground truth videos for novel viewpoints to compute these metrics. As illustrated in the first two rows of Fig.R3 in the attached PDF, Consistent4D and DreamGaussian4D encounter the multi-face problem. Consequently, when computing metrics between the input video and synthesized videos, as shown in Tab.1 of the main paper, the performance gap between our method and the baselines may not be apparent. Due to time constraints, we selected the best of two runs when conducting this experiment.

W6: Minor: Given the small improvement over Consistent4D, further evaluation of statistical significance is needed.

As discussed in W5, we conduct additional experiments to further evaluate our method. As shown in the table above and Fig. R3, our method significantly outperforms Consistent4D.

W7: Minor: The paper would benefit from more precise writing.

Thank you for pointing out the writing problems. We will revise the training details in lines 176-182 and clarify the annotations in Eq.10. Furthermore, we will simplify and refine the method section to enhance readability.

Q1: The paper assumes access to a multi-view video dataset.

This question is related to W1. Please refer to W1 for detailed response.

Q2: The rationale behind the need for SDS.

We utilize SDS to optimize dynamic NeRF, enabling effective rendering from any novel viewpoint across the temporal dimension, which is essential for 4D generation. This is particularly challenging for our multi-view video diffusion model, as it is limited to generating only four orthogonal viewpoints at a time, which hinders its ability to produce consistent videos from any novel viewpoint.

Q3: Why not fit the dynamic NeRF directly on the generated multi-view videos？

Our multi-view video diffusion model can only produce four orthogonal viewpoints at one time. Training with such sparse views often results in overfitting to the training viewpoints, as proved in SPARF[3]. To further demonstrate this challenge, we conducted an experiment where we used only four generated videos to optimize dynamic NeRF. The results, which illustrate the overfitting problem, are presented in Fig.R4 in the attached PDF.

[1]Ren J, et al. L4GM: Large 4D Gaussian Reconstruction Model. arXiv preprint arXiv:2406.10324, 2024.

[2]Liang H, et al. Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models. arXiv preprint arXiv:2405.16645, 2024.

[3]Truong P, et al. SPARF: Neural radiance fields from sparse and noisy poses. CVPR, 2023.

Thank you for appreciating and acknowledging our work. We address your constructive comments below:

W1: Some technical details are unclear.

We will provide detailed information about the motion module in the revised version. Specifically, we incorporate a zero-initialized motion module at the end of each spatial attention block in ImageDream, as illustrated in Fig.3. Each motion module begins with group normalization and a linear projection, followed by two self-attention blocks and one feed-forward block. A final linear projection is then applied, after which the residual hidden feature is added back at the end of each motion module. For a more detailed architecture overview, please refer to Fig. R1 in the attached PDF. Moreover, we will open our code to enable researchers to reproduce our results and further build upon our work after the completion of the anonymous review process.

W2: Overfitting and parameters of the motion modules.

Thank you for your insightful question. Our motion module contains 453.209M parameters. We have evaluated our model on real-world videos from the Consistent4D dataset, including the squirrel in Fig. 1, the egret and robot in Fig. 4, and the jay in Fig. 8, indicating that our model does not overfit the training data. To be more concrete, the reasons are as follows:

• Our training dataset comprises 966 multi-view videos, with each video containing 32 viewpoints. This setup results in approximately 741,000 frames in total. Additionally, the average number of frames per video is 24. During the training iteration of 4DM, we randomly sample 4 orthogonal viewpoints with 8 frames from the training data, this strategy effectively augments our training data, helping to mitigate the risk of overfitting.
• Our 4DM model employs several 3D self-attention layers and 1D temporal self-attention layers to capture spatial-temporal relationships, which requires approximately 6 seconds to optimize per step. As a result, two days of training (30,000 iterations) is insufficient for 4DM to overfit the dataset.
• In 4DM, we only finetune the parameters of the motion module while keeping the parameters of the original ImageDream model frozen, thereby preserving ImageDream's generalization ability, even when trained on a small curated dataset.

W3: Anchor videos.

Thank you for your comments. As the reviewer pointed out, the input video is generally of better quality than the generated videos. Therefore, we select the viewpoint closest to the input video as the anchor. This approach ensures that the anchor video maintains the same quality as the input, which improves the results. Moreover, our multi-view video diffusion model is currently limited to generating multi-view videos with 8 frames. When the input video exceeds 8 frames, we must apply our multi-view video diffusion model multiple times to generate anchor videos. However, this process may lead to temporally inconsistent results due to the stochasticity of the diffusion model, particularly when the viewpoint is far from the input video as shown in Fig.R2 in the attached PDF. This inconsistency would degrade the 4D generation performance.

W4: CLIP-I score.

In Tab.2 in the main paper, our method performs only slightly lower (-0.0146) than ImageDream in the CLIP-I metric, likely due to the stochasticity of the diffusion model. However, our method surpasses ImageDream in the LPIPS metric, which also reflects image quality. Moreover, our method exhibits comparable visual quality to that of ImageDream as shown in Fig.5 in the main paper. To further assess our model, we select 5 additional test cases from Objaverse, alongside the test data provided by Consistent4D (None of these test data are included in our training dataset). To account for the stochasticity of the diffusion model, we conduct five runs for each test case and report the average metrics as follows:

Despite the comparable performance in CLIP-I, our model excels in generating spatial-temporally consistent multi-view videos, a primary focus of our research. This is evidenced by the superior performance on metrics such as LPIPS, CLIP-C, and FVD, which better capture the spatial and temporal fidelity of video content. These metrics demonstrate that our method effectively balances image quality with temporal consistency, making it a robust solution for multi-view video generation.

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper.

We would like to know if our response has adequately addressed your questions and concerns. We are more than willing to engage in further discussion.

Best regards,

Authors