Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models

We thank the reviewer for the positive feedback and insightful comments. Our responses to the reviewer’s concerns are below:

Q1: If the viewpoints differ by only 30 degrees, can they still be considered independent variables?

Yes, although these two views may have some overlapping regions, the conditional independence still holds. Because the case in Figure 3 does not contain some special motions like twisting. The overlap between the future front view $I_{i,j’}$ and the current side view $I_{i’,j}$ also exists in the current front view $I_{i,j}$. In practice, even when twisting or rotation movement occurs, it is often subtle between frames, contributing negligible information gain and thus having no substantial violation on the conditional independence assumption. As demonstrated at in the supplementary video (01:04~01:08), our method is also capable of handling objects with such dynamics.

Q2: More explanation for why VRS works and why it would reduce the variance.

Thanks. We have explained its motivation in Section 3.3 and provide further clarification here.

The similar rollback operations have been adopted in many diffusion-based inpainting methods [1] for harmonizing potential conflicts.

In our work, the utilization of this operation aims to deceive the diffusion model into denoising a noisy latent with a smaller variance at a higher noise level through the following process: before $t$-th denoising step, we add Gaussian noise with a variance smaller than $\sigma_{t+1}^2-\sigma_{t}^2$ onto the latent, and then denoise it again at noise level $t+1$.

After that, at $t$-th step, the noisy latent to be denoised by the model actually resides at a level below $\sigma_t$.

We provide a visualization of how variance evolves during denoising with and without VRS in Figure 10 of the revised appendix.

A more intuitive explanation for why it works is that when the latent resides at a lower noise level, the diffusion model can allocate more capacity to reconcile the conflicts brought by two scores. Additionally, a latent with less noise can provide clearer guidance for denoising, which can also help to mitigate potential conflicts.

Q3: If we set the sampling steps larger, will it achieve a similar effect with variance-reducing sampling?

We would say no.

In the revised supplementary video (00:12~00:21), we provide two videos generated using 500 denoising steps (10x than the default setting) without VRS.

These assets still exhibit severe flickering artifacts.

This may because the unresolved conflicts in the early stage of denoising will drive the latent away from the desired distribution, which cannot be rectified by merely extending the denoising schedule.

[1] Lugmayr, Andreas, et al. Repaint: Inpainting using denoising diffusion probabilistic models. CVPR. 2022.

Thanks for your clarification. With the development of 3D generation and Video generation, they both converge to some standard formulation. Like transformer-based regression 3D-Gen method(Clay, LRM..) and CasualVAE+DiT-based Video Generation method(OpenSoraPlan, CogVideo-X). However, 4D generation is still under development. Though I think composition of both SoTA methods is far from perfect, I would like to see more researchers make their works open-source. I will keep my rating until I see final comments from other reviewers.

Sincerely speaking, I should give a weak accept(6) due to the novelty at present time. But actually, many follow-up works like Vidu4D, SV4D are inspired by this formulation(marrying VSD and Multi-View Denoiser). So I strongly recommend accepting this work as a milestone of 4D generation. (The quality gap lies in the ability of foundation video diffusion model but not the framework.)

We thank the reviewer for the positive feedback and insightful comments. Our responses to the reviewer’s concerns are below:

Q1: The backside seems to be over-smoothed.

Thanks.

To reconcile the scores from the two generative models without significantly impacting runtime, we adopted the relatively aggressive variance-reducing sampling (VRS) strategy, which sacrifices details of some cases in specific views that are under-constraint due to far from the reference frame and view. Our rationale for this trade-off is based on the following considerations:

(i) This compromise does not affect the consistency of the generated results, it is still a seamless 4D assets that is consistent with the reference view.

(ii) It does not degrade the details in views closer to the reference frame and view. Actually, as demonstrated in the video, our results often exhibit better detail in these views compared to other alternates thanks to the consistent direct photometric supervision in high resolution. In some applications, users may more care about quality in these views rather than back views without any hint.

(iii) Finally, this issue can be mitigated by texture-dynamic decoupled reconstruction as shown in the ablations in Figure 6 of the main paper and 00:03~00:11 of the supplementary video.

Besides, this issue could be partly ascribed to the multi-view prior. We observed that for some examples, their multi-view references already lack details in backside as shown in Figure 8. Further scaling of the used diffusion prior may alleviate this issue.

Q2: Whether finetuning the SVD on the object-centric data would be helpful.

Thanks for the insightful comment.

The authors agree with this point, that fine-tuning the SVD on videos containing only foreground objects could improve quality.

The current adopted video generator has learned a broader distribution than the desired single-view videos for dynamic objects, which may introduce some conflicts.

We also raise this concern and propose VRS to mitigate potential conflicts.

However, to preserve the training-free merits of the proposed framework, we have not included further fine-tuning of the video generative model as part of our contribution. We would like to attempt this direction in the future.

Q4: Can the proposed method generalize to other multi-view 3D generation and video generation methods?

Yes, as long as they are diffusion-based generative models and share the same latent space. Our framework adopts an architecture-agnostic approach to combine heterogeneous scores from multi-view and video diffusion models.

Q5: Time consumption for stage-1 and stage-2.

We have already reported the total generation time in Table 2 and indicated the time spent in the first stage at L469 in the initial submission. Now we present them in the table below for more clear illustration.

Q6: Extending to multi objects.

Good question. The multi-view generator used in our framework is trained on object-centric data, limiting its generation to scenes containing a single object. Constrained by this, we limit the scope of our method to the generation of single dynamic objects, similar to prior 4D generation works [1]. Usually, 4D results of multiple objects can be created by composing multiple dynamic objects generated independently. To better answer this question, we additionally provide an example of a composed scene with multiple dynamic objects in the revised supplementary video (01:10~01:18), demonstrating scalability and generalization ability of our approach.

[1] Jiang, Yanqin, et al. Consistent4d: Consistent 360° dynamic object generation from monocular video. ICLR, 2024.

[2] Voleti, Vikram, et al. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. ECCV, 2024.

[3] Tang, Jiaxiang, et al. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. ICLR, 2024.

[4] Ren, Jiawei, et al. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint, 2023.

[5] Li, Zhan, et al. Spacetime gaussian feature splatting for real-time dynamic view synthesis. CVPR. 2024.

We thank the reviewer for the detailed review as well as the suggestions for improvement. Our responses to the reviewer’s concerns are below:

Q1: Comparison with the baseline mentioned in W1.

Great suggestion. In ablation experiments of the main paper and supplementary video, we have compared with a similar baseline that only utilizes SV3D [2] for each frame. To ensure a fair evaluation of the improvements brought by score composition, we also incorporated variance-reducing sampling (VRS) into it. We further provide the quantitative comparison in the table below (also in Table 3 of the revised paper) with the suggested baseline (reconstructing images from only multi-view 3D generator with [5]):

The qualitative results from only the multi-view 3D generation model without VRS are shown in the 00:31~00:35 of revised supplementary video. It can be observed that the generated videos exhibit significant temporal inconsistencies due to the inherent stochasticity in the diffusion denoising process, making it challenging to be reconstructed into meaningful 4D assets, even with state-of-the-art 4D reconstruction methods [5] with superior fitting capability.

Q2: Similar results in two stages.

Thanks for the careful observation. Note that the second stage is a reconstruction stage, the almost identical results are partly ascribed to the Gaussian’s strong fitting ability for the training views. It also corroborates the high consistency of the images generated in the first stage. Besides, the multi-view references of examples in Figure 4 already have textureless back views (as shown in Figure 8 of the revised appendix), so the first-stage results usually do not exhibit significantly more blurry details in these views.

Q3: Quantitative results for ablation studies.

Thanks for the suggestion. We conduct quantitative ablations on the Consistent4D dataset, the results are presented in Table 3 of revised appendix and below:

It aligns with our observations drawn from the qualitative ablations, where our result shines in video-level metric of FID due to its better temporal coherence.

In the submitted version, we following common practice [3,4], only report qualitative ablations, as we believe they are sufficient to demonstrate the characteristics and differences from these baselines. More importantly, there is a lack of comprehensive quantitative metrics to evaluate the quality of generated assets in such tasks. Besides, we modified the evaluation of FVD in the revised paper by aligning the background color. While the comparison between previous metrics can also reflect the severity of temporal artifacts in the foreground object, the new results help to better align the scale with those reported in other 4D generation works.

Dear Reviewer AvRx,

We sincerely appreciate the reviewer's time for reviewing, and we really want to have a further discussion with the reviewer to see if our detailed explanations and additional results solves the concerns. We have addressed all the thoughtful questions raised by the reviewer (eg, results, time consumption, extending to multi objects) and we hope that our work’s contribution and impact are better highlighted with our responses. As the discussion phase is nearing its end, it would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors

Thanks to the reviewers for the time and effort dedicated to thoroughly reviewing our work.

As a 'training-free' or 'plug-in' module, I cannot see the detailed generalization examples of the proposed method to other multi-view 3D generation and video generation methods. This cannot support the claim in the paper.

Thanks for the comments. We first clarify the following facets:

(i) For the 'training-free' property, our method indeed only utilizes the off-the-shelf multi-view image generative model and the video generative model without additional training.

(ii) Although we did not claim the generalization ability implied by "plug-in" as our contribution in the paper for rigorousness, this property is evident in logic, as the proposed pipeline holds no bias on the model architecture. Therefore, we give an affirmative response to Q4.

Considering the above points, we use SV3D [4] as the multi-view 3D generator given its superior performance than other MVDiffusions. It is sufficient to demonstrate the feasibility of our proposed approach. However, to solve the reviewer's concern, we additionally conducted experiments on an alternative multi-view generative model V3D [3] to empirically demonstrate our framework's applicability on other diffusion models. Due to the time constraint, we test this configuration on two cases. The results (v3d.mp4) are included in the supplementary material. Despite the slight Janus problem resulting from the limited capability of the adopted multi-view generative model, the generated 4D assets still exhibit overall spatial-temporal coherence, with significantly better consistency than that of the multi-view generative model-only baseline. This result further demonstrates the potential of our approach to scaling with various foundational diffusion models.

For Q3, the quantitative ablation for the decoupled reconstruction is not provided. I still think that limited qualitative results cannot fully support the effectiveness of different designs.

Actually, there is a lack of comprehensive quantitative metrics to evaluate the quality of generated assets in this task, so we following the widely recognized works [1,2] by providing the qualitative ablations in our initial submission, which can clearly illustrate our advantage against the baselines. In addition, we have also provided comprehensive quantitative ablations for the design of our first stage (which is the core part of the proposed framework) in our last response to Reviewer AvRx's Q3 (Quantitative results for ablation studies). Now we additionally included the ablations on the decoupled reconstruction in Table 3 of the revised appendix. From Table 3 (Appendix), we can observe that our full model (f) outperforms the baseline (g) without decoupled reconstruction.

Moreover, the table results in Q3 are different from those inthe Appendix, please correct them.

Thanks for the eagle eyes! Actually, the results in Q3 and those in the Appendix (Table 3) are essentially the same except for the format of CLIP score. The results in the response of Q3 were reported as percentages omitting the "%", while the numbers in the CLIP score column of Table 3 (Appendix) are presented in the actual decimal form. For example, 94.82 in Q3 and 0.948 in Table 3. We have fix it in the response of Q3.

For L484 and L505, the titles for two paragraphs are same, please correct them.

Thanks for pointing out this typo, we have fix it.

Hope our responses clarify the questions above. The reviewer’s constructive comments have been instrumental in guiding these refinements, and we are sincerely grateful for this guidance.

[1] Tang, Jiaxiang, et al. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. ICLR, 2024.

[2] Ren, Jiawei, et al. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint, 2023.

[3] Chen, Zilong, et al. V3d: Video diffusion models are effective 3d generators. arXiv preprint, 2024.

[4] Voleti, Vikram, et al. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. ECCV, 2024.

Dear Reviewer AvRx,

We sincerely thank the reviewer for the valuable time on reviewing and discussion. As the discussion phase is nearing its end, we wondered if the reviewer might still have any concerns that we could address. After the thorough discussion with the reviewer, we believe our response on more experiments on different multi-view 3D generator, time consumption, extending to multi objects, and typos have addressed all thoughtful questions raised by the reviewer, and hope these results helps the final recommendation. We genuinely appreciate the reviewer’s valuable suggestion for improving our work!

Best wishes,

Authors

We thank the reviewer for the acknowledgment of the innovation of our work as well as the valuable suggestions for improvement. Our responses to the reviewer’s comments are below:

Q1: Requires 8-16 A6000 GPUs for basic operation？

This is a constructive question regarding the application of our Diffusion$^2$. We first clarify the following facets:

1. The use of 8~16 A6000 GPUs for inference is not mandatory. We adopt it as a setting to demonstrate the advantage on maximum inference efficiency benefited from high parallelism, which is more important for the online tasks.
2. As mentioned in L469, multi-GPU is only used in the first stage, which takes less than 10 minutes. In addition, if we sacrifice parallelism and only use a single GPU, our framework is still significantly faster than representative SDS-based methods (e.g. 2x than Consistent4D [1]). We depict this trade-off in Figure 7 of the revised appendix.

Actually, even without considering parallelism, our approach almost constitutes a lower bound on the time required for sampling $V(=21) \times F(=25)$ images with diffusion models to reconstruct 4D asset: we only perform the denoising process twice for each image without iterative refinement. However, sampling such a dense set of multi-view, multi-frame images may be unnecessary for reconstructing 4D representations. So the further reduction in the computational requirements lies in the redundancy of such a dense image array. Now we sampled 21 views for each frame to leverage an existing multi-view generator [2]. In practical applications, fine-tuning this generator to support a smaller $V=8$ (as in concurrent work [4]) may not substantially impact performance but can immediately reduce computational requirements by 2/3.

Besides, in real applications, some efforts on engineering optimizations can also be made to mitigate the efficiency bottleneck introduced by multi-GPU synchronization and communication revealed in Figure 7 of the revised appendix.

Q2: Further exploration of challenging cases.

Great suggestion. The examples we currently use are primarily sourced from commonly used public 4D generation datasets [1,3]. Among them, the example sighing frog (shown in Figure 4 and supplementary video) retains the largest motion within the Consistent4D dataset, thus posing significant challenges for many prior methods. Both SDS-based and feature blending-based approaches struggle to establish the association between frames with substantial differences. In contrast, our proposed method demonstrates the potential to handle complex motions with global receptive fields of two score estimators.

For 4D objects with rotational movements or articulated motions which pose distinct challenges, We show additional examples in the revised supplementary video (01:04~01:08) to provide more insights and demonstrate the effectiveness of our approach.

Q3: Extending to more diverse scenarios.

The capability of the proposed framework is mainly constrained by the foundational diffusion model. Since the multi-view generator [2] used in this work focuses on object-centric data, we limit our exploration to the generation of single dynamic objects, following existing works in this field [1,3,4].

For more diverse 4D scenarios, as long as the two diffusion models share a same latent space to model their marginal distributions, a similar sampling approach can be applied.

[1] Jiang, Yanqin, et al. Consistent4d: Consistent 360° dynamic object generation from monocular video. ICLR, 2024.

[2] Voleti, Vikram, et al. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. ECCV, 2024.

[3] Zhao, Yuyang, et al. Animate124: Animating one image to 4d dynamic scene. arXiv preprint, 2023.

[4] Xie, Yiming, et al. Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency. arXiv preprint, 2024.

Dear Reviewer s4pt,

We sincerely appreciate the reviewer's time for reviewing, and we really want to have a further discussion with the reviewer to see if our detailed explanations and additional results solve the concerns. We have addressed all the thoughtful questions raised by the reviewer (eg, computational requirements, extending to more challenging cases) and we hope that our work’s contribution and impact are better highlighted with our responses. As the discussion phase is nearing its end, it would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors