Animate3D: Animating Any 3D Model with Multi-view Video Diffusion

W1: Pipeline without significant innovation.

Thanks. Our work enjoys good novelty in both task formulation and solution pipeline. Firstly, we redefine the concept of 4D generation by introducing a novel task: animating any off-the-shelf 3D objects. This innovative task holds significant relevance in fields with limited exploration (Lines 30-41). Given the rapid advancements in 3D generation, our proposed task is both practical and meaningful to directly drive these high-quality 3D assets. Although some SDS-based methods have been investigated in previous 4D works, the community is still in need of a fast, feed-forward model specifically designed to animate 3D objects. Furthermore, such a model has the potential to facilitate mesh generation with PBR materials, which could enhance commercial applications in software platforms like Rodin-Gen1 and Meshy. This positions our work as a crucial step forward in the evolution of 4D generation technology.

As for the novelty of pipeline, the main goal of the proposed pipeline is to learn accurate motions of static Gaussians. Note that unlike previous text-video-4D works, e.g., DG4D, 4DGen, STAG4D, we don't update any gaussian points and don't include gaussian densification/pruning. This is because only in this way, we could learn the accurate trajectory of each points in static gaussian and when initializing static gaussian by the vertices of the mesh, we can achieve mesh animation, as shown in PDF attached (Figure 6). Once we could animate mesh, 4D generation could benefit from high-quality 3D assets generated by commercial tools.

W2: Concern about the motion diversity.

Please refer to global response (3. Motion Diversity and Dataset).

W3: Concern about motion controllability.

Admittedly, text prompt could not achieve precise control. We'll include more controls such as monocular video in the future.

W4: Lack of experimental result CLIP-I; provide more views and moments for qualitative comparison.

Please refer to global response (1. Comparison methods) for CLIP-I metric. Please refer to the attached PDF for more visualization results.

Q1: Time cost of reconstruction and 4D-SDS.

For 8-frame version model, the Reconstruction and 4D-SDS both costs around 15 minutes, totaling 30 minutes (Line 242). For 16-frame, they both takes around 20 minutes.

W1: Data copyrights and licensing

We confirm that all models downloaded from Sketchfab have a distributable Creative Commons license and were obtained using Sketchfab’s public API. Besides, models marked as ``NoAI'' and restricted due to objectionable or adult thematic content were excluded from the dataset. We provide a detailed elaboration of the licensing information as below.

W2: Clarity of the details of our method.

Thanks very much for careful reading. Sorry for the confusing presentation. We will revise the figures and the text in the updated version.

W3: Discussion with respect to 4D generation methods

Benefiting from the unified spatiotemporal consistent supervision of our 4D foundation model (MV-VDM), our approach can leverage the multi-view attributes of 3D objects to achieve more consistent 4D generation. In contrast, existing methods based on separated text or single-view diffusion models and video diffusion models struggle to preserve spatiotemporal consistency.

More importantly, they also fail to faithfully maintain the multi-view attributes of an existed 3D object.

Due to limited space，we would like to discuss this further in the discussion phase.

W4: More components ablations.

We provide the ablation for MV2V-Adapter, alpha blender and cross-attn layer (image) as follows. Note that we don't ablate the cross-attn layer (text) since it is a necessary component of MVDream and we freeze MVDream in our pipeline. The results indicate that all components are effective.

W5: The outputs are only 8 frames

Thanks for this good point. To validate the scalability of our MV-VDM, we have trained a 16-frame version and reported the quantitative and qualitative results in global response (1. Comparison methods) and Figure 8 in the attached PDF, respectively. We find that 16-frame model generates larger amplitude motion while maintaining similar identity preservation ability as the 8-frame version.

W6: Faithfulness to the text prompt, similar animations

For the concerning about faithfulness of prompts, we clarify that our MV-VDM is capable of producing generations well-aligned with the prompts. First, our MV-Video dataset is designed to encompass a wide variety of motion prompts, as illustrated in the word cloud presented in the rebuttal PDF (Figure 4). Second, for the same object, our model can synthesize various motions according to different prompts as verified in the rebuttal PDF (Figure 7).

However, we acknowledge that achieving extremely precise motion control through detailed prompts presents certain challenges. This limitation is primarily due to the constraints of the CLIP model, which serves as the text encoder for MV-VDM. It would be interesting future work to further develop more powerful 4D models for superior text alignment.

Please also refer to our global response (3. Motion Diversity and Dataset) for more discussions about motion diversity.

Actually, we have omitted some text prompts to make the video more clear. Most objects in the supplementary video are from our test set and the prompts are listed in Appendix (Figure 9). Text prompts for reconstructed objects presented in the video are listed in Appendix (Figure 6). Below we list the rest text prompts:

• A glowing blue butterfly is flying.
• A cute cartoon dog is dancing.
• A monster dog is walking.
• A cartoon bear wearing a swimming goggles is getting ready to dive.
• A cartoon frog is jumpping up and down.
• A panda with very adorable features is dancing.
• A cartoon sea lion, adorned with an expressive and charmingly animated face, is singing.
• A eagle in cartoon style is flapping its wings.
• A gaint panda in cartoon is walking.
• A cool spiderman is dancing.
• A cute lemur is dancing.

W7: Issues about typos.

Thanks, we appreciate your careful review. We will correct it in the revised paper and check if there remains any typos.

Q1: In-depth discussion between 3D animation via multi-view video diffusion and two-stage text-to-4D approaches

Please refer to W3.

Q2: empirical comparisons to additional methods (e.g., Dream-in-4D)

Please refer to global response (1. Comparison methods)

Q3: Ablation of MV2V-Adapter

Please refer to W4

Q4: Ablation: Cross-attention layers for image-condition and text prompt

Please refer to W4.

Q5: The use of 4D-SDS for fine-level animation is counter-intuitive

The role of 4D-SDS is to alleviate small floaters, similar to smoothing but without the blurry effect. Our coarse reconstruction is sparse-view reconstruction, i.e., we only have 4 views, and the reconstruction results are inevitably not perfect in novel views, especially when the number of Gaussian points is large. Similar SDS techniques are adopted in sparse-view 3D reconstruction works, such as ReconFusion[CVPR2023] and Sparse3D[CVPR2023]. We also provide detailed qualitative comparisons in the PDF attached (Figure 2).

Q6: Issues about the diversity in the generated motions.

Please refer to the global response (3. Motion Diversity and Dataset).

Thank you for the detailed rebuttal. Most of my concerns have been addressed and I intend to keep a positive rating. I appreciate the extended discussion regarding existing methods and I would suggest integrating it into the main paper or the appendix.

I also appreciate the authors' efforts in providing additional experiments and comparisons during the rebuttal phase.

One concern that remains is still about the writing and presentation, which I hope the authors will improve substantially in their revision.

W1: Missing Reference: Animate124

Sorry for missing Animate124, which is a great pioneering work. Comparison is in the global response (1. Comparison methods), required by other reviewers. Reference will be added in our revised paper.

W2: More comparisons with 4DGen, TC4D and original 4Dfy.

Thanks for your suggestion. Please refer to the global response(1. Comparison methods)

Q1: 4Dfy results is strange

Thanks for this point. Actually, 4Dfy is not designed for animating off-the-shelf 3D objects, instead, it is designed for text-to-4D (the task in TC4D/4Gen). So it doesn't fit our task and presents not so good results. Technically, we follow the default training setting in 4Dfy's official implementation for animating off-the-shelf static Gaussian/NeRF. Notably, that implementation reduces the learning rate of static Gaussian/NeRF to a relatively low value in the dynamic stage in hope that high-quality appearance would be preserved. Admittedly, when the static object is generated by image/3D SDS loss and the exact image/3D SDS loss is continually used in the dynamic stage, the high-quality appearance could be preserved, as shown in previous works. However, the image/3D SDS loss doesn't match the off-the-shelf 3D object in our task, so using it in the dynamic stage as the original paper results in appearance changes. As the learning rate for static NeRF is very low, the appearance change doesn't finish at the end of the training, so the results are strange. We tried to use a higher learning rate, but found that resulted in very low I2V values, as the object appearance was changed completely.

Besides, we found 3DGS sometimes fail to generate good results when supervised by MVDream SDS loss. This problem was discussed by other researchers in issue section of threestudio-3dgs repo.

Q2: Ablation of MV2V-Adapter

Please refer to our global response (2. Ablation of MV2V-Adapter)

W1: Inappropriate title: 3D objects instead of 3D models

Thanks for the advice, we will consider revising it in the revised version.

W2: Effectiveness of the proposed dataset in video or 3D diffusion models

Given limited time, we only finetune SVD on a subset (20%) of our dataset, and glad to see some improvements. For evaluation, we evaluate the finetuned model on the validation dataset used for the ablation of MV-VDM.

We also use the generated video from finetuned model as the input for DG4D in 4D generation and report the results as below.

Please note that the decrease in Dy. Deg. is because SVD w/o fintuning leads to artifacts and noise in the generated input video, hence leading to the failure of 4D generation. We believe that further improvements could be achieved by fintuning it on the full training dataset.

W3: Motion Diversity: object performing different actions.

Thanks for your insightful question. Please refer to our global response (3. Motion Diversity and Dataset).

W4: About the picture quality.

Actually, the picture quality is mainly affected by the quality of static 3D generated/reconstructed object, since our method is good at preserving the appearance of the given object. We should clarify that the reconstruction quality is not the primary contribution of this paper, which is potentially influenced by the hyper-parameter adjustment of GRM (or other reconsturction method). Besides, The compression of images in PDF can lead to a certain degree of loss in image quality.

We provide high-quality images in Figure 6-8 in PDF attached. Although they do not have the same quality as the original renderings at 1024$\times$1024 resolution, they are better than those provided before.

W5: Better 4DGS algorithms will make 4D-SDS unnecessary

Though MV-VDM generates spatiotemporally coherent multi-view videos as the ground truth for reconstruction, the ground truth has only 4 views. Existing 4DGS algorithms cannot straightforwardly address this sparse multi-view video reconstruction.

Following your suggestion, we have experimented with an improved 4DGS reconstruction algorithm [2]. The results are reported in the table below. Better 4DGS[2] doesn't improve the reconstruction results primarily because the task here involves sparse multi-view video reconstruction using only 4 views. Better 4DGS[2] is not designed to tackle such scenarios. In our work, we apply arap loss to 4DGS[1] to effectively handle sparse views. We fail to apply arap loss to Better 4DGS[2] given that arap loss is designed to constrain the motion of the 3D Gaussians, however, Better 4DGS[2] doesn't learn the motion (It regards the time dimension as a property of Gaussian points, similar to scale/rotation/opacity). We find the proposed 4D-SDS improves the results of Better 4DGS. We will add this discussion in our revised paper.

[1] Wu, Guanjun, et al. "4d gaussian splatting for real-time dynamic scene rendering." (CVPR2024)

[2] Yang, Zeyu, et al. "Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting." (ICLR2024)

Q1: Validate the effectiveness of the proposed dataset.

Please refer to the response of W2.

Q2: Ablation of MV2V-adapter

Please refer to our global response (2. Ablation of MV2V-Adapter)

Q3: The effectiveness of better 4DGS algorithms.

Please refer to the response of W5.

Q4: Comparison with existing single-view video diffusion models

We conducted some empirical comparisons between our MV-VDM and SVD, finding that our model outperforms SVD when animating 3D object renderings, as we specifically trained our model in this domain. However, we struggled to animate realistic images with complicated backgrounds, which is an area where SVD excels. It’s important to note that the comparison is made between the multi-view video results of our model and the single-view video results of SVD, which may be an unfair assessment for our model. For quantitative evaluation, we tested SVD on the dataset used for the ablation study of MV-VDM, and the results are reported in the table below.

Q5: Issue about the basketball interaction in Fig. 4

Yes, it has the interaction. Please refer to the Figure 3 in our attached PDF.

W1: (1) Issues about unfair comparison: 4Dfy and DG4D do not leverage multi-view images; (2) 4Dfy is based on nerf instead of 4DGS; (3) Add comparison with AYG and Animate124.

(1): We proposed a new task of animating any off-the-shelf 3D object, and there is no previous work specially designed for this task to compare with, so we could only chose two representative 4D generation works to serve as the comparison baselines. Due to the lack of unified spatiotemporal consistent supervision, existing works (e.g., 4Dfy, DG4D) can only focus on using text or single-view image conditioned diffusion models for motion modeling. Experiments have shown that this often leads to spatiotemporally inconsistent 4D generation results. Our Animate3D is the first 4D generation framework to animate any 3D objects with detailed multi-view conditions. We believe this workflow is more suitable for generating spatiotemporal consistent 4D objects by leveraging advanced static 3D object reconstruction and generation literature.

(2) and (3) Thanks for the suggestion. Please refer to our global response (1. Comparison methods)

W2: Motivation and effectiveness of the proposed spatiotemporal attention block.

To enhance the spatial and temporal consistency of our MV-VDM, we design a new spatiotemporal attention module, which mainly contains a temporal attention branch and a spatial attention branch. This block is build upon the motion module of video diffusion model to inherit the temporal prior. In the early experiments, we found that simply adopting the temporal attention branch is not enough. This is because the features are likely to lose multi-view consistency after being processed by the temporal branch. Therefore, we add a parallel spatial attention branch to address this issue. The effectiveness is validated in Tab. 3(a) and Fig. 4 in our original paper. Note that w/o S.T. Attn means we replace the proposed spatiotemporal block with motion modules in video diffusion models. We will add this details in our revision.

W3: The influence of the Alpha Blender.

In our spatiotemporal attention block, we employ an alpha blender layer with a learnable weight to fuse the features from both temporal and spatial branches. In the following table, we show the influence of the alpha blender layer and the results validate that it can enhance the spatiotemporal consistency of the 4D generation results.

W4: Implementation details: trainable modules in spatiotemporal attention block.

Yes, we train all modules in our proposed spatiotemporal attention block, as illustrated in Figure 2 in our paper. Thanks for your reminder, we will add these details in implementation section in our revised version.

W5: Implementation details: Alpha blender

As illustrated in Eq. (1) in our paper, we perform alpha blender with a learnable weight $\mu$, which is implemented via nn.Parameter. We will add this detail in the revised version.

W6: Implementation details: Temporal encoding

Thanks for your reminder, we adopt sinusoid positional encoding in AnimateDiff as temporal encoding. We will add these details in our revised version.

W7: Implementation details: number of views of each aniamted 3D object in our dataset.

Please refer to the Appendix (C.1 Rendering Details), 16 views are evenly sampled in terms of azimuth, starting from values randomly selected between $-11.25^{\circ}$ and $11.25^{\circ}$. The elevation angle is randomly sampled within the range of $0^{\circ}$ to $30^{\circ}$.

W8: Training dataset size is limited, is it capable of animating any 3D object?

Adimittedly, our dataset is much smaller than 2D video datasets, but since we inherit the prior knowledge learned by 3D and video diffusion models, our model can animate most dynamic 3D objects commonly seen in daily life. We have tested various categories of objects, including humans, mammals, reptiles, birds, insects, vehicles, weapons, etc, and obtained good results. We provide more examples in the PDF attached. Please also refer to our global response (3. Motion Diversity and Dataset) to see our discussion about the dataset.

W9: Implementation details: camera parameters consistent with MVDream?

Yes, the camera parameters in our multi-view video dataset are processed to ensure that they are consistent with the prior knowledge learned in the MVDream. Please refer to the Appendix (C.1 Rendering Details) in our paper, 16 views are evenly sampled in terms of azimuth. Specifically, during training, we randomly sample four orthogonal views to form a multi-view video for each animation, which is consistent with MVDream's camera setting.