Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels

We sincerely thank you for your valuable comments and insightful suggestions. We address all your comments below. If our response has addressed the concerns, we will highly appreciate it if the reviewer considers raising the score.

1. Foundation model: Our foundation model is Vidu. We believe the qualitative results are largely attributed to DGS. We compare the results of other dynamic NeRF and 3DGS in Fig. 4, and our DGS demonstrates state-of-the-art performance.

2. Open source video foundation models: For comparison with open-source video generators, we found that generating plausible videos using our specific prompts is challenging for these open-source tools. Instead, we use ToonCrafter to interpolate two frames from our generated video for a fair comparison. It can be observed that, although the generated video lacks consistency, our reconstruction remains relatively stable. Please refer to Fig. I.

3. More results: Please refer to Fig. J for results with more styles and categories.

4. Runtime: As discussed in Sec. 4.1, for each reconstruction, the overall training takes over 1 hour on an A800 GPU. Specifically, generating a text-guided video takes 4 minutes, preprocessing takes 10 minutes, initialization takes 15 minutes, and the DGS stage takes about 30 minutes.

5. Challenges in 4D reconstruction: Please refer to the Motion regularization part in the common response for our discussion about occlusions. For lighting changes, it is promising to introduce a time-dependent spherical harmonic function for Gaussian surfels. To handle complex scene dynamics, such as fluids or flames, adding more control bones or incorporating a dense MLP motion field should be an effective approach.

6. Benchmark: Please refer to Q1 in the common response for more quantitative benchmarks.

Thank you again for your time and effort in reviewing our work and providing the constructive comments. Please feel free to let us know if you have any further questions by August 13 AoE, we are more than happy to address them.

Thanks for the efforts of the authors for solving some my concerns. I will keep my initial positive rating.

We sincerely thank you for your valuable comments and insightful suggestions. We address all your comments below.

1. Anonymous link: Thank you for your feedback. We believe the issue with the anonymous link might be due to a temporary network problem and we appreciate it if you could try again. Other reviewers might have successfully accessed the videos through the link. If the link remains inaccessible, please refer to Fig. 11-12 in the supplementary and Fig. J for more results.

2. Annotation and presentation: Thank you for your valuable advice. We acknowledge the need to enhance the clarity of our manuscript. To make our work easier to follow, we will simplify our annotations in future revisions. Additionally, we plan to release our code to aid in the replication and understanding of our research. While reviewers crn6 and yVYk mentioned that our manuscript is well-written and easy to follow, we recognize that there is some room for improvement. We will improve the overall presentation and ensure that our findings are communicated as clearly as possible.

3. Framework details: In Supp. A, we provide more detail about Vidu4D. Here we also provide more details of initialization and refinement.

In initialization, we first segment the foreground using TrackAnything. Then we extract the unsupervised features with DINOV2, optical flow with VCN [Learning to segment rigid motions from two frames], and metric depth with ZoeDepth. Considering the consistency of the generated video is limited, we register pair-wise camera poses using mask, depth, and flow. Then, we register the root pose of objects using DensePose and initialize the SDF field and warping field with volume rendering. During this process, we also refine the camera poses and root poses using a combination of photometric loss. To enhance registration with unsupervised features, we train an additional channel in NeuS specifically for rendering DinoV2 features, which are then employed for registration purposes as described in RAC. Compared with rasterization, the sampling strategy and continuity of volume rendering make it more suitable for refining poses.

For NeuS rendering, we backward warp sampling points in camera space $\mathbf{X}^t$ to canonical space $\mathbf{X}^*$: $\mathbf{J}^{t, -1} = \mathcal{R}\Big(\sum_{b=1}^{B} w_{b}^{t} \mathcal{Q}(\mathbf{J}^t_{b})^{-1}\Big),$ $\mathbf{X}^t = \mathbf{J}^{t, -1} \mathbf{X}^{*},$

which is an inversion of Eq. 4 in the main paper. By querying the SDF with $\mathbf{X}^*$, we render RGB and compute the photometric loss to optimize the SDF and the warping field defined in Eq. 6 of the main paper. However, there are two gaps between NeuS warping and DGS warping. First, the sampling points of NeuS are distributed in the frustum of the camera, while the DGS are distributed on the surface. Additionally, we train the inversion of $\mathbf{J}^t$ during initialization, while we utilize the non-inverse ones in DGS. To resolve the distribution gap and ensure that $\mathbf{J}^t$ faithfully models the forward warping, we add a cycle loss:

where $\mathbf{X}^{t}$ are a combination of mesh surface points and ray sampling points.

After initialization, we extract the mesh in canonical space with the marching cube and initialize Gaussian surfels on the mesh. We set the spherical harmonic in 0-th order to the RGB value of the nearest vertices. Here we keep the warping field and the learned camera poses.

4. Evaluated on realistic scenes and benchmarks: Please refer to the common response for more benchmarks. Specifically, we provide quantitive and qualitative comparisons on realistic scene-level benchmarks (Neural 3D Video dataset and NeRF-DS dataset) and object-level benchmarks (D-NeRF dataset).

5. Limitation of joint representation: Our method is not limited to reconstructing object-centric scenes. Please see Fig. D and Fig. G in our rebuttal PDF.

6. Latency: On an Nvidia A800 GPU, generating a 1080p video takes approximately 10 minutes. Preprocessing requires around 12 minutes, initialization takes another 10 minutes, and reconstruction takes 30 minutes. Rendering a 1080p image takes less than 0.1 seconds. We provide rendering latency in Tab. A on the Neural 3D Video benchmark, indicating the superiority of our rendering latency.

7. Reference video: Due to the submission guidelines, we are unable to upload videos. Instead, we have provided some frames from our reference videos in Fig. K.

8. Occluded parts: Thank you for your insightful comment. The generative capabilities of Vidu4D are indeed derived from the video generation model. If certain parts of an object are not visible in the reference video, those parts cannot be reconstructed. However, we have found that using the rendered results to guide the video generation model allows for training-free view completion. This approach leverages the strengths of the video generation model to hallucinate and fill in the occluded parts, thereby enhancing the reconstruction process, as shown in Fig. I.

Once again, thank you for your constructive feedback. We'll revise our paper according to your suggestions, move the skinning representation to the supplementary material, and add more details about the Vidu4D pipeline. We will open-source our code to facilitate understanding for readers. We kindly request that you consider raising the score accordingly if we addressed your concerns.

Thank you again for your time and effort in reviewing our work and providing the constructive comments. Please feel free to let us know if you have any further questions by August 13 AoE, we are more than happy to address them.

We sincerely thank you for your valuable comments and insightful suggestions. We address all your comments below. If our response has addressed the concerns and brings new insights to the reviewer, we will highly appreciate it if the reviewer considers raising the score.

1. Motivation/necessity of building surfels: Surfels improve the quality of surfaces, which in turn allows for the extraction of high-quality meshes, as illustrated in Fig. F. Furthermore, detailed geometry can enhance downstream applications, such as Gaussian-guided frame continuation at a specific camera pose, as demonstrated in Fig. I of our rebuttal PDF file, where a good depth/normal largely improves the continuation performance.

2. Organization of the method: Thank you for the constructive feedback. We will reorganize the methodology section to first introduce the overall framework of Vidu4D, followed by a detailed demonstration of the DGS technique. As discussed in Q2 of the common response, we will provide a more detailed description and analysis of Vidu4D. We hope this adjustment will enhance the clarity and flow of our presentation.

3. Real monocular videos: We evaluate our method on realistic scene-level benchmarks (Neural 3D Video dataset and NeRF-DS dataset). Please refer to Tab. C, Tab. D, Fig. D, and Fig. G of the rebuttal PDF file for experiments on real monocular videos.

4. Meshes and depth: Please refer to Fig. F for reconstructed meshes and depth.

5. Missing related work: Thank you for pointing out the missing related work. We will include a discussion of Liu et al.'s (2024) "Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos" in our revised manuscript to ensure comprehensive coverage of related research.

We sincerely thank you for your valuable comments and insightful suggestions. We address all your comments below.

1. Novelty: To the best of our knowledge, our method is the first to generate 4D content using a text-to-video model. We primarily focus on addressing the spatial-temporal inconsistencies in geometry and texture in generated videos, which is a novel question. Previous works, like SV3D and V3D, focus on enhancing the consistency of video diffusion. This approach is challenging and may lose the knowledge learned from high-quality videos and images when fine-tuning on videos rendered by 3D assets. Instead, we improve the tolerance of the reconstruction method to deal with this inconsistency. Specifically, we absorb multiview inconsistencies into the global motion of bones and unstable textures into the local motion of surfels. We select DGS of a specific frame when we want to obtain statistical meshes or 3DGS. We believe this design is novel.

Our dynamic Gassian surfel is novel. Compared with LARS, we design additional neural skinning weights and a registration technique using DINO features and an invertible warping function. Please refer to Sec. 3.2 and Fig. 6. Compared with Gaussian Surfels/2DGS, we design a warped-state normal regularization to enhance the surface across all frames. Also, we elevate Gaussian Surfels to 3D dimensions for better visual quality, as shown in Fig. 5(c). Compared with SC-GS/4DGS, we applied a relatively simple but plausible warping model containing only 25 anisotropy control bones. This simplification, in contrast to the 500 control points of SC-GS and the dense motion field of 4DGS, helps to regularize motion and prevent overfitting, especially when the camera poses are inaccurate and the object's multi-view consistency cannot be assured, as shown in Fig. B. This method of mitigating Gaussian overfitting is also robust against noise or floater occlusion, as illustrated in Fig. C, where the mask of the dragon covered by sand is missing a piece.

We also design a novel refinement to deal with flickering textures in generated videos. When Gaussian surfels are well reconstructed, its' normal is aligned with the surface normal, and this makes the gradient of density along the surface normal very large:

where $\mathbf{G}(\cdot)$ is the density of surfel, $x$ goes along the surface normal, $\theta$ and $\gamma$ are the angles between u, v, and the surface, respectively. Considering that $\theta$ and $\gamma$ are very small, the density is sensitive to the motion along the surface normal. Consequently, this direct of gradient leads the flickering of the texture to be modeled as the position of the surfels moving back and forth over time. When the front-back relationship between the rear surfels and the surface surfels changes, the texture flickers accordingly.

To alleviate this flickering, we introduce a refinement stage that elevates surfels to Gaussian ellipsoids. Compared to surfels, 3D ellipsoids have a smoother density field, and provide a more robust representation during warping, reducing the impact of flickering. In addition, $\Delta **R**^*_k$ and scaling $\Delta **S**^*_k$ defined in Eq.8, remove the constraint of aligning the shortest axis with the surface normal. This flexibility makes the 3DGS less likely to unintentionally introduce texture flickering during motion. This is illustrated in Fig. E.

2. Evaluation details: We follow the standard evaluation protocol for 4D reconstruction methods such as Hyper-NeRF and SC-GS. Specifically, we build the dataset by using every 4th frame as a training frame and taking the middle frame between each pair of training frames as a validation frame. We will provide additional details in the revised manuscript.

3. Special designs for generated videos: Please refer to Q2 in the common response for more details.

4. Experiments on 4D reconstruction datasets: We follow your advice and add more experiments on the commonly used 4D reconstruction datasets. Please refer to Q2 in the common response for more details.

Once again, thank you for your constructive feedback and for considering our paper for acceptance. We'll revise our paper according to your suggestions. We kindly request that you consider raising the score accordingly if we addressed your concerns.

Sincerely thank you for your feedback. We have some further responses w.r.t. (a) 4D reconstruction and (b) special designs for generated videos.

(a) 4D reconstruction

Since our focus is on the 4D reconstruction, to the best of our knowledge, there are no pose-free benchmarks for dynamic scenes. To address your concern, we build the benchmark by collecting openly available videos from the SORA official webpage. We then strictly compare our method against existing state-of-the-art 4D reconstruction methods. We provide the details and results below.

Benchmark details: We collect 35 sub-videos from the SORA [1] webpage, including Drone_Ancient_Rome (20 seconds in total, split into 4 sub-videos), Robot_Scene (20 seconds in total, split into 3 sub-videos), Seaside_Aerial_View (20 seconds in total, split into 3 sub-videos), Mountain_Horizontal_View (17 seconds in total, split into 3 sub-videos), Snow_Sakura (17 seconds in total, split into 4 sub-videos), Westworld (25 seconds in total, split into 4 sub-videos), Chrismas_Snowman (17 seconds in total, split into 3 sub-videos), Butterfly_Under_Sea (20 seconds in total, split into 3 sub-videos), Minecraft1 (20 seconds in total, split into 4 sub-videos), Minecraft2 (20 seconds in total, split into 4 sub-videos).

Evaluation method: We follow the standard pipeline for dynamic reconstruction (Hyper-NeRF, SC-GS, etc), to construct our evaluation setup by selecting every fourth frame as a training frame and designating the middle frame between each pair of training frames as a validation frame.

Results:

Upon acceptance, we will open-source this benchmark and the corresponding codebase to ensure its reproduction. Besides, since currently there are no pose-free benchmarks for dynamic scenes, we believe this built benchmark is a contribution.

Our full model achieves 4.24 PSNR improvement compared to the existing best method. The result proves the superiority of our method and each element we propose (field initialization, dual branch refinement) on the pose-free dynamic scene benchmark.

(b) Special designs for generated videos

As we summarized in the common response, properties of generated videos include both larger-scale aspects (unknown poses and unexpected movement) and small-scale aspects (flickering, floater occlusion).

• For larger-scale aspects, we propose the Field initialization stage which provides a proper start for our Dynamic Gaussian Surfels (DGS) regarding both the pose and the movement (please see warping transformation in Eq. 6 of our main paper). Here we'd like to highlight that the field initialization also benefits movement learning since the warping transformation is learned as a continuous field. This design is novel and especially beneficial for generated videos. We will provide more details of the field initialization during revision.

• For small-scale aspects, we have proposed the Dual Branch Refinement (Line 187) and provided ablation studies to prove its effectiveness in alleviating flickering.

Again we are grateful for your feedback.

[1] Video Generation Models as World Simulators.

Thanks for the author's response. This work might be a pioneering work of pose-free dynamic scene reconstruction. In this case, I would suggest to design a standard and rigor evaluation protocol using real-world multi-view video captured with camera poses. Currently, the evaluation camera pose is perhaps the same as training camera pose, lacking the changes in views, perhaps limited by generated video. I would take it as a benchmark for 4D reconstruction from generated video instead of a benchmark for general pose-free 4D reconstruction methods.

As for the novelty, despite the explanation of the authors, I think this project is without significant technical innovation.

But as the first attempt to reconstruct 4D content from generated video, this work is encouraging. Thus, I'll keep my positive score. The authors are suggested to test their work on various video generation models in the future.