DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation

The authors are grateful for the insightful feedback and support from the reviewer. Below we address the mentioned concerns separately.

Q1: Video qualitative results.

A: In the first paragraph of appendix, we have provided the link to our anonymous project page, which contains the input video and our rendered video results as well as our blender demo. Please check it.

Q4: Concerns about computation cost of training and inference

A: For inference, which is the process of computing the deformed 3D object under a given timestamp, we report the time consuming of different methods as below:

Among these methods, Consistent4D utilizes implicit representation and volume rendering, leading to much more query points for the deformation network and hence the longest inference time (51ms). Compared to those Gaussian-based methods, DreamGaussian4D, 4DGen and STAG4D, our method has a little bit slower inference time (14ms). The extra computation overhead comes from the skinning calculation, which is implemented with pure PyTorch. This overhead can be eliminated through some engineering effort of implementing it as a CUDA extension, and we leave it for future works.

As for the computation cost of training, we have reported the overall training time and GPU memory usage in our global rebuttal. Compared to other methods, our method is the most computational efficient considering both the time and memory consuming (0.8h and 8GB).

Q5: Concerns on updating spherical harmonics.

A: Following previous 3D generation works on 3D Gaussians, we use 0-degree spherical harmonics which is actually equivalent to RGB color without view-dependent effects. Hence we do not compute the deformation of spherical harmonics. We would explore to use higher level SH-coefficients for view-dependent effects in future.

Q6: Why 6 Gaussians per triangle in the static stage?

A: By compromising on the appearance representation capability and computational cost, we choose 6 Gaussians per triangle. More Gaussians would enable a better capability to represent the surface appearance, with an expense of more computational cost. As shown in Figure 2 and Table 3 of the supplemented PDF rebuttal, the experimental results demonstrate 6-Gaussians can already deliver satisfying performance by considering the rendering quality and training time. It is worth mentioning that, even with only 1 Gaussians per face, our method still largely outperforms previous methods with the fastest training time.

Q2&Q3&Q7&Q8: Concerns about writing.

A: We really appreciate your advice on the paper writing, and we will revise the manuscript according to the constructive suggestions. We will include the works on 4D neural implicit surfaces in the 4D representations paragraph in the Related Work section. All mathematical notations in the paper will be carefully simplified for easier reading. And we will supplement a proper explanation about Equation 10 as you mentioned in Question (5). Finally, the references in line 38 are indeed a typo, and the words before these references should be "differentiable 3D representations" rather than "3D generation".

The authors are grateful for your support! We will further polish our method in the final version.

The authors are grateful for the valuable and in-depth feedback of the reviewer. Below we address the mentioned concerns separately.

Q1: Limited novelty compared to previous methods.

A: Previous methods for monocular video-to-4D generation include Consistent4D, DreamGaussian4D, 4DGen and STAG4D. All of them directly employ existing dynamic neural radiance fields or 3D Gaussian Splatting as their base representations, and predict query points' deformation with a MLP network. In contrast, we are the first to exploit mesh as the underlying representation for the problem and demonstrate superior effectiveness compared to prior methods. Although it might be predictable, no prior work has really tried and demonstrated the possibility of the idea for this challenging task. As both reviewer 1 and reviewer 3 also agree on, the method to generate dynamic meshes is new and informative. The authors humbly think the proposed method would provide valuable insight for following works.

Q2: Which aspects of the pipeline have contributed to the quality improvement compared to previous methods?

A: The main aspect contributing to the quality improvement is the employment of mesh representation. 3D mesh models surface explicitly and is crucial for appearance modeling and animation. To validate the claim, we further conduct a thorough ablation study by replacing the mesh representation with no surface bound 3D Gaussian Splatting, and leave other parts unchanged. The experimental results demonstrate the mesh representation achieves a PSNR 36.73 dB compared to 28.71 dB with 3D-GS. It demonstrates the superior advantage by exploiting surface bound Gaussian Splatting for this challenging less constrained optimization problem. Detailed quantitative and qualitative evaluation results can be found in the rebuttal PDF (i.e. Table 2 and Figure 1).

Q3: Concerns about computation cost.

A: The detailed comparisons against prior works in terms of time and GPU memory usage can be found in the global rebuttal section. We evaluate all the methods on a single NVIDIA RTX 6000Ada graphic card since Consistent4D cannot run on a 24G GPU. Our method requires 0.8h and 8G memory for training, which is superior compared to Consistent4D, DreamGaussian4D, 4DGen and STAG4D. For example, Consistent4D requires 2h and 28G memory for training, STAG4D requires 1.6h and 7G memory for training.

Q4: Insights of this work.

A: The 4D generation task is inherently not a well constrained problem, usually only a sequence of monocular video is given. Additional constrains should thus be enforced to achieve better performance. Prior works usually rely on a pre-trained image diffusion model to provide extra constrain in the form of score distillation loss. Although they can deliver impressive performance, the quality of the generated 4D asset still has room for improvement, e.g. the asset usually exhibits blurry appearance or unsatisfactory geometry as shown in our experimental results.

The authors suspect the performance gap might be caused by the used loosely constrained 4D representation, i.e. NeRF or 3D-GS with an additional deformation network for dynamic modeling, which is commonly used for multi-view 4D reconstruction problem. Since multi-view cues are usually sufficient to constrain the problem, prior commonly used 4D representations would thus still be able to deliver good performance. However, it is not the case for 4D generation from a monocular video, which provides much less multi-view cues.

Therefore, the authors thought additional constraint should be enforced to better constrain this more challenging problem, which motivates us to explore the usage of the Gaussian-mesh hybrid representation. Gaussian-mesh representation was firstly proposed in SuGaR and used for triangular mesh extraction from a pretrained 3D-GS model. It enforces 3D Gaussians to bind on the object surface and degenerate towards 2D Gaussians. This kind of enforcement reduces the parameter searching space of the optimization problem, and thus could deliver better performance. The experimental results also prove our hypothesis and demonstrate superior performance compared to prior works.

However, it is not trivial to simply integrate Gaussian-mesh representation into a 4D pipeline. For example, 1) how to efficiently and effectively deform the mesh-Gaussian representation for 4D dynamics? 2) how to make sure the surface bound Gaussians are still able to render high-quality textures under deformation? It is those challenges motivating us: 1) to propose the new skinning algorithm combines the best of LBS and DQS for this particular problem; 2) to propose the deformation of surface Gaussian attributes; 3) the usage of deformation graph to provide additional geometric deformation constraints. It is all those factors and insights leading to the final formulation and superior performance of our method.

Furthermore, mesh based representation is already very mature for modern CG pipelines. Generated 4D assets with mesh representation and skinning method can be conveniently exported to existing commercial pipelines, which further motivates us to exploit this representation. The authors humbly think our work would be valuable to the community and provide good insight for following works in relevant fields.

The authors are grateful for the thoughtful feedback and valuable questions. Thanks for the support for this work! Below we will address the questions and concerns separately.

Q1: The difference between the proposed method and SuGaR.

A: While SuGaR proposes a pipeline reconstructing static surface via a Gaussian-mesh hybrid representation from pre-trained 3D Gaussians, our work mainly exploit the proposed hybrid representation as our base representation, and focus on extending it for video-to-4D generation task, similarly as most prior works which rely on NeRF or 3D-GS as the base representation.

Although we utilize the static 3D representation proposed by SuGaR, trivially applying it to 4D tasks brings challenges. For example: 1) how to effectively preserve the Gaussian appearance while the asset undergoes deformation (i.e. surface Gaussian attributes update during deformation); 2) how to efficiently and effectively deform the vertices of the mesh-Gaussian representation (i.e. deformation graph and hybrid skinning).

Q2: Concerns about Figure 1.

A: The input of our method is a monocular video of the target object under a fixed view, which is consistent with the setup used in previous works. As for the composite scene video in Figure 1, it is a demo created by ourselves in Blender using our generated assets. It is to demonstrate that the generated 4D assets by our method can be directly integrated into downstream graphic engines, which further prove the advantage of our work. We will make this more clear in the final version.

Q3: Innovative contribution of this work.

A: The innovative contribution of our work is that we for the first time demonstrate the possibility and superior advantages of extending Gaussian-mesh representation for 4D tasks, in comparison to prior works, which mainly exploit NeRF or 3D-GS as the underlying representation. As R3 also points out that it is informative to the community that SuGaR works well in a more complex 4D pipeline. Although SuGaR, deformation graph, LBS and DQS are existing techniques, trivially integrating them would not lead to optimal performance. We therefore 1) proposed to change the Gaussian attributes during deformation; 2) proposed a novel skinning algorithm suitable for such optimization-based scenarios with insufficient constrains. The experimental results demonstrate the effectiveness of our design and our method achieves superior performance in comparison to prior baseline methods.

Q4: Computation cost of the proposed method.

A: The time and memory complexities are 0.8h and 8G for training on a single NVIDIA RTX 6000Ada graphic card, which is also superior compared to prior methods and further demonstrates the advantage of our formulation. The detailed metrics in comparison to other methods can be found in the global rebuttal.