MVTokenFlow: High-quality 4D Content Generation using Multiview Token Flow

Thank you for the helpful review. We carefully address your questions and comments below.

Q1: The author does not present clearly about the motivation and main contributions of this work in the introduction part.

A1: The challenge in high-quality 4D generation is to create temporally consistent multiview images from monocular 2D videos. Our core motivation is to first construct a coarse 4D field, temporally consistent in the front view, and refine it to achieve high temporal consistency across all views. Specifically, in the coarse stage, we utilize multiview diffusion and the temporal consistency of the front view to enhance the spatiotemporally consistent 4D field for better rendered 2D flow. In the refinement stage, this rendered 2D flow guides the regeneration of consistent multiview images, culminating in a high-quality 4D field.

Our contributions can be summarized as follows.

1. Based on our motivation, we design a novel pipeline that interleaves dynamic Gaussian field reconstruction with multiview token flow to generate multiview consistent videos. To the best of our knowledge, we are the first to adopt such an interleaved refinement of 2D flow and multiview token flow for 4D generation.
2. We are the first to extend the idea of 2D token flow to multiview diffusion to improve temporal consistency.
3. We have demonstrated improved quality in 4D content creation from a monocular video, enhancing both quantitative and qualitative evaluations, and without the need for any bells and whistles or post-process optimization.

Q2: In the related work, the author firstly claimed the diffusion model applied in 4D generation will be discussed, while in the section, 4D Scene Representation, only NeRF and 3DGS based method have been discussed. The related works need to be reorganized to ensure the logical coherence between each parts.

A2: Thanks for the comment. 4D generation methods are discussed in L150. The related works section discusses 1) diffusion methods, 2) 4D representations, and 3) 4D content generation with diffusion models. The third part is about the diffusion model for 4D generation. We have improved the logical coherence here in the revision.

Q3: 1) Applying the self attention on the temporal dimension has been well studied and proved to be effective way to maintain the temporal consistency. Incorporating this strategy may not have enough novelty. 2) No need to further training not have enough novelty since building the original temporal multi view diffusion model requires huge training while no need for retraining for enlarging the self attention is straight forward.

A3: Thanks for your comment. While enlarging self-attention is an important component of our method, it is not the main contribution of our method. Our key idea is to generalize the token flow method to 4D content creation. By interleaving Gaussian reconstruction and token flow, we enable high-quality 4D content creation. We have included further clarification of our motivation and contributions in the introduction.

Thank you for the helpful review. We carefully address your questions and comments below.

Q1: Era3D uses Row-wise Multiview Attention (RMA) to achieve multi-view consistency. In L232, the authors say that they use enlarged self-attention to improve the temporal consistency of multi-view video frames. Using viewpoints and a keyframe interval of 8 frames of video, the enlarged self-attention across all different timesteps may be quite memory-heavy. I think this process is a distinguishing feature compared to the previous work Era3D, but the authors don't provide an ablation study on how this component improves temporal consistency in the coarse stage.

A1: Thanks for the comment. Our enlarged multiview attention does not bring too much memory consumption. Given the tensor (T,N,H,W) with T frames and N views. The RMA of Era3D is applied on the N dimension to maintain multiview consistency while our enlarged self-attention is applied on the T dimension to associate frames of different timesteps. These are two different attention layers in the implementation, which does not result in too much memory consumption. Our enlarged self-attention layers are orthogonal to RMA of Era3D with different functions. In Table 3, we include an ablation study that removes enlarged self-attention. The FVD results show that enlarged self-attention alone provides limited benefits without our proposed multiview token flow.

Q2: As I understand it correctly, the coarse stage produces less temporal-consistent 3DGS, and the fine stage renders 2D flows to guide the re-generation of the multi-view images and create the final temporally consistent 3DGS. When describing this, Fig. 2 does not intuitively illustrate the process. It would be better if the two stages were completely separated.

A2: Thanks for the suggestion and we have modified Fig.2 in the revised version.

Q3: The ablation studies on token propagation and flow loss for multi-view video generation show only qualitative results. Quantitative results (using the same metrics as in Tab. 1) are needed to show the generality of these modules.

A3: Thanks for the suggestion. We have added the quantitative results in Table 3.

Q4: Similar to point (3), the authors use the normal loss, but this feature is not ablated, either qualitatively or quantitatively.

A4: Thanks for the comment. We have included an ablation study on the normal loss as shown in Table 3 on the Consistent4D dataset. The results show that the normal loss effectively constrain the geometry and thus improve the rendering quality.

Q5: How long does it take for the coarse stage and the fine stage, and how much GPU memory is required for each stage? Also, how about the other baselines?

A5: The whole process costs about 2 hours. The diffusion process costs about 30G while Gaussian field reconstruction costs about 10G memories. We have included the comparison of generation process time with other methods in Table 4 of the appendix.

Thank you for sharing the results. Since all my concerns have been resolved, I raise my rating from 5 to 6.

Thank you for the helpful review. We carefully address your questions and comments below.

Q1: Token Propagation: As illustrated in Section 3.1, token propagation is only conducted between key frames and non-key frames, which means the temporal consistency among key frames is not guaranteed. Although the authors utilize enlarged self-attention for key frames, I think this cannot ensure full temporal consistency, especially when the objects have large movements.

A1: Thanks for your comments. We agree that even after the multiview token flow, there are still some remaining inconsistencies on keyframes. However, we can still reconstruct a high-quality 4D field because

1. All the keyframes are generated from the input video by multiview diffusion. Since input video is temporally consistent, these keyframes do not show too strong inconsistency from each other.
2. We reconstruct a dynamic 3D Gaussian field from these images, which are supervised by the temporal consistent input video and thus can smooth the remaining inconsistency.

We have added this discussion to the paper in L251 of the Method section.

Q2: Dataset: The authors are encouraged to conduct qualitative experiments on the official Consistent4D dataset instead of its demo videos. The official dataset contains 7 synthetic objects with multi-view videos; please refer to the SV4D paper for the performance of other video-to-4D methods on that dataset. Q3: Comparison: The authors are encouraged to conduct novel-view synthesis comparisons with other methods if possible.

A2、A3: Thanks for the suggestion. We have added the quantitative results in Table 2 L442. The results show that our method achieves slightly better performance than Diffusion^2. These results follow the same evaluation setting as Consistent4D, which compares the novel-view rendering quality.

Q4: Ablations: The authors are encouraged to conduct quantitative ablations and provide more qualitative analyses.

A4: Thanks for the suggestion. We have conducted additional quantitative ablation studies on the Consistent4D dataset. The results are shown in Table 3 in L493 in the revised paper. The ablation studies demonstrate the effectiveness of our enlarging self-attention layers, multiview token-flow design, and normal loss.

Q5: Visualizations: The supplementary videos have only two views of the object, one of which is the input view, which does not provide readers with a full understanding of the performance of the proposed method. The authors are encouraged to provide videos with more views. Additionally, in Figure 3, the comparison with state-of-the-art methods is conducted in the input view, which may not be appropriate.

A5: Thanks for the suggestion. We have added the orbit video in the Supplementary Material. Additional Comparisons with baseline methods on novel views are shown in Figure 8 of the main paper.

Q6: The authors claim they compare with all open-sourced methods, but as far as I know, there are other open-sourced video-to-4d works which are not included for comparison, for example, DG4D and Diffusion^2. I'm not asking the authors to comapre with all open-sourced works, instead, I suggest to modify the inappropriate expression.

A6: Thanks for the suggestion. We have tried our best to reimplement the baseline methods. We agree that some of these works are still not included due to the rapid development of this field. We have added the comparison with DG4D and Diffusion^2 and changed the sentences in the revision.

Thanks for the efforts of the authors in rebuttal. But my concerns are not adequately addressed.

1. Quantitative evaluation on Consistent4D dataset: The reported results of other methods in Table 2 differs from results in SV4D and also differs from results in their papers. To be honest, it is hard to believe other method performs so bad in terms of FVD metric. Could the authors explain the reason? Do the authors re-implement other methods by themselves or adopt official implementation? Do the authors use official evaluation code? I have run the official implementation of some other methods before and the results of FVD seems much better than what is reported by the authors.

2. Conflict between ablation and comparison: ous in Table 2 and full in Table 3 report very different results. Do the authors use the same evaluation data and the same model?

3. Visualization results: some visualization results appear flat effect, and some have small motions.

Thank you for your timely feedback! We have conducted the requested quantitative ablation study on the Consistent4D synthetic objects to align with the comparison results. The results are as follows:

The results in the ablation table indicate that incorporating the normal loss contributes to visual quality of novel views(full vs w/o normal), while the flow improves temporal consistency(full vs w/o flow).

We appreciate your engagement and effort in reviewing our work and look forward to any further suggestions you might have.

It's glad to see that the proposed method outperform other methods in term of all metrics.

To make this work better, I suggest the authors to perform a detailed analysis of why the proposed method outperform other methods. I guess the significant gain in CLIP and LIPIS is due to the use of advanced 3D diffusion model, Era3D, which enables novel view results with better visual quality and generates normal map for geometric reconstruction. However, previous works all adopt Zero123 as 3D diffusion. (I compare the results in ablation w/o normal and comparison SC4D to draw this conclusion) But the reason might also be that the proposed method uses reconstruction while previous works adopted SDS. Could the authors provide some experiments to explore this?

Additionally, I want to explain why I emphasize FVD metric. FVD evaluates both visual quality and temporal consistency, while previously adopted CLIP/LPIPS/SSIM only evaluates visual quality. Temporal consistency is important for 4D generation, especially when you regard lifting a 3D diffusion to 4D in training-free way as your main contribution. To be honest, the gain in FVD in ablation table is not very satisfying. And it's strange that w/o enlarged SA lags behind in LPIPS and CLIP, since I think the per-frame 3D diffusion generation results should have better visual quality yet worse temporal consistency. Try to perform a in-depth analysis, please.

I'm willing to raise the rating if the authors conduct in-depth analysis of their model and comparison methods, regardless of whether their conclusions further highlight the contributions of their method.

I was expecting to see some additional quantitative experiments to explore what I mentioned before. And I think one important experiment is missing, i.e. evaluating the multi-view video from Era3D directly, and conduct ablation on it, given that the main contribution is multi-view video generation not the following 4d reconstruction.

Anyway, I hope you can spend more time to improve this work. I'll raise my rating to encourage the author to further improve the article and bring more insights to the community

Thank you for the helpful review. We carefully address your questions and comments below.

Q1: Minor point, but I'd recommend the authors add a clear list of contributions at the end of the intro.

A1: Thanks for your comments. Our contributions can be summarized as follows.

1. We extend the 2D token flow to multiview diffusion to improve temporal consistency.
2. We design a novel pipeline that interleaves dynamic Gaussian field reconstruction with Multiview Token flow to generate multiview consistent videos.
3. We have shown improved quality in 4D content creation from a monocular video.

We have included this part in the revision.

Q2: Presented videos in the supplemental are pretty limited. The columns also aren't labeled. I'm guessing the left most image and flow field correspond to the input and the right most is some arbitrary second view? It would have been nice to see some sort of orbit around the object as the video plays

A2: Thanks for the suggestion. We have added the orbit video in the Supplementary Material.

Q3: Will the custom dataset be made available?

A3: Yes, we will make all our examples available after accepting.