Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video

Dear Reviewer YMYg,

We're grateful for your appreciation of the innovation and paper writing of our work, as well as your suggestions for improvement. Below is the response to your questions:

• The “real image” in video enhancement: The real image is generated by a pre-trained SR model. We use images output by the SR model as the “real image” in GAN because the SR model is designed for single images and does not maintain temporal consistency across frames. This lack of temporal consistency is challenging to detect in still images, so we attach a video of “real images” from the SR model as well as the output of our video enhancer in the supplementary material (video_enhancer.mp4). Please refer to it to see temporal inconsistency. We found training another lightweight Convnet as the video enhancer can alleviate this temporal inconsistency.

• Some noticeable artifacts in supplementary material: We did not exclude all the failure cases when we provided the demo video in supplementary material, for example, the failure cases discussed in the failure case section (Sec. A.5) are also presented in the demo video. Some failures in our work stem from inaccurate supervision signals provided by the diffusion model. For instance, as seen in Fig. 9 of the appendix, the dog's tail occasionally vanishes. Other failures are linked to the flaws in our 4D representation and optimization processes, such as the blurring of the frog in Fig. 9.

• Increasing the cascade layers: Previously, we didn't increase the number of cascade layers because increasing it to 3 and 4 took around 45 and 60 GB of GPU memory for training, respectively, and we didn’t have access to computing resources with such large GPU memory. Now, we find Increasing cascade layers improves the texture of the generated dynamic object, alleviates the blurring effect, and facilitates spatial and temporal consistency. However, it can only slightly alleviate the negative effect of inaccurate supervision signal, which means failure cases caused by the diffusion model couldn’t be improved much by increasing cascade layers. Please refer to response 4 in “Responses to All Reviewers” for more details. Thank you for your suggestions.

Dear Reviewer B4LH,

Thank you for the recognition of the proposed ICL loss as well as the overall paper writing. Below is the detailed response to your concerns:

First and foremost, we sincerely apologize for the typographical error in the SDS weight. It should be 0.1 instead of 0.01. We regret any confusion this mistake may have caused regarding our method.

The hyperparameters of reconstruction weight and sds weight, i.e., 500 and 0.1, are exactly the same as the default config of Zero123 implemented in Threestudio[1]. The two loss weights generate good results in Zero123, so we did not finetune them in our experiment. Other losses, including normal orientation loss and 3D smoothness loss, were kept exactly unchanged in all experiments in the paper to ensure a fair comparison.

• Generative Baseline: We add zero123-per-frame as suggested, please refer to response 1 in “Responses to All Reviewers” for more details.

• Ablations: Previously, we presented extensive ablations for all of the novel modules introduced in our paper, including Cascade DyNeRF, ICL, and the video enhancer, as detailed in Sec. 5.3. Quantitative ablations are newly added, as shown in response 2 in “Responses to All Reviewers”. For other commonly adopted losses in image-to-3D methods, such as the reconstruction loss and 3D normal smoothness loss, we maintained them as default settings and did not provide ablations. However, we are more than willing to conduct additional ablations to provide readers with a more comprehensive understanding of our work:

  1. The ablation of reconstruction loss, foreground mask loss, normal orientation loss and 3D smoothness loss: While these losses have been adopted from Zero123 implemented in Threestudio and do not constitute our original contributions, we understand the importance of providing a comprehensive analysis. Below is the ablation for losses:

     	LPIPS ↓	CLIP ↑	FVD ↓
     ours w/o rgb loss	0.16	0.85	1260.39
     ours w/o mask loss	0.15	0.87	1157.97
     ours w/o normal orientation loss	0.15	0.86	1146.39
     ours w/o 3d normal smooth loss	0.16	0.86	1191.75
     ours w/o SDS loss	0.19	0.81	1699.83
     ours	0.16	0.87	1133.44

     It is evident from our metric analysis that the SDS loss holds a pivotal role, exerting a significantly greater impact compared to the RGB reconstruction loss. This observation underscores the generative nature of our method, emphasizing the importance of the SDS loss.

  2. Remove video enhancer: The Video enhancer, while a valuable addition to our framework, does not constitute our core contribution. Its inclusion is aimed at enhancing the overall comprehensiveness of our approach. The improvement from the video enhancer is limited, as indicated in response 2 in “Responses to All Reviewers” (the last row of the table).

  3. Probability of ICL: we provide the ablation of probability of ICL as bellow:

     	LPIPS ↓	CLIP ↑	FVD ↓
     probability=0.	0.16	0.87	1226.92
     probability=0.1	0.16	0.87	1190.01
     probability=0.25 (ours)	0.16	0.87	1133.44
     probabilityt=0.5	0.16	0.86	1179.67

     FVD metric suggests that a probability value of 0.25 is an appropriate setting.

  4. SDS weight: We provide experiments about changing sds weight as bellow: | | LPIPS ↓ | CLIP ↑ | FVD ↓ | | -------- | -------- | -------- | -------- | | sds_weight=0. | 0.19 | 0.81 | 1699.83 | | sds_weight=0.01 | 0.16 | 0.85 | 1321.19 | | sds_weight=0.1 (ours) | 0.16 | 0.87 | 1133.44 | | sds_weight=1. | 0.16 | 0.86 | 1270.86 | The results indicate sds_weight=0.1 is a suitable setting.

Dear Reviewer 6R15,

Thanks for the acknowledgment of the innovation and effectiveness of our framework. Below, we provide detailed responses to address your concerns:

• Paper writing: we have re-origanize the paper as you suggested. We list the changes we made below:

  1. Confusing sentences about Cascade DyNeRF are removed
  2. Video enhancement is claimed as an optional step
  3. Loss function part is moved to the main body of the paper
  4. Comprehensive implementation details regarding both DyNeRF training and video enhancer training are provided in the appendix (Sec. A.4)
  5. Zero123 is added in comparison with state-of-the-art, and quantitative ablations are added as well

• Training time: Training time for dynamic NeRF/video enhancer is around 2.5 hours/15 minutes, measured on a single V100 GPU. For details, please refer to response 3 in “Responses to All Reviewers”.

• Comparison with Zero123: Zero123 lags behind us in terms of temporal consistency. Please refer to response 1 in “Responses to All Reviewers” for more details.

• Quantitative ablation: The improvements in the metric FVD validate the effectiveness of each proposed module. Please refer to response 2 in “Responses to All Reviewers” for more details.

• Reproducibility: We have provided extensive implementation details, please refer to response 5 in “Responses to All Reviewers”.

• PSNR and SSIM as losses: Thanks for suggesting using the two losses. Actually, we had conducted experiments using L1/PSNR/SSIM as the rendering loss. We empirically find that the L2 loss generates the best result, and that is the reason for choosing the L2 loss in our method.

• Inconsistent texture: The texture inconsistency problem is widely observed in generative image-to-3D studies [1,2,3,4], which might be related to the SDS optimization. It will be our future work to improve the texture consistency.

• Key-point tracking: Currently, our model employs an implicit 4D field to model dynamic objects, which precludes the possibility of key-point tracking. This could be potentially addressed by using a point-based 4D model (e.g., dynamic 3D Gaussians) and we leave it as a future work.

• Evaluation on previous dynamic NeRF settings: Although conventional reconstruction is not the focus of the proposed method, we are glad to test our model in that setting. We will do this experiment.

• Explanation

  a）Temporal coherence as the main challenge: You’re right. Temporal coherence is a unique challenge in our pipeline, cause we adopt an image diffusion model without the notion of time to provide the supervision signal.

  b) K-planes with low-rankness: Thanks for pointing out. We agree that the use of “low-rankness” is inaccurate and confusing, so we have removed it in the revised paper. What we want to express is, K-planes is naturally prone to temporal coherence given that it has temporal interpolation operation. Considering temporal coherence is the main challenge in our pipeline, we choose to build our dyNeRF on K-planes.

  c) Typos in Eq.6 and questions about spatial consistency: Thanks for pointing out. Yes, you’re right about Eq.6 and we have fixed those typos. For spatial consistency, we render batch images using virtual cameras at a single timestamp when applying ICL loss.

  d) The meaning of resolution in implementation details: The resolution in the second statement stands for the resolution of rendering images during training. Due to limited GPU memory, we can only render 64x64 images when using batch_size=4.

  e) Loss functions: We have moved them to the main body of the paper and have cited corresponding literatures in the revised paper.

References

[1] Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors. In CVPR, 2023.

[2] Realfusion: 360deg reconstruction of any object from a single image. In CVPR, 2023.

[3] Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. In ICCV, 2023.

[4] Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In arXiv, 2023.

Dear Reviewer ZZjx,

Thank you for acknowledging the significant contribution of our work to the field, especially the proposed ICL loss. Below, we try to carefully address each of your concerns:

• Reliance on pre-trained models: our method necessitates the use of pre-trained diffusion models to generate supervision signals for 4D generation. This is crucial for ensuring the generalization ability of the 4D generation task as only limited 4D data is publicly available for training.

• Computational efficiency: The training time of dynamic NeRF and video enhancer is 2.5 hours and 15 minutes respectively on a single V100 GPU. Please refer to response 3 in “Responses to All Reviewers” for more details. Our method is not capable of real-time application at this juncture, however, we are optimistic about achieving real-time efficiency in the future.

• Comprehensive comparison:
  a) dataset: Previous multi-view dynamic scene datasets are mainly scene-level, but our method can only tackle object-level generation. So we render multi-view video from animated modes by ourselves for quantitative evaluation. We will improve and extend our work to scene level in the future.

  b) comparison methods: Since there is no previous method capable of both zero-shot 3D generation and temporal modeling, our comparison includes dyNeRFs, which excel in temporal modeling but lack generation capability, and Zero123, which offers generation ability but lacks temporal modeling. Please refer to responses 1 and 2 in “Responses to All Reviewers” for quantitative results.

• Limitations and potential failure cases: Thanks for pointing this out. We previously provided failure cases in the appendix section (Sec. A.5), and we have elaborated the limitations in the updated appendix as suggested (Sec. A.6) . A brief summary of limitations of the current approach is as follows:

  1. Easy to fail when the object movement is complicated and fast
  2. Easy to fail when the input video is blurred or noisy
  3. Easy to fail when the foreground segmentation is not good
  4. Easy to fail when the input video contains multiple foreground objects
  5. Rely on the pre-trained diffusion model.
  6. Cannot be real-time
  7. Cannot handle scene-level reconstruction

• The effect of limitations on practical deployment:

  1. The efficacy of our model is closely tied to the quality of the input video. In practical deployments, it may struggle to accurately reconstruct dynamic objects when faced with low-quality input videos
  2. The dependency on a pre-trained diffusion model can adversely affect the generalization capability of our approach. This concern is particularly pronounced given that the diffusion model employed in our work is trained on synthetic data. Consequently, there's a potential for a domain gap when processing real-world input images or videos. Our experiments indicate a higher likelihood of reconstruction failures with real-world videos as compared to those sourced from animated films
  3. The relatively extended training time of our model presents a challenge for its practical deployment

  We will try to improve our work to address the above limitations in the future.

• User study

  1. We are the first work aimed at 4D dynamic object generation from uncalibrated monocular video, so it’s difficult to find proper methods to compare with. Current comparison methods include two dynamic NeRF methods (previously provided by us) and one image-to-3D method (suggested by the reviewers, run in a per-frame reconstruction way). The former suffers from failure reconstructions and the latter results in severe temporal inconsistency, as shown in qualitative and quantitative results in Figure 3, Table 1, and the visualization video in the supplementary material (comparison_zero123_baseline.mp4). Obvious bad performances are observed in all comparison methods given that they are not specially designed for this task. Thus, we regard a user study between them and our method as unnecessary.
  2. Instead, we previously provided two user studies for the ICL loss, the key contribution of our work. Please refer to Sec.5.3, Table 2 in the main paper for details.

Dear Reviewer Yv9V,

Thanks for the appreciation of the novelty as well as the paper writing of our work. Below is our detailed response to your questions:

• Explanation of Figure 2:

  a) The input is a monocular video captured by a static camera, and the image sequence from this video is used as the condition of diffusion models for SDS loss as well as the ground truth for reconstruction loss .

  b) The output is a dynamic NeRF representing the dynamic object. Besides, we can use the video enhancer to further enhance the video rendered from dynamic NeRF.

  c) Loss propogation: Our method contains two stages. In first stage, SDS loss provided by the diffusion model is used to supervise the training of dynamic NeRF through the rendering images. The proposed ICL loss is used to supervise the dynamic NeRF in the same way. In the second stage, we employ a GAN framework, with the generator acting as the video enhancer. We use the conventional GAN loss, and we apologize for an accidental error in the loss direction depicted in Figure 2. We have rectified this mistake.

• Comparisons on other datasets: Existing multi-view dynamic scene datasets are usually scene-level, however, currently our method can only tackle object-level generation. A few datasets are object-level, however, the dynamic objects in those datasets are so complicated that even single-image-to-3D methods fail to generate the static 3D object from a single frame in the video, let alone our methods for dynamic object generation. We will improve our work and extend it to the scene level and complicated objects in the future.

• Training time and inference time comparison with other methods: Since Zero123 requires more than 40GB GPU memory for training, we compare the training and inference time of all methods on an 80G A100 GPU. We provide the averaged results on 4 objects as follows (inference speed is measured under image resolution of 256x256): | | training time | inference speed | |-----------------|-----------------|-----------------| | D-NeRF | 61 min | 2.8 fps | | K-planes | 76 min | 19.2 fps | | Zero123(per-frame) | 79 min | 41.8 fps | | ours w/o video enhancer | 91 min | 63.8 fps | | ours w/ video enhancer | 97 min | 62.9 fps |

The training time of our method is a little longer than per-frame Zero123, whilst we have fast inference speed.

• Detailed usage of the diffusion model: We illustrated the use of the diffusion model in the Preliminaries (Sec. 3.1) and we elaborate on it below: The diffusion model adopted in this work is Zero123-xl, an image-to-image translation model. In the original design, this diffusion model takes a reference view image and relative camera position between the target view and the reference view as the input and outputs the target view image. In our pipeline, we use images from the monocular video as the reference images. Following image-to-3D methods, we random sample target views for rendering images from dynamic NeRF, and calculate the relative position between the target views and the reference view to serve as the camera position input of the diffusion model. Then, the diffusion model can calculate SDS loss for the rendering images of the target view via Eq. 2 in the paper. The gradient is backward to dynamic NeRF through the rendering images.

  For easy understanding, we now provide another interpretation of SDS loss in Eq. 2. SDS loss is equal to the L2 loss between the noisy image (obtained by adding random noise to rendered target view images) and the 1-step denoised image output by the diffusion model. Thus, the distribution of rendered images from target views is always optimized toward the distribution of target images in the diffusion model.