Controlling Space and Time with Diffusion Models

Thank you for your review. We address the reviewer’s concerns below:

Weakness 1 (metrics): Reconstruction based metrics have several limitations with respect to evaluating the performance of generative models. This is particularly exacerbated for out-of-domain evaluation (e.g. on LLFF) where the generated samples are expected to diverge from the ground truth even more. Given the imperfections of various metrics, they must be analysed in aggregate to draw conclusions. In Tab 1, our method consistently outperforms baselines on sample fidelity (FID, FDD) and also in 3D consistency and pose alignment (TSED). A similar trend is observed in Tab 2 for our calibrated vs uncalibrated models. We are happy to discuss this further during the discussion.

Weakness 2 (mixing data): Absolutely, we do not claim that mixing datasets is novel; on the dataset front, our main contribution is highlighting the value of training on metric-scale calibrated data and the demonstrated effectiveness of the specific mixture we use.

Weakness 3 (comparisons / 2-image-to-video): Thank you for this valuable feedback. We have now included qualitative comparisons to FILM for the video interpolation task in our anonymized website. We additionally included other baselines, RIFE (non-generative), and diffusion baseline LDMVFI where the same samples were made available to us for comparison, as the reviewer also suggested a generative baseline) [1,2]. Our model is competitive with all of these baselines, and does a much better job when there is a lot of motion between the start and end frame, whereas non-generative baselines present severe artifacts in such cases. We include FID scores below:

• [1] RIFE: Real-time intermediate flow estimation for video frame interpolation, Huang et al 2022
• [2] LDMVFI: Video Frame Interpolation with Latent Diffusion Models, Danier et al 2023

Weakness 3 (comparisons/panorama): As for comparisons with panorama generation methods, please note that our method is not text conditional, and the baselines the reviewer suggested are all text-conditional (in particular panodiff has a two-stage pipeline where the second stage is text-conditional). For a fair comparison with ours, we run MVDiffusion with an empty prompt (which was their unconditional setting during training for CFG on text). We find that text-unconditioned MVDiffusion performs poorly for panorama generation. We include a few qualitative comparisons in our anonymized website.

Weakness 4 (co-training video): Thank you for this feedback. We have now included some side-by-side qualitative comparisons of our model trained with and without video data in our anonymized website to demonstrate that, without access to large-scale video data of the real world, these models do not achieve a good understanding of the geometry of objects and in-the-wild scene content.

Weakness 5 (related) Thanks for pointing these out, we will include these in the final version, in particular that 2-guidance was originally proposed in InstructPix2Pix and that our contribution is the generalization to N variables and its effectiveness for 4D NVS. As for training on mixtures of datasets, please see our response to Weakness 2.

As for the remaining question:

Question 1 (from-scratch vs. fine-tuning): The rationale behind this is non-technical. Due to our institution's policies around potential safety risks with training data used in external generative models, we were unable leverage prevalent pre-trained models for our research.

Thank you for your review. We address the reviewer’s concerns below:

Weakness 1 (terminology): We agree with the 3D+time interpretation of 4D. We think that the video-to-video translation examples on our website do illustrate 4D capabilities since we are able to generate the given input scenes (with dynamics) along new camera trajectories. We agree that the fidelity of temporal dynamics can be further improved but that is to be expected given we are one of the first to target this very challenging problem and the limited scale of our models. We strongly believe that fidelity in dynamics can be improved with further advancements in training mixtures and scaling up the size of these models.

Weakness 2 (data): Please note that the majority of our training data actually consists of unposed videos (30 million videos) which are rich in temporal dynamics and include motion of objects, people/animals, camera, etc., i.e. it is not true that the model is trained mostly on static data. Indeed, one of our main contributions is precisely how to achieve novel view synthesis combining camera and time control (which we have demonstrated e.g. in our video-to-video translation results originally included in the anonymized website) despite present limitations in the availability of 4D data.

Weakness 3 (keypoint distance):

While our proposed keypoint distance metric does not truly disambiguate camera and object motion, in practice, we find it useful for comparing the dynamics capabilities of models with good camera pose alignment (e.g. for tuning tgw in Fig 7 in supplementary) since it effectively serves as a measure of dynamics in that case. Please note that we also included FVD in the paper, which does correlate with the quality of temporal dynamics, i.e. keypoint distance was not the only metric included to assess video quality.

One question for the reviewer:

Additionally, the evaluation protocol is not clear enough -- is camera parameters the same for all the frames when generating the videos?

We are not sure what is precisely being asked here; if the reviewer can clarify a bit more, we are happy to answer any remaining questions.

Weakness 4 (measuring disentanglement):

We agree that disentangled metrics for camera and object motion would be useful for evaluating 4D generative models and that our included metrics are limited in their ability to do this. However, to our knowledge, such metrics do not yet exist in literature, but we welcome any specific suggestions for metrics to try.

Re fixing camera and changing t: If there existed a technique to reliably annotate static vs. dynamic masks for a generated video, one could then measure what the reviewer suggests. Unfortunately, research progress in motion segmentation has been limited to date, with results too inaccurate and noisy to attain a reliable metric. Qualitatively, we demonstrate in our video-to-video results that fixing the camera and changing t indeed results in rich motion.

Re fixing t and changing camera: The epipolar metrics we report (TSED) should capture this. Did the reviewer have other metrics in mind?

Weakness 5 (FiLM / Plucker): Thank you for the suggestion. We will attempt to include these ablations in the final version. These were too expensive to conduct in the rebuttal timeline against the models we originally trained for the paper submission.

As for some of the remaining questions:

Question 1 (MotionCtrl): The specific version of MotionCtrl that we use was trained on StableVideoDiffusion. We used this particular version because we needed a model that supports image conditioning (the prior version did not). However, since this was a follow-up of the original MotionCtrl, details on how this model was trained are not included in their paper. This makes it difficult to speculate on the potential causes for the reported performance. Unfortunately, finetuning this model on cRE10K was infeasible for us due to our institution's policy around potential safety risks in the training data used in the base model.

Question 2 (multi-guidance): Thank you for pointing this out, we will revise our paper to mention that 2-guidance had been previously proposed by InstructPix2Pix, which we here extend to N-variables and for a different task.

Question 3 (open sourcing): We will release our novel calibrated RealEstate10K data and allow it for open research use, including the evaluation code. While we are additionally pushing to open-source the model, we cannot promise this due to institutional policies.

Question 4 (related work): Thank you for pointing out this work; we will be sure to update our literature review to additionally make note of VD3D and other recent/contemporaneous work.

Thank you for your review. We address the reviewer’s concerns below:

Weakness 1: Thank you for the interesting suggestion of using Objaverse / ObjaverseDy [1]. Using this dataset may improve results, but it’s uncertain whether the gains from training on an object-centric dataset will transfer to real-world scenes. An empirical study is definitely warranted. Other ideas like conditioning on the amount of motion, or motion filtering (c.f. Stable Video Diffusion), could also be used.

• [1] SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency, Xie et al, 2024

As for single-image-to-4D, it remains a challenging task as we note in the paper. Qualitative samples suffer from low fidelity in dynamics.

Weakness 2 + Question 2: The loss of texture in long range generation is indeed quite surprising. One potential solution would be to leverage reward fine-tuning (e.g. [1]) to improve both image fidelity and detail. Supervised fine-tuning on a high-quality subset of our data (e.g. in [2]) may also be beneficial.

• [1] Directly Fine-Tuning Diffusion Models on Differentiable Rewards, Clark et al, 2024
• [2] Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, Blattmann et al, 2023

Question 1: While fidelity (FID, FDD) improves across the board when adding video data, certain pose alignment and reconstruction metrics get worse. One possible explanation is that training on posed data alone should be expected to perform better on pose alignment than a diluted training mixture with both posed and unposed video. To help demonstrate the benefits of including large scale unposed video in our training mixture further, we provide qualitative samples of our models trained with and without video data in the rebuttal section of the website.

Thank you for your review. We address the reviewer’s concerns below:

Weakness 1 (low res): While higher resolutions can be achieved with pixel space diffusion models, they typically require deeper cascades [1, 2]. Latent space models provide a simpler and more compute-efficient alternative. Since the time of the original submission, we also trained a 512x512 latent-space 4DiM. We have now added some qualitative examples for 3D generation with this model in our anonymized website. As such, we emphasize that our findings in this submission are also applicable, “as is”, to higher resolution generation with latent diffusion models.

• [1] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, Saharia et al, Neurips 2022
• [2] Imagen Video: High Definition Video Generation with Diffusion Models, Ho et al, 2022

Weakness 2 (MipNeRF360): The MipNerf360 dataset is out-of-distribution for our training data; the object-centric scene content and trajectories (spins around objects) in MipNerf360 are very different from those in any of our training datasets. Hence an evaluation may not provide meaningful insights. With appropriate training data added to our mixture, we are confident we would obtain high quality results on object-centric benchmarks, however, we leave that to future work as our work focuses on scenes and demonstrating the effectiveness of training with metric scale-calibrated data.

Question 1 (efficiency): Below we provide latency comparison of 4DiM and MotionCtrl.

While we evaluated both methods with a batch size of 1, there are some differences worth noting. MotionCtrl (576x1024) has a larger spatial resolution than 4DiM (256x256). However, it only generates 16 frames whereas 4DiM generates 32 frames. Furthermore, 4DiM was evaluated on a TPU v3 with 16GB HBM whereas MotionCtrl on a NVIDIA A100.

It is worth highlighting that with recent advances in efficient sampling of diffusion models, it is possible to run inference with much fewer denoising steps leading to much faster inference. However, we leave such optimizations to future work.

Question 2 (gray padding): To enable fair comparisons across models and datasets with different aspect ratios (e.g. MotionCtrl uses 576x1024, 4DiM works on square images, while LLFF is at 756x1008) we chose to evaluate on center square crops. However, MotionCtrl, owing to a wider aspect ratio than LLFF, was evaluated on a vertically cropped LLFF (to match the desired aspect ratio). The gray padding illustrates the cropped out region and was not included when computing the metrics. Please let us know if anything remains unclear here.

Question 3 (notation): Thanks for noticing this. We will amend the caption to make this more clear. More specifically, “gw” is standard “guidance weight”, while “tgw” stands for “timestamp”-gw, and “pgw” stands for “pose”-gw, where the latter refer to multi-guidance; varying the weight on these respective signals while keeping image guidance weight constant.