4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models

Context embedding. First, the dataset for “static real-world objects with circular camera trajectories”, consists of data similar to the MV-ImagNet, with real-world backgrounds. Second, the use of context embedding is critical, as it controls the video model to generate freeze-time-like video. Without this step, the model predominantly generates dynamic videos whenever nonrigid subjects are included in the text prompt, regardless how the text prompt is engineered. The usage of context embedding can definitely be abandoned in the future when the video models are equipped with better camera-control and better consistency to input prompts.

Compare with AYG and correct description of Dream-in-4D. We thank the reviewer for pointing out our mistake in wrongly describing Dream-in-4D as a 3D GS-based method. We will correct this error accordingly. Additionally, we have included a comparison against AYG, as shown in Figure 3 and Table 1 in the PDF. We find that our overall assessment of the advantages of our method over prior object-centric methods remains valid.

Consistency between reference video and freeze-time-video. To minimize inconsistency, we generate reference videos without camera motion. This approach reduces disoccluded regions caused by camera movement, which is a major source of potential inconsistency between the reference video and the freeze-time video. Given this setup, we find the low-res video model is able to generate mostly consistent pixels in the remaining disoccluded regions caused by object motion, partially due to any inconsistency is hard to notice in low-res video. To further improve consistency, we merge two videos into a single video when feeding it into the video upsampler, leveraging the temporal self-attention in the upsampler to improve consistency.

Applying recent 3D scene generation works. Yes, we believe recent image-mv models trained for 3d scene generation can be a drop-in replacement for the part of generating freeze-time video. Unfortunately, none of these concurrent approaches (i.e., RealmDreamer and CAT3D) are open-sourced yet, so it is not clear how they perform compared to the video model we employed. We will include a discussion of these methods in the revised version.

Video editing. Thanks for the reminder. We mentioned the flexibility in selecting and editing videos in our contributions based on our generation-via-reconstruction pipeline that allows users to generate, select, and edit the reference video they want. We will include our results of applying simple video editing techniques, such as face-swapping or attributes manipulation, in the revised draft of the supplementary material.

Latency comparison. In Ln. 64, we mentioned that our overall runtime is approximately 1.5 hours on a single A100 GPU, compared to over 10 hours for 4Dfy and Dream-in-4D. To provide additional latency analysis, we break down our runtime as follows: approximately 10 minutes for reference and freeze-time video generation, about 2 minutes for running COLMAP, around 20 minutes for reconstructing the canonical GS, and less than 1 hour for the remaining reconstruction steps with SDS.

Additional qualitative metrics. We included additional metrics to quantitatively evaluate the rendered videos (see Table 1 of the PDF). We employ the XCLIP score [1], a minimal extension of the CLIP score for videos. We also run VideoScore [2], a video quality evaluation model trained using human feedback, which provides five scores assessing visual quality, temporal consistency, text-video alignment, and factual consistency. Our method significantly outperformed other methods in these metrics. However, we remain conservative about the effectiveness of these metrics since they have not been thoroughly studied for text-to-3D/4D generation.

Additionally, we excluded the FVD metric, as computing it with a limited number of rendered videos is statistically not meaningful. Evaluating with real-world multi-view videos is also not easily applicable to our pipeline. This is because we made specific design choices, such as requiring the reference video to have no camera motion to avoid inconsistency between the reference video and the freeze-time video. This assumption does not hold for casually captured dynamic videos.

Analysis of freeze time view deformation field. The reason for introducing view-dependent deformation (referred as per-view deformation in the paper) is to treat inconsistency in the generated freeze-time videos as geometric deformation. Without the view-dependent deformation, the canonical GS tends to underfit the freeze-time video, leading to blurrier regions, particularly in the background, where the video model often produces geometrically incorrect results.

However, while view-dependent deformation can improve underfitting, it can also lead to overfitting to the reference video, resulting in noisy reconstruction. To mitigate this, we employ a regularization loss of the magnitude of the deformation (as mentioned in lines 180-186) to prevent overfitting. Through empirical testing with a few samples, a weighting of 0.01 was found to be effective and is used consistently throughout the experiments.

Additional visual comparisons are provided in Figure 5 of the PDF to support this analysis. These comparisons demonstrate the effects of removing either the per-view deformation or the small motion regularization.

Does freeze time view deformation field apply to other temporal frames? No, it is only applied to fit the frames from the freeze-time video. It is not used for any other frames.

How does the video model handle different sampled fps? The video model we used can take FPS as input. Additionally, as described in Ln. 142, it is a frame-conditioned model capable of performing either autoregressive generation or frame interpolation. Therefore, theoretically, our method can handle longer videos in the cost of longer processing time in the following reconstruction stage.

Multiview video prior. Yes, we also strongly believe that developing a generalizable multiview video diffusion model is one of the most promising directions for 4D generation. We briefly mentioned potential implementations, such as cross-frame attention, in Ln. 308. In the new draft, we will include a discussion of more concurrent efforts in multiview video generation. At this point, however, our method serves as a pioneering effort in demonstrating the possibility of generating realistic 4D scenes.

References:

[1] Expanding Language-Image Pretrained Models for General Video Recognition.

[2] VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation.

Baseline by 3D scene generation + 4D object-centric generation. Although this is a straightforward baseline, implementing it with high quality is challenging.

• First, inserting 3D objects with realistic and physically correct placement is not trivial. Common issues include floating objects, misaligned scales between objects and background, and unrealistic scene layout.

• Second, generating object motions that align well with the backgrounds is difficult, such as ensuring a generated target subject sits on a sofa rather than stands on it.

• Third, complex procedures are needed to relight objects to match environment lighting.

There exists several recent works either specialized in human-environment interaction [3] or generating simple 3D scene layout [4]. Currently, we are not aware of methods able to achieve all above points automatically. We tried our best to create a few baseline results with heavy manual efforts, as shown in Fig. 2 in the PDF. We create the 3D background by removing the object from our generated freeze-time video. We then employ 4Dfy to generate the foreground objects. Finally, we manually insert the objects into the background with the most plausible position and scale we can find. Despite these efforts, we find the inserted object appears disconnected from the background, especially when compared to 4real’s results. For example, in the first example, the fox stands on the sofa instead of lying on the sofa; in the 2nd example, the racoon appears floating above the DJ equipment, and the lighting is inconsistent with the purple/blue environment lighting; and in the last example, the racoon appears popping out of the newspaper instead of lying on it.

Overall, we think that 3D scene generation + 4D object-centric generation could be a viable alternative. However, this approach requires significant and complex research and engineering efforts. At this point, directly lifting 3D from generated videos, as demonstrated in our pipeline, appears to be a more generalizable and straightforward method. This is because the video model inherently produces realistic layouts, motion, and lighting. Furthermore, we envision that incorporating concurrent advancements in 4D reconstruction (e.g., MoSca [1]) and 4D object-centric generative priors (e.g., SV4D [2]) will further improve the quality of our pipeline.

Consistency between reference video and freeze-time-video. To minimize inconsistency, we generate reference videos without camera motion. This approach reduces disoccluded regions caused by camera movement, which is a major source of potential inconsistency between the reference video and the freeze-time video. Given this setup, we find the low-res video model is able to generate mostly consistent pixels in the remaining disoccluded regions caused by object motion, partially due to any inconsistency is hard to notice in low-res video. To further improve consistency, we merge two videos into a single video when feeding it into the video upsampler, leveraging the temporal self-attention in the upsampler to improve consistency.

Failure cases. As stated in Ln. 303, our method may fail under conditions such as rapid movements, the sudden appearance and disappearance of objects, and abrupt lighting changes. We will include failure samples in the new draft. To clarify what we mean by “good results,” we initially generated approximately one hundred results with various styles of reference videos without human filtering to understand the limits and ideal use cases of our method. After analyzing these results, we identified our limitations as mentioned above and retained the set of “good” videos—those with moderate movements, no sudden appearance or disappearance of objects, and no significant lighting changes—as example results for the intended use case of our method. The selection process can be entirely automatic via heuristic rules based on our observations.

Highlight of Novelty. In the 3D generation field, the paradigm where a 3D reconstruction following multiview image generation is one of the mainstream approaches. Although the idea is straightforward, achieving convergence to this solution and making it work effectively is not trivial. Similarly, the proposed generation-via-reconstruction pipeline for 4D generation is non-trivial in practice. Naively combining existing video models and 4D reconstruction techniques does not yield desirable results. The major contributions and novelties of our approach are (1) a workflow to obtain plausible freeze-time videos from reference videos, given current video models of limited capability, (2) a deformation field to handle the inconsistencies in imperfect freeze-time videos, and (3) a joint temporal and multiview SDS to help reconstruct temporal deformation. With these proposed techniques along with other efforts to adopt various existing components and regularization effectively, we achieve the presented realistic 4D generation results.

Reference:

[1] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds.

[2] SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency.

[3] GenZI: Zero-Shot 3D Human-Scene Interaction Generation

[4] GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting

Interaction between objects and environments. Our approach aims for a generalizable approach to generate realistic interaction between objects and environments, which includes natural multi-object placement, realistic environmental lighting effects, and relative motions between objects and environment. We acknowledge the ambiguity of the term “interaction” and will clarify these points. As per the reviewer’s request, we provide additional samples (see Fig. 4 in PDF) with moderate global motion between the object and the environment, such as “a storm trooper walking forward …” and “a dog running from left to right in front of a chair”. While our method currently faces challenges in reconstructing content with more rapid motion (as stated in Ln. 303), this limitation may be alleviated using concurrent advancement in 4D reconstruction method with long-term point tracking, such as MoSca [1].

Baseline by 3D scene generation + 4D object-centric generation. Although this is a straightforward baseline, implementing it with high quality is challenging.

• First, inserting 3D objects with realistic and physically correct placement is not trivial. Common issues include floating objects, misaligned scales between objects and background, and unrealistic scene layout.

• Second, generating object motions that align well with the backgrounds is difficult, such as ensuring a generated target subject sits on a sofa rather than stands on it.

• Third, complex procedures are needed to relight objects to match environment lighting.

There exists several recent works either specialized in human-environment interaction [4] or generating simple 3D scene layouts [5]. Currently, we are not aware of methods able to achieve all above points automatically. We tried our best to create a few baseline results with heavy manual efforts, as shown in Fig. 2 in the PDF. We create the 3D background by removing the object from our generated freeze-time video. We then employ 4Dfy to generate the foreground objects. Finally, we manually insert the objects into the background with the most plausible position and scale we can find. Despite these efforts, we find the inserted object appears disconnected from the background, especially when compared to 4real’s results. For example, in the first example, the fox stands on the sofa instead of lying on the sofa; in the 2nd example, the racoon appears floating above the DJ equipment, and the lighting is inconsistent with the purple/blue environment lighting; and in the last example, the racoon appears popping out of the newspaper instead of lying on it.

Overall, we think that 3D scene generation + 4D object-centric generation could be a viable alternative. However, this approach requires significant and complex research and engineering efforts. At this point, directly lifting 3D from generated videos, as demonstrated in our pipeline, appears to be a more generalizable and straightforward method. This is because the video model inherently produces realistic layouts, motion, and lighting.

Could up-to-date text-3D models improve realism. The reason why 4Dfy and dream-in-4d is less realistic is because relying on text-MV model (i.e. MVdream) trained with synthetic data. As a result, the model is biased to generate synthetic-style objects, even when “realistic style” is added to the text prompt (see Fig.1). Some text-3D methods achieve a certain level of realism by using generative models trained with real data, either performing SDS with text-image model [2], or training image-MV model using real data [3]. In this regard, our approach can be loosely considered an updated version of these methods with an improved 3D prior. We will include this discussion in the next draft.

Does the deformation model also time-varying appearance? No, we only model geometry deformation. We find that incorporating time-varying appearance worsens reconstruction results by making the 4D reconstruction problem more ambiguous.

We thank the reviewer for the valuable comments, and will modify accordingly.

Reference:

[1] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds.

[2] HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance

[3] CAT3D: Create Anything in 3D with Multi-View Diffusion Models

[4] GenZI: Zero-Shot 3D Human-Scene Interaction Generation

[5] GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting

Ablation of finetuning with synthetic data. This is a great suggestion to rigorously prove the benefit of using models trained with real world data. However, conducting such experiments requires significant GPU hours to reach a meaningful conclusion, which we are unable to fulfill for rebuttal. Instead, we refer to results from text-MV image models finetuned using synthetic data, e.g. MVDreamer. As shown in Fig 1 of the PDF, images generated by MVDreamer are cartoonish, even when “realistic style” is included in the input text prompt.

Importance of training with static multi-view data. Training video models with static multi-view data is crucial not only because it supplements the training set with videos of static scenes but, more importantly, because it allows us to learn a context embedding (associated to the multi-view dataset) that serves as a tool to control the model to generate freeze-time-like video. Without this step, the model predominantly generates dynamic videos whenever nonrigid subjects are included in the text prompt, regardless how the text prompt is engineered.

Details of training data. The full training set combines a dozen of datasets, each corresponding to different type/style of videos, and is associated with a context embedding. For the static multi-view dataset, it is mainly created from MVImageNet, and sampled based on a predefined ratio during training. The ratio is set according to the number of video hours in each dataset, which results in roughly 1% for the static multi-view dataset.

Efficiency contribution claim. It is partially correct that one source of efficiency comes from the video model we used. However, the main source of efficiency, as stated in Ln. 58, is that we "transform the generation problem into a reconstruction problem and reduce reliance on time-consuming score distillation sampling steps." Specifically, as described in Ln. 263, during each stage of the reconstruction procedure, we perform SDS only during the last 5,000 iterations out of the total 20,000 iterations. This choice significantly reduces runtime, as each SDS iteration takes over 20 times longer than simply minimizing reconstruction losses.

Is choosing amount of pose variation automatic? This part is automated in our experiments. First, we perform auto-regressive view extension until the video reaches a maximum of 72 frames, which is often more than enough for 180+ degree coverage. Next we run Colmap with the generated video. If Colmap fails to converge, indicating the current video is significantly geometrically inconsistent, the video is automatically shortened until Colmap is able to reach a solution. We will include this in the supplementary.

Does Colmap work for all the dynamic videos generated? We recognize the challenge of pose estimation for dynamic videos. Therefore, we purposely generate dynamic videos without camera motion. This is achieved by using text prompts “camera is static”. We will include this detail into the supplementary.

Failure case. As stated in Ln. 303, our method may fail if there are “rapid movements, the sudden appearance and disappearance of objects, and abrupt lighting changes”. We will include failure samples in the new draft.