Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach

Thank you for your constructive feedback. We appreciate your recognition of the contribution of our work. Below, we address your questions and concerns:

Q1 Can the rigid loss be applied directly to object pixels in the RGB space instead of 3D point clouds?

We propose the use of point trackings due to the lack of 3D spatial information in RGB space. Directly applying rigidity losses in RGB space is suboptimal. For instance, in a video of a race car moving from a distance toward the camera, the object’s size in the RGB frame increases from a small region of pixels to a large portion of the image. Applying rigidity loss directly to these pixel regions fails to preserve the object’s shape and may even cause undesirable shrinkage. To address this, we introduce 3D point tracking as a prior to overcome this limitation.

Our ablation study in Figure 6(i) of the manuscript partially illustrates this issue, showing that training solely on RGB pixel differences does not achieve the desired 3D consistency. We agree that applying similar physical constraints directly in RGB space is a promising direction, provided that alternative methods can be developed to extract 3D information from RGB videos efficiently. We look forward to exploring such approaches in future work.

Q2 How does the method generalize to longer or more complex videos with multiple moving objects or occlusions?

The length of the generated video is constrained by the clip lengths in the training data. It can be generalized to predicting longer videos by scaling up the dataset. In our current training pipeline, we track foreground objects in the first frame and use point tracking to determine their 3D positions across time, accounting for possible interactions and occlusions between them. This added point modality enables our model to resolve multiple moving objects (see ‘Aston Martin’ example at 2’06’’ of the supplementary video) and handle contact-rich, multi-object scenarios with occlusion (see our task-oriented video examples). Even when an object is only partially visible or absent in the first frame, our model can still effectively handle it using the RGB modality. (For example, in the ‘tennis player’ case at 4’37’’, the player’s arm is not fully visible in the first frame, yet our model correctly infers its motion.)

To extend our approach to longer videos, more complex scenes and object interactions, we plan to scale up the training data with richer scene compositions and more accurate, higher-resolution 3D tracking.

Thank you for your constructive feedback. We appreciate your recognition of the novelty and effectiveness of our method. Below, we address your questions and concerns:

Q1 Can you quantify the extra compute cost your regularizer adds compared to the standard diffusion loss? How does that overhead grow as you increase point density?

We report the additional computational cost of our UNet model, trained on an A100 GPU with a batch size of 4 as follows: our 3D incorporation and regularization losses increase training times (∼9s per iteration, compared to ∼4.75s when fine-tuning on RGB video only) while maintaining comparable GPU memory usage (less than 2GB on top of ∼45GB). This additional computational cost is acceptable, as we are fine-tuning a pretrained video model rather than training one from scratch. Further statistics can be provided if needed. In our current design, the increased computational cost is primarily driven by the resolution of the training video rather than the density of the tracked points (see Q5 for more details).

Q2 To what extent might the regularizer inadvertently suppress desirable motion dynamics, and how might this be mitigated?

In our ablation studies (Figure 6 of the manuscript), we demonstrate that training with RGB alone lacks 3D awareness and results in failure cases, while using only the regularization loss leads to severe degradation of the RGB modality. In our approach, we adopt a joint optimization of diffusion and regularization losses, appropriately balancing them to be of comparable magnitude. This strategy yields a consistent decrease in regularization losses and a noticeable reduction in non-physical artifacts.

We observe that when the regularization loss significantly outweighs the diffusion loss (by orders of magnitude), it tends to suppress motion. To address this, we apply the regularization loss less frequently, for example, once every five iterations. This adjustment helps preserve larger motion dynamics while requiring more training steps.

Q3 Why choose a custom image subset over the public VBench split for evaluation?

We use a custom test set for VBench to ensure consistency across all evaluation metrics, including generic video quality (using VBench), physical common sense (using VideoPhy), ablation studies on model design, and user studies (reported in the supplementary material). The test images are sampled from a single batch of video clips (unseen by our model) from video dataset [1], with the initial frame randomly selected within each clip.

Below, we report the VBench results of our UNet-based model using their provided image set for your reference:

Q4 The technique seems readily applicable to text-to-video generation. Have you tested this scenario?

We chose image-to-video tasks for a more controlled comparison of physical correctness, as the same objects are present in the scene. While our method is adaptable to text-to-video models, we did not evaluate them, as this is not our main focus. We look forward to exploring this direction in the future. As an alternative, text-to-video applications could be demonstrated by incorporating an additional text-to-image step prior to our model, and we are happy to include such examples if requested.

Q5 How does the number of tracked points affect output quality? Can you achieve similar results with a much sparser point cloud, and what trade-offs arise?

First, we would like to clarify that although we initially obtain point trajectories through sparse 3D tracking, during post-processing, we fill in all untracked foreground pixels (i.e., pixels inside the foreground mask) by interpolating from neighboring tracked points. Furthermore, they are stored as a tensor of the same dimension of the video and used for subsequent computations. As a result, the point modality used in diffusion effectively becomes ‘dense’ tracking from the perspective of the video model. Thus, during training, the number of tracked points depends on how much of the reference frame is occupied by foreground objects, rather than on the tracking resolution used in data generation. On the other hand, during inference, we do not provide the point cloud, so there is no explicit control over point resolution.

Currently, our dataset is generated using a fixed sparsity setting in SpaTracker [2]. We acknowledge that the tracking sparsity used during data generation affects accuracy of 3D information, and likely affects our training, we did not further explore this as this requires costly regeneration of large-scale dataset; we leave further investigation of this aspect for future work.

Q6 The paper does not examine failure modes of the SpaTracker point-tracking stage or analyse tracking quality in the generated videos.

Since we do not prompt the model with point tracking during the inference stage, we are unable to directly assess whether generation failures are caused by inaccurate tracking. To evaluate the quality of predicted point tracking during training, we report straightforward MSE loss measurements between the generated points and the ground truth:

[1] Chen T S, Siarohin A, Menapace W, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13320-13331.

[2] Xiao Y, Wang Q, Zhang S, et al. Spatialtracker: Tracking any 2d pixels in 3d space[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 20406-20417.

Thank you for your constructive feedback. We appreciate your recognition of the effectiveness of our method. Please find our response to your question below:

Q1: Can the proposed method handle the cases where new objects appear in the mid of videos?

Yes, our method can handle these cases effectively. For objects not present in the first frame, our model treats them as background and relies on the RGB modality to predict them. For example, in the ‘Aston Martin’ case (2’06’’ in the supplementary video), a second race car appears midway through the video, despite not being visible in the first frame. Since the RGB modality is jointly trained with the point modality, our method can accurately handle such scenarios. A similar situation occurs when objects are only partially visible initially but are still correctly modeled in the RGB space—for instance, the human hand in the ‘tool’ example at 3’42’’. We would be happy to include additional visual results in the revision if needed.

Dear gf94, I see your review is positive, and maybe nothing has changed, but we need you to show up and say something, and then click the "mandatory acknowldgement" button.

Thank you for your constructive feedback. Below, we address your concerns:

Q1: Using only I2VGen-XL and CogVideoX 1.5 is insufficient to convincingly demonstrate general effectiveness.

We chose I2VGen-XL and CogVideoX 1.5 as representative models due to their distinct underlying architectures, which are based on UNet and Diffusion Transformer (DiT) frameworks, respectively. These two architectures are foundational, as most current video diffusion models share a similar backbone structure with one of them. We acknowledge that including experiments with additional baseline models would further strengthen our paper. However, due to constraints on computational resources, we leave this for future exploration.

Q2: VAE was not specifically trained on point-based data.

Using a VAE trained on RGB video to encode other modalities has been shown to be effective in prior work. For example, [1] uses a pretrained RGB VAE to encode depth maps without fine-tuning. Similarly, [2] leverages the RGB VAE to encode both depth and normal maps as input.

During our training, we transform the point modality to the same range as color intensities ([-1, 1]) and reshape it to match the input format of the video. We observed that even without fine-tuning, the VAE is capable of reconstructing the main shape and motion of point clouds, while the point regularization further reduces noise (as shown in Figure 4 of our main manuscript). Additionally, we report that the mean squared error (MSE) of the encoded-decoded point tensor is less than 0.01, comparable to that of the reconstructed video tensor. Further statistics on point augmentation and regularization are provided in Q4.

Q3: The second and third terms in the reconstruction loss may introduce a trade-off.

In our design of the reconstruction loss in Equation 1, the first term (position) serves as the primary component for controlling macroscopic motion dynamics. The second and third terms (velocity and acceleration) act as complementary losses (with small weights) to suppress high-frequency, microscopic noise. While we acknowledge that incorporating motion smoothness losses introduces a trade-off, the global dynamics are primarily governed by the position term, and we observe faithful reconstruction of ground-truth dynamics during training. Moreover, discouraging abrupt changes in velocity is often beneficial, as such changes can lead to artifacts in the generated videos. For example, in the ‘race car’ case (5’00’’ in the supplementary video), highly dynamic baselines exhibit non-physical morphing, whereas our result demonstrates physically plausible motion while preserving a high degree of dynamism.

Q4: Evaluation of the generated 3D points

We report the MSE of predicted 3D point trajectories in diffusion (using the input point tracking tensor as ground truth) as a quantitative metric, summarized below:

We observe that our network design effectively predicts 3D point trajectories after the augmentation stage and further reduces the error through the regularization stage.

[1] Shao J, Yang Y, Zhou H, et al. Learning temporally consistent video depth from video diffusion priors[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 22841-22852.

[2] Liu X, Ren J, Siarohin A, et al. Hyperhuman: Hyper-realistic human generation with latent structural diffusion[J]. arXiv preprint arXiv:2310.08579, 2023.

Dear Df51, we need you to show up and engage in the discussion phase. The guideline is:

• read the author rebuttal
• engage in discussions (reviewers must talk to authors, and optionally to other reviewers and AC - ask questions, listen to answers, and respond to authors)
• fill in "Final Justification" text box and update “Rating” accordingly (this can be done upon convergence - reviewer must communicate with authors first)

The (new) deadline is Aug 8, 11.59pm AoE