PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation

Thanks for your insightful comments for improving our work. We appreciate your positive feedback and your constructive suggestions, which has been a source of great encouragement for us. The following are detailed responses to your concerns.

Some of the phrasing was a tad vague and misleading.

We will make it specific in the beginning that we use the generated trajectories as input of a trajectory-conditioned model.

The claim on line 251 ("we are the first method to inject physics prior into a video model") is false/misleading.

Our model is conceptually different from PhysDreamer from a high level. PhysDreamer exploits physics knowledge from pretrained video models. It estimates physical properties of a trained 3DGS using the motion priors from video models with differentiable MPM. Differently, our model is injecting physical priors into video models by conditioning it on our generated motion trajectories. Moreover, we show in the supplemental that our model is also able to estimate physical properties given known motion trajectories. There are concurrent works, such as WonderPlay [1], Force prompting [2] (both appear after our submission), that attempt to improve the physics plausibility of video models. We will discuss these works and adjust our tones in the revision.

[1] WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions, Arxiv, May 2025 [2] Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals, Arxiv, May 2025

Beginning of Method Subsection

Thanks for your advice. We will adjust the beginning of the method section to a more logical order.

How to use Diffusion as Shader.

Diffusion as Shader (DaS) takes a tracking video as the input of the conditioning branch of ControlNet. Each frame of the tracking video is the projected 3D point trajectories of 2D anchor points (fixed grid positions for all videos) from the first frame. Our model generates the 3D point trajectories of the object, so we need to convert it to the tracking video of DaS for inference. Specifically, for each 2D anchor point in the first frame, we associate it with the nearest 3D point in the object. Then, we can project the 3D point trajectories into 2D and generate the tracking video frame by frame. We will add these details to the next version.

Explore GNN.

We chose a Transformer-based architecture due to its strong representational capacity and general effectiveness across a range of tasks. While we did not exactly test a GNN, we experimented with a variant of the Transformer that has a similar spirit of GNN, that is, each point only attends to its k-nearest neighboring points to model local relationship, which is similar to the local aggregation behavior of GNNs. We found that this approach doesn’t work well, particularly has severe error accumulation. This supports our decision to apply attention across all points, which may allow the model to better capture long-range dependencies and complex interactions.

It seems to me like the physics loss is the most complicated part of the paper (there's half a page of math there) and it seems the least important to the downstream performance? In Table 3, the row without physics loss is barely worse than the other rows, whereas the spatial attention and temporal attention are clearly crucial. Is this true? If this is the case, I would strongly recommend moving the Material Point Method section in the paper to the appendix, so the reader can black box the loss function, and move brief summaries of the experiment details from the appendix (e.g. construction of synthetic dataset, design of human study) into the body of the paper.

The physical loss helps the model to generate more physics plausible trajectories. In Figure 7, we show four examples that using physical loss helps generate more physical plausible trajectories. For these cases, if there is no physics loss, generating videos with these trajectories will not look consistent with the input force. We appreciate your suggestion and will move some experiment details into the main paper.

Discussion of failure cases due to poor 3D point cloud extraction / our diffusion model / Diffusion as Shader

Classification of failure cases: (1) Failure due to initial poor image-to-3D reconstruction is very rare. Single image-to-3D produces reasonable geometry overall and is unlikely to yield severely distorted or implausible shapes (note that we do not need appearance information). While minor artifacts (e.g. geometry not perfectly smooth or noise points on the surface) may occur, they have minimal impact on our results. We achieve such robustness to geometric variations because we trained the trajectory generation network on the diverse Objaverse dataset and applied surface noise augmentation during data generation. Moreover, the video model only needs point trajectories as motion signals and will generate the details. (2) Failure cases of our diffusion model: we compared our results with MPM simulation on the Objaverse testing set. We found that cases with low vIoU typically involve thin structures or scenarios where our model predicts global motion but fails to accurately capture internal deformations. (3) Failure cases due to video model: we found in some cases, especially when the trajectories might conflict with the prior of video model, DaS cannot fully follow our trajectories. For example, when the user input for the object material is very stiff, but it appears soft according to RGB information. Then the resulting video might still be stiff. The video model might also hallucinate some unexpected content in the occluded region. For example, for an animal, it might generate five legs.

We will add these discussions and visualization of failure cases in the revision.

Did the video prior make certain classes of demos more or less likely to succeed? Or more generally, what is the significance of the choice of the video prior for the base video model?

As discussed above, when the trajectories conflict with the prior of the video model, DaS might not fully follow our trajectories. We selected CogVideoX, as it was the strongest available model at the time of our submission. We think that using more powerful video models (especially those that have better world knowledge) will generally bring better results, e.g. generate more plausible details for occluded parts.

Line 264.

Thanks for pointing this out. We agree that using “physical conditioning” is more accurate.

Thank you for your insightful comments and questions, which is crucial to improving the quality of our work. The following are detailed responses to your concerns.

The model takes inputs similar to MPM and the necessity of training a neural trajectory predictor is not clear.

We would like to first clarify that our user inputs are less than MPM. We only require minimal information: force and material information, while MPM requires many other hyperparameters: grid size, damping scale for grid velocity, frame dt and substep dt etc. Besides this, using our model has several other benefits: (1) More robust: we use a traditional simulator to generate training data by tuning their parameters and filtering failure cases. The filtered training data helps us train a robust neural simulator, even for cases where classical simulators struggle. For example, we found that when using MPM to simulate a waterdrop falling onto the ground from different heights, there are around 10% failure cases (due to numerical instabilities), while a trained neural network is more stable; (2) Unify different simulators and generalization to different materials: we unified four materials in one unified network by creating data from both MPM and a rigid body simulator. If using an actual simulator, users would need to switch simulators for different materials. Moreover, due to our unified point representation, users do not need meshes to simulate rigid objects, while standard rigid body simulator requires mesh as input; (3) Inference speed: our base model and large model costs 1s and 3s for forward diffusion sampling, while MPM requires more than 4s. For one backward step, MPM requires 3min, while our method only needs 0.1s (no diffusion).

It is unclear how the model accounts for external elements like the ground plane if only the foreground object is segmented. For example, in Figure 1, how does the model determine that the penguin is supposed to collide with the table while the plane should stay in the air when rotating?

Our model takes floor condition as an additional condition (Line 172 of the paper) and the floor condition is calculated automatically from the input. The model will automatically determine whether the points will collide with the boundary because the generated point trajectory is conditioned on the boundary. To calculate the floor condition, we use grounded segmentation or user input to select the floor and use VGGT to obtain the 3D points of the whole input image (including the floor and object), as our image-to-3D only reconstructs the foreground object. Then we align the coordinate systems of VGGT and our image-to-3D using the correspondences of the foreground object. Finally, we obtain the value of the floor using principal component analysis based on the transformed ground plane points. If there is no floor detected, the value will be set to a placeholder.

In Figure 1, we would like to clarify that the penguin won’t collide with the table because of the upward force. The plane case doesn’t contain a floor.

It is unclear whether the method would generalize to multi-object interactions.

As acknowledged in the limitations section, modeling multi-object coupling or complex boundary conditions is beyond the scope of this work. To the best of our knowledge, no existing neural network approach currently addresses this challenge effectively. We did a preliminary experiment by creating a small dataset that has an object dragged towards a cube and colliding with the cube from different angles and distances. We trained our model on it and achieved 93.70% vIOU on the held out testing set. This preliminary experiment demonstrates the potential of our framework for multiple objects. Nevertheless, fully addressing this problem remains nontrivial and future work could explore incorporating stronger physics-informed losses to better handle complex deformations, as well as developing scalable and modular architectures that can flexibly support different numbers of interacting objects.

Comparison with Go-with-the-flow

We run Go-with-the-Flow using the same test set and test protocol as in our paper. Following the official Github Repo, we use the cut-and-drag tool for motion control and then perform video generation. We observed that it supports only limited motion control, restricted to translation, rotation, and scaling of the original object, which hinders its ability to model complex deformations in elastic objects or morphological changes in materials like sand. Furthermore, it lacks support for 3D motion, such as the inward rotation in our UFO test case. These constraints result in Go-with-the-Flow performing worse than our method:

Relevant papers.

Thanks for pointing them out. We will add them in the revision.

Dear Reviewer F6p4,

I notice that the authors have submitted a rebuttal. Could you please let me know if the rebuttal addresses your concerns? Your engagement is crucial to this work.

Thanks for your contribution to our community.

Your AC

We thank the reviewer for the in-depth comments and helpful discussions.

Substep size in Water Drop

We use a fixed substep size $\Delta t=10^{-4}$ in the MPM simulation, a conservative choice to avoid numerical instability following the common configuration in the widely used code of PhysGaussian [1]. We sincerely thank reviewer for pointing out adaptive time stepping based on CFL condition. We admitted that choosing substep according to CFL can help prevent numerical instability. However, to our knowledge, CFL number is another hyperparameter and choosing an effective CFL number is not trivial, as studied by many research works such as [2], [3].

CFL check suggests that there is a tradeoff between robustness and speed of MPM. In our waterdrop example, when using the common fixed substep size $\Delta t=10^{-4}$, the simulation will still fail in $10$% cases. After incorporating CFL check ($\max_ p \left\lVert\mathbf{v}_ p\right\rVert\le C_ {cfl}\frac{\Delta x}{\Delta t}$) and the fluid wave speed restriction $\Delta t \le C_{fluid} \frac{\Delta x}{\sqrt{\kappa/\rho}}$ (where we set $C_{cfl}=C_{fluid}=0.4$ following the convention) during MPM simulation, the substep size will shrink to $5 \times 10^{-5}$ for previous failure cases. The simulation can always success then but has lower speed, which takes more than $15$s to finish the simulation.

Handle Multiple-object interaction

Thanks to the point representation, our method is also inherently potential for multi-object interaction. MPM achieves multi-object interaction by representing scene dynamics as point movement, and our model also predict point trajectories. Our previous response has shown our model's potential for multiple objects on a simplified setting. We will add that to the revised version. Future work might consider generate more diverse multi-object data, design better losses and architectures to fully address this problem, which is out of scope for this paper.

"Major advantage of the proposed neural simulator lies in the speed"

Our method has other advantages besides speed. As mentioned in previous response, we have advantages of needing less hyperparameters, unifying other simulators, etc.

Inference speed across materials

Our method still has faster speed than MPM even when it uses adaptive time stepping based on CFL condition. The below table shows the inference speed comparison of our model and MPM (we set the initial substep size to $\Delta t = 3 \times 10^{-3}$ and set the CFL condition to \max_ p \left\lVert\mathbf{v}_ p\right\rVert$$\le 0.4\frac{\Delta x}{\Delta t} following [4], shrinking $\Delta t$ by 2 when violating the CFL condition) on elastic material with varying Young's modulus:

Since the uncertainty in our trajectory generation is relatively small, we are able to do 4-step inference, while still achieving strong performance. Thanks the reviewer for this question, which helps us demonstrate the speed advantage of our method.

MPM benefits from techniques of numerical analysis, such as adaptive stepping based on CFL condition, while neural-based methods like our model can also be benefitting from many neural network accleration and diffusion model speeding up strategies, such as 1-step diffusion distillation, model quantization etc. Besides, our model's backward speed is much faster, which is helpful for inverse problems.

Once again, we sincerely thank the reviewer for the constructive feedback and the effort in improving this work. We hope these additional clarifications demonstrate the advantages and potentials of our method. We are also open to further suggestions and would be happy to incorporate additional improvements. We hope our responses address your concerns and would greatly appreciate your reconsideration of the overall evaluation.

[1] Physgaussian: Physics-integrated 3d gaussians for generative dynamics, Xie et al., [2] A precise critical time step formula for the explicit material point method, Ni et al., [3] Effective time step restrictions for explicit MPM simulation, Sun et al., [4] IQ-MPM: an interface quadrature material point method for non-sticky strongly two-way coupled nonlinear solids and fluids, Fang et al.

Thanks for your insightful questions, which is crucial for improving the quality of our work. The following are detailed responses to your concerns.

Advantages of a learned generative physics model over an actual simulator.

Using neural simulator has several benefits: (1) More robust: we use a traditional simulator to generate training data by tuning their parameters and filtering failure cases. The filtered training data helps us train a robust neural simulator, even for cases where classical simulators struggle. For example, we found that when using MPM to simulate a waterdrop falling onto the ground from different heights, there are around 10% failure cases, mostly due to numerical instabilities, while a trained neural network is always stable; (2) Easy-to-use: our model requires minimal input from user, including only force direction and material. In contrast, MPM requires many other hyperparameters, including grid size, damping scale for grid velocity, frame dt and substep dt etc; (3) Unifying different simulators and generalization to different materials: we unified four materials in one unified network by creating data from both MPM and a rigid body simulator. If using an actual simulator, users would need to switch simulators for different materials. Moreover, due to our unified point representation, users do not need meshes to simulate rigid objects, while standard rigid body simulator requires mesh as input; (4) Inference speed: our base model and large model costs 1s and 3s for forward diffusion sampling, respectively, while MPM requires more than 4s. For the backward step, MPM requires 3min, while our method only needs 0.1s (no diffusion). Fast backward speed is especially beneficial for inverse problems.

Results of using a traditional simulator.

We use the point trajectories from traditional simulators to generate videos on our testing set and use GPT for evaluation, the results are shown in the table below. Using our model is even slightly better than using the traditional simulator in certain metrics. The visual results mostly look similar. We will update these results in the paper. As such, using our model not only achieves great video generation results on successful cases of MPM, but also has the 4 benefits mentioned above.

Common failure cases of the method.

(1) Failure cases of our diffusion model: we compared our results with MPM simulation on the Objaverse testing set. We found that cases with low vIoU typically involve thin structures or scenarios where our model predicts global motion but fails to accurately capture internal deformations. (2) Failure cases due to video model: we found in some cases, especially when the trajectories might conflict with the prior of video model, DaS cannot fully follow our trajectories. For example, when the user input for the object material is very stiff, but it appears soft according to RGB information. Then the resulting video might still be stiff. The video model might also hallucinate some unexpected content in the occluded region. For example, for an animal, it might generate five legs. (3) Failure cases due to image-to-3D is very rare.

We will add these discussions and visualization of failure cases in the revision.

How did you settle on using N=2048 points for the point-cloud representation?

We choose N=2048 considering that: (1) Prior works on 4D generation and reconstruction (PhysDreamer [1], SC-GS [2], MoSCA [3]) demonstrated that real-world motions can be represented with a sparse number of basis or control points; (2) The point trajectory is only providing a physics deformation signal for the video model and details will be generated by the video model itself. Our choice of 2048 points for the foreground object is further motivated by Table 3 in Diffusion as Shader (DaS), where the authors show that using 4900 points (including background static points) achieves similar results with more points (8100). In the Table above, we tested on simulated trajectories of more points as the input for the video model, results also show that using more points didn’t bring significant difference in performance.

[1] PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation, Zhang et al., ECCV 2024 [2] SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes, Huang et al., CVPR 2024 [3] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds, Lei et al., CVPR 2025

Misc: Tab.1 seemingly contains a faulty looking value for DragAnything

Thanks for pointing this out. The number should be 2.8.

Dear Reviewer yGS5,

I notice that the authors have submitted a rebuttal. Could you please let me know if the rebuttal addresses your concerns? Your engagement is crucial to this work.

Thanks for your contribution to our community.

Your AC

Thank you for your insightful comments, which are crucial to improving the quality of our work. The following are detailed responses to your concerns.

Material types and modelling complex phenomena

Our four material types (elastic, sand, rigid, plasticine) already cover a wide range of objects. As acknowledged in the limitations section, modeling multi-object coupling or complex boundary conditions is beyond the scope of this work. To the best of our knowledge, no existing neural network approach currently addresses this challenge effectively. However, since it was requested and to assess our framework’s potential in this setting, we conducted a preliminary experiment, where we created a small dataset that has an object dragged towards a cube and colliding with a cube from different angles and distances. We trained our model on it and achieved 93.70% vIOU, demonstrating the potential of our framework for multiple objects. Nevertheless, fully addressing this problem is nontrivial and future work could explore incorporating stronger physics-informed losses to better handle complex deformations, as well as developing scalable and modular architectures that can flexibly support different numbers of interacting objects.

Computational efficiency

Compared to current AI models, the experimental setup of our trajectory generation model is not that resource-intensive. Our base/large model only needs 8L40 48G GPUs for training 30/80 hours, which is accessible in most academic settings. Our trajectory generation model also require less resources than our baselines for image-to-video generation: DragAnything needs A100s (training time not available) to directly finetune a much heavier video model (SVD); Wan2.1 and CogVideoX require at least hundreds of GPUs for training.

How do the high costs and instabilities of physics simulators (for training data generation) impact the learned model's generalization or physical fidelity?

The high costs and instabilities are minor issues and have little effect on our model’s generalization and physical fidelity. Although the simulator runs slower than our model, we can utilize multiple low-end GPUs (e.g. 2080Ti) and generate data efficiently. The instability is mostly due to numerical issues, we filter them out by checking out of boundary values and NaN values. Our final dataset is high-quality, large-scale and contains diverse categories and geometries, which enables our trained model to be generalizable and stable. We will release the dataset.

Does the fixed 2048-point representation limit expressive power or fidelity when handling complex geometries or fine deformations?

Prior work on 4D generation and reconstruction (PhysDreamer [1], SC-GS [2], MoSca [3]) has shown that real-world motion can be effectively captured using sparse basis or control points. Also, our choice of 2048 points for the foreground object is motivated by Table 3 in Diffusion as Shader (DaS), where the authors show that using 4900 points (including background static points) achieves similar results with more points (8100). From our observations, our method, trained on the diverse Objaverse dataset, performs reliably using 2048 sparse points, and we have not observed notable issues in fidelity or expressiveness. To further validate this, we tested using simulated trajectories of different numbers of points as the input for the video model. We follow our paper’s protocol to use GPT for evaluation and present the results in the table below. There is indeed minimal difference (GPT itself might also have some errors) when using different numbers of points to drive the video.

We agree that sparse points may not model extremely fine deformations. However, in such cases, the video model itself might also not be able to generate those delicate details. To the best of our knowledge, no existing methods for physics-grounded video generation are capable of handling such fine-grained deformations effectively. We will discuss this challenge for all the existing methods (including our work) in the limitation and leave this for future work.

[1] PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation, Zhang et al., ECCV 2024 [2] SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes, Huang et al., CVPR 2024 [3] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds, Lei et al., CVPR 2025

Limitations of single image to 3D and its errors on subsequent steps

The single-image-to-3D step produces reasonable geometry overall and is very unlikely to yield severely distorted or implausible shapes. While minor artifacts such as surface noise may occur, they have minimal impact on subsequent steps. We achieve such robustness to geometric variations because we trained the trajectory generation network on the diverse Objaverse dataset and applied surface noise augmentation during data generation. Moreover, the predicted point trajectories are used as motion signals to guide the video model, and the video model will generate fine visual details with this guidance.

Dear Reviewer vXqE,

I notice that the authors have submitted a rebuttal. Could you please let me know if the rebuttal addresses your concerns? Your engagement is crucial to this work.

Thanks for your contribution to our community.

Your AC

Dear Reviewer vXqE,

Thank you for your thoughtful and constructive reviews. We have submitted our rebuttal and hope it addresses your concerns effectively. We would greatly appreciate it if you could join the discussion and share any further thoughts or questions.

We are happy to provide additional clarifications or details as needed and are committed to improving the paper based on your feedback.

Best regards, The Authors