Seeing the Wind from a Falling Leaf

We sincerely appreciate your valuable comments and suggestions. Your feedback has helped us refine our work, and we are excited to clarify and discuss each of your points in the following.

Q1: Several existing studies [a-b] have demonstrated the ability to estimate a velocity field from video observations, which is analogous to the proposed method that estimates a force field (assuming constant mass). It is recommended that the authors discuss the differences between these studies and the proposed method. Additionally, they should present quantitative comparisons to determine which approach yields more effective results.
References
[a] NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos. NeurIPS 2023.
[b] FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocity. CVPR 2025.

A1: The key difference between our method and the mentioned NVFi and FreeGave is that these methods don’t rely on explicit and differential physical processes to model physical interactions. Specifically, these methods rely on a neural network that mimics the physical interactions and predicts particle velocities, with a physics-based regularization term to constrain the outputs. In contrast, our method leverages an explicit physical process to model the physical interactions and learn the force field via a differential physics engine. In the revision, we will add a dedicated discussion of both NVFi and FreeGave to clarify these distinctions. Thank you!

Q2:It is challenging to reconstruct the 3D scene given only the first frame of the entire video. How does the method handle scenes that include object occlusions? Could the rest frames provide some information for reconstructing the 3D scene?

A2: Our method relies on the first frame to initialize the 3D reconstruction of the scene, as described in Section 3.1. The 3D Gaussians representing the objects are initialized using pixel-aligned point clouds extracted from the first image via a pre-trained metric-depth model. This initialization is performed once using the first frame and does not get updated by subsequent frames.

While the first frame provides the essential geometric initialization, we also mention in Section 3.1 that we have explored both single-view (line 122) and multi-view (line 127) reconstruction techniques. In the case of multi-view input, additional perspectives can further refine the 3D model, but the geometry itself is still primarily initialized from the first frame.

Regarding object occlusions, the method handles them through sparse tracking of keypoints, which allows for robust motion tracking even in the presence of partial occlusion. By focusing on key points that are less likely to be occluded, we can maintain the accuracy of the 3D reconstruction and force estimation despite object occlusions.

Q3: In Section 3.1 of the manuscript, the authors mainly discuss the modeling of foreground objects. However, it is unclear whether the background is modeled in the same way as the foreground objects. If that is the case, how does the method obtain the physical parameters (e.g., mass and Young's modulus) for the background particles? Additionally, in the supplementary videos, while the original clips show noticeable background motion, the videos generated through physics-based motion editing do not display any background movement. Can the authors provide some clarification on this?

A3: In our framework, we specifically model only the foreground objects and not the background. The background is treated as a static element in our simulation. The physical properties such as mass and Young’s modulus are assigned only to the foreground objects, and background is not modeled with these physical parameters.

Regarding the lack of background motion: for all demos we assume the background is static. We extract the background from the first frame, inpaint occluded regions, and then composite our simulated foreground motion on top. This choice keeps the focus on foreground dynamics and sidesteps the additional complexity of modeling background physics.

We have indeed experimented with segmenting and inpainting every frame, then compositing our simulated objects into those reconstructed backgrounds. Current off-the-shelf inpainting techniques, however, still exhibit minor temporal flicker across frames, which can distract from our main contribution. For clarity of presentation, we therefore adopt the simpler static-background pipeline in the demos, allowing the reader to focus on the foreground physics.

Q4: In Section 3.3 of the manuscript, what is the consideration for applying external forces on each particle before the particle-to-grid step, rather than applying them on each grid node, similar to internal forces? For Equation 13, have the authors considered alternative interpolation strategies (e.g., K nearest neighbors), and how do they perform? What is the ratio between the number of x and P?

A4: We chose to apply external forces to each particle rather than to the grid nodes to achieve finer force sampling. Because the scene volume is large compared with the objects, only a small fraction of grid cells are occupied in any given frame; applying forces at grid nodes would therefore leave most cells with zero force, yielding a coarse and noisy field.

By applying forces to particles, we can achieve more accurate and detailed force estimations, as the forces are directly linked to the moving particles, allowing for better resolution and precision.

Regarding Equation 13, we selected barycentric interpolation because it works well for structured data like skeleton-driven animation. It ensures smooth transitions between keyframes, which is essential for our setup involving small deformations or bending-only deformations. While KNN interpolation could be useful for more irregular or unstructured data, it may result in less smooth transitions and be computationally expensive when dealing with large datasets.

In our experiments, the number of particles x typically ranges around 10k to100k, while the number of P (keypoints) is on the order of 10, so the ratio of x to P is generally in the range of 1k to 10k.

Q5: The examples presented in the paper primarily involve force fields that have similar directions and a fixed input view. Can the proposed method address the following scenarios: (a) cases where the force directions are distinct (for instance, two leaves float from the left and right edges of the image toward the center, influenced by opposing forces) and (b) input videos that capture both camera movements and object movements?

A5: For Scenario A (distinct force directions): Our method is already capable of handling situations where forces act in different directions. In our real-world experiments, such as those involving in-the-wild videos, we demonstrated that the framework can successfully model forces that vary across the scene, e.g., in Figures 1 and 4. The force field is estimated at each point in the scene, allowing for distinct directions of force acting on different objects simultaneously. This flexibility in force direction handling has been validated in our experiments with random forces applied to objects, as shown in our results.

For Scenario B (camera and object movements): Our model doesn’t assume a static camera. As long as we can obtain a robust estimation of object motions in 4D, our method can infer forces from the motions. With the recent advances in 4D reconstruction and camera pose estimation, such as Shape of Motion[1] and MegaSAM[2], our method can be naturally extended to moving cameras.

[1] Wang, Qianqian, et al. "Shape of motion: 4d reconstruction from a single video." arXiv preprint arXiv:2407.13764 (2024).

[2] Li, Zhengqi, et al. "MegaSaM: Accurate, fast and robust structure and motion from casual dynamic videos." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

We are grateful for your thorough and insightful review. Your feedback has highlighted several important aspects, and we will carefully address each of your concerns in our response.

Q1: How sensitive is the method to inaccuracies in depth estimation or object segmentation from vision-language models (VLMs) and SAM?

A1: Our method is generally robust to inaccuracies in depth estimation and object segmentation due to the following reasons. As noted in Section 3.1, while object segmentation is important for assigning physical properties to the 3D Gaussians, SAM is generally reliable for this task, and the object-background separation it provides is usually sufficient for our method.

Inaccurate depth estimation can impact tracking, but our sparse tracking objective (described in Section 3.4) effectively handles this issue. By focusing on sparse keypoints rather than relying on dense depth data, we reduce the sensitivity to depth errors, making the method more robust to noise in depth estimation. You can also refer to Table 3 for related results.

Our experiments on in-the-wild video in Section 4.2 show that the method performs well even with minor errors in depth or segmentation. While significant errors can still affect the results, the sparse tracking approach ensures that the force recovery process remains reliable.

In summary, our sparse tracking mechanism effectively mitigates the impact of depth estimation inaccuracies, as demonstrated in our experiments.

Q2: What directions might improve the realism of the generated and edited videos? Could this method eventually support practical applications such as content creation or simulation?

A2: As demonstrated in Figures 4, 5, and 6, our method has shown potential in content creation and simulation for real-world applications. These examples highlight how realistic object interactions and force fields can be modeled in various scenarios.

To improve realism, future work could refine the physical simulation to handle complex materials, such as fluids or hyper-elastic deformations, and integrate more advanced models for collisions, friction, and lighting effects, enhancing both visual fidelity and physical accuracy.

This method could support practical applications like content creation and simulation. For example, in film production, it could simulate realistic environmental interactions, such as wind, while in virtual reality, it could enable physics-driven interactions. It could also be used in simulation for robotics and industrial design, where accurate object behavior under forces is critical.

Q3: Can the framework be adapted for real-time use in mobile or robotics settings, where interactive or reactive force estimation is required?

A3: Our framework can be applied to offline robotic data to estimate physical interactions, and with this information we can train a robotic policy that can be deployed in the real world. For instance, accurate wind-force estimation is valuable for pneumatic non-prehensile “blowing’’ manipulation, where a mobile blower herds scattered objects into a receptacle[1]. We will investigate this direction in the future. Thank you for the advice!

Q4: Limited applicability to object types: As acknowledged in the paper, the proposed framework currently supports only solid objects with small or bending-only deformations. More complex materials such as fluids or granular substances remain out of scope.

A4: Our current implementation is indeed limited to solid objects exhibiting only small or bending-dominant deformations; fluids and granular media remain beyond its scope. However, it can be addressed within our framework, albeit with a particle-based differentiable physical process. We will include more discussion in the limitation section of our paper and plan to address it in future iterations of the framework, where we hope to expand its applicability to a wider range of materials.

Q5: Heavy reliance on pretrained models: The accuracy of the force estimation pipeline depends heavily on initial object geometry and physical property estimations derived from pretrained vision-language and depth models. This reliance could become a bottleneck in more complex or noisy real-world scenarios.

A5: As discussed in Q1 and Section 3.4, our framework is designed to be resilient to errors in these initial estimates. The sparse tracking objective focuses on sparse, reliable keypoints for motion tracking. This approach helps mitigate the potential bottleneck caused by errors in the pre-trained models, particularly in real-world scenarios where noise and complexity may arise.

That being said, we agree that further improvements in the robustness of pre-trained models, especially in noisy or complex environments, would benefit our approach. We continue to explore methods to refine the initial object geometry and physical property estimations in future iterations of the framework.

Q6: Limited realism in generation and editing: Although the model demonstrates strong performance in force recovery and simulation, the visual quality of the generated and edited videos appears noticeably less realistic compared to recent state-of-the-art video generation models. This limits the perceived quality and potential practical deployment of the method in high-fidelity applications.

A6: Currently, the visual quality is constrained by the limitations of neural rendering techniques, but this can be addressed by integrating diffusion models to rectify the artifacts from neural rendering or using them as motion guidance to control a video diffusion model, similar to approaches like Difix3D[2]. This will be explored in future work.

[1] Wu, Jimmy, et al. "Learning pneumatic non-prehensile manipulation with a mobile blower." IEEE Robotics and Automation Letters 7.3 (2022): 8471-8478.

[2] Wu, Jay Zhangjie, et al. "Difix3d+: Improving 3d reconstructions with single-step diffusion models." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

Thank you for your comprehensive and constructive feedback. We truly appreciate the time you spent reviewing our paper and are grateful for your suggestions, which we will address point by point.

Q1: Missing discussion on the substantially related interactive simulation line of work (e.g. DragGAN). This is very important. The authors should show when those methods fail relative to this one, as they can also infer motion through optical flow and allow for insertion/editing.

A1: DragGAN and Motion Prompting are relevant to our work. The key difference is that these methods do not follow an explicit physical law. These methods infer motion through optical flow, using the term "force" to represent a direction of motion, which is more akin to displacement than to actual physical forces. In contrast, our approach explicitly models the underlying physical interactions—such as wind, gravity, and interaction forces—that govern the motion and deformation of objects. This distinction is crucial because our method estimates real physical forces, enabling more accurate and physically plausible simulations, especially in complex scenarios where precise force fields need to be recovered. In the revision, we will add a dedicated discussion of both papers and include a qualitative comparison to highlight these distinctions.

Q2: The "in-the-wild" qualitative results provided are entirely focused on wind, which limits the scope of applicability. There could be other unseen forces, e.g. gravity/object occluded actions. Also potentially interested in the elasticity results for in-the-wild data.

A2: First, it’s important to note that our framework does indeed account for gravity in the scenarios, as gravity is inherently included in the physical processes modeled in our system. This is particularly evident in how we capture and simulate object motion under natural conditions, including the effects of gravity on objects in the scene.

Regarding elasticity and handling object occluded actions, the results shown in Figure 1, particularly with the sticker example, demonstrate that our method is capable of handling objects with significant deformation and occlusion. The sticker example illustrates the model’s ability to track and simulate forces on objects undergoing elastic deformations, even in the presence of partial occlusion.

In summary, while the qualitative results presented focus on wind, our approach is capable of handling a broader range of forces, as demonstrated by the sticker example and other scenarios.

Q3: Qualitatively, the new object insertion results are strange. In particular, for the paper airplane, the leaves demonstrate a slight upward lift close to the start of the video which is not reflected in the airplane's trajectory. As such, I would appreciate some more uncurated examples for the rebuttal.

A3: Thank you for noting that the leaves rise slightly at the start, whereas the inserted paper airplane stays nearly level. This difference in behavior is due to the different physical properties of the two objects, such as mass and aerodynamic characteristics, which affect how they respond to the forces applied in the simulation. The leaves, being lighter and more affected by the wind, show a noticeable upward lift, while the heavier paper airplane reacts differently to the same forces.

Q4: All results/demo videos are done with respect to short video and a static camera. With the existing pipeline, it seems difficult to extend many of the results to real video with dynamic cameras (which could be important for broader applicability).

A4: Our model doesn’t assume short video or a static camera. As long as we can obtain a robust estimation of object motions in 4D, our method can infer forces from the motions. With the recent advances in 4D reconstruction and camera pose estimation, such as Shape of Motion[1] and MegaSAM[2], our method can be naturally extended to moving cameras.

[1] Wang, Qianqian, et al. "Shape of motion: 4d reconstruction from a single video." arXiv preprint arXiv:2407.13764 (2024).

[2] Li, Zhengqi, et al. "MegaSaM: Accurate, fast and robust structure and motion from casual dynamic videos." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

We would like to express our sincere gratitude for your thoughtful and detailed feedback. Your comments have provided valuable insights, and we are eager to address them in detail.

Q1: As mentioned above, the only real-world application presented for this model is the ability to edit or generate videos. Are there other applications for this work that might be of use to the broader NeurIPS or computer vision community?

A1: Beyond video generation and editing, one of the potential applications of our work is in augmented and virtual reality (AR/VR), where our method can model physics-driven behaviour of virtual objects. This could enable more immersive AR/VR experiences by dynamically adjusting objects' movements based on simulated forces like gravity or wind, which could be particularly useful in interactive gaming environments.

Another potential application is in robotics and simulation. By estimating the forces involved in object interactions, our framework could enhance robotic manipulation in dynamic environments—for example, guiding pneumatic non-prehensile “blowing’’ systems that corral objects into a receptacle [1]. This could improve robots’ ability to handle deformable or flexible items by modelling the forces acting on them, leading to better motion planning and control.

[1] Wu, Jimmy, et al. "Learning pneumatic non-prehensile manipulation with a mobile blower." IEEE Robotics and Automation Letters 7.3 (2022): 8471-8478.

Q2: All of the examples in this paper present videos with a single focus object (e.g. one leaf or a swing). Would this model work with multiple objects? With particulate matter (e.g. snow)? Or is the object modeling limited to a single thing?

A2: Our model does support multiple objects. For example, Figure 1 illustrates a scenario with four stickers, each modeled individually, and Figure 5 shows four sakura petals being modeled independently. The pipeline allows for separate modeling of each object’s geometry, physical properties, and interactions, making it compatible with multi-object scenarios by treating each object as an independent entity while still modeling the overall force field acting on them.

Regarding particulate matter like snow, as we mentioned in our limitation section, we acknowledge that our model is primarily designed for objects with small or bending-only deformation. That said, our method can model particulate matter with a particle-based differentiable physical process; this is beyond the scope of this work and we leave it for future exploration.

Q3: A VLM is used in the object identification phase to identify material properties to add latent physical properties, and against a small benchmark provides good property estimation. But how important is this to the larger model – e.g., if you assume only average information, or a misidentification (e.g. something that looks like cloth but acts like cardboard) how brittle is the full model?

A3: We found that if the physical parameters are in a reasonable range, our framework is quite robust to object identification. Since VLM contains commonsense knowledge about object physical properties, we found it is sufficient for our use. A misidentification in the case you referred to may lead to a failure in our model, but in our experiments, we didn’t notice such a severe hallucination of VLMs.

Q4: What types of forces are used in the synthetic video set? Are the uniform in magnitude and direction but time varying? Or can the wind be blowing in two directions at once? Looking at Figure 4, the force field appears to be pointing in the same direction for each frame, and the forces appear to have a smooth gradient across the frame (generally peaking around the object) so it’s not clear whether this is just because the examples focus on a single object with wind where local direction and magnitude estimates are most important, or whether this is a heuristic found by the model

A4: Thank you for your inquiry! Firstly, for a clear and fair quantitative evaluation of our method, we set the forces to be time-varying but spatially uniform in our synthetic data.

Secondly, yes, the wind can indeed be blowing in two directions at once. This is because we are capturing the force field at each time step, and the force field is a result of the combined forces acting on the object. In scenarios where multiple forces are acting simultaneously, they are captured as a cumulative force field, which can have different directions and magnitudes at different spatial locations in the scene.

Regarding the observations in Figure 4, indeed the force field appears to be pointing in the same direction for each frame with a smooth gradient. This apparent uniformity is specific to that example and not a systematic tendency of our model’s estimates. In other examples, such as those in Figure 1, the force field is more complex and varies in both direction and magnitude, highlighting that the method is capable of handling more diverse scenarios, not just uniform forces.