Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering

1. Motivation and Definition of Explicit Scene Representation

We apologize for any miscommunication and will revise the paper to better define terms like "explicit scene representation," "symbolic reasoning model," and "generative model." We thank the reviewer for their constructive feedback and will incorporate these clarifications in the revised manuscript.

• Motivation for Explicit Scene Representation. As cited in related works like NS-VQA [1], NS-DR [2], and P-NSVQA[3], VR-DP[4], PO3D-VQA[5], the compositional reasoning model with explicit scene representations has a long history and is proved effective for tasks requiring reasoning about object’s properties and interactions. These previous works demonstrate that the compositional reasoning models are beneficial from the explicit 2D / 3D scene representation in terms of accuracy, interpretability, and robustness for out-of-domain reasoning[3]. This motivation for explicit scene representations is consistent with our aim for 3D dynamical scenes.

  Especially, for our video questions answering tasks, capturing explicit 3D and temporal (4D) dynamics allows us to understand and reason about object interactions in a structured and interpretable way, which is crucial for answering 4D dynamics questions.

• What is Explicit Scene Representation: As written in Section 4.1 (line 308) in the paper, the explicit scene representation is a 3D representation of objects and their properties (e.g., position, velocity, acceleration) over time. This structured representation serves as the foundation for symbolic reasoning.

• The concept between "scene parsing module" and "3D generative model". As written in Section 4.2 (line 318 to line 323), the scene parsing module uses the 3D generative model as its core component to infer the dynamic properties of the scene. The generative model provides the underlying framework for estimating 3D world states from video frames, while the scene parsing module applies this model to sequential data, integrating temporal and physical reasoning.

• How do the "3D generative representation" and "symbolic reasoning module" work together. The implementation follows the prevailing neural symbolic reasoning pipeline (NS-VQA [1], NS-DR [2], and P-NSVQA[3]). The 3D generative representation provides a structured, interpretable representation of the scene, including object positions, poses, and physical properties over time. The symbolic reasoning module operates on this representation to answer high-level questions step by step. For each step, the operation is implemented by a function, to query the related properties and get the intermediate result, to pass to the next operation [1].

2. Significance of the Proposed Dataset

We appreciate the feedback and agree that the dataset does not cover all real-world scenarios. However, similar to previous benchmarks like CLEVR, CLEVRER and SuperCLEVR, our dataset is intended as a starting point for studying 4D dynamics in a controlled setting. While the scenarios are simplified, they provide a strong foundation for understanding and benchmarking dynamic reasoning tasks.

Compared to previous datasets (CLEVR, CLEVRER, SuperCLEVR), our dataset offers significantly more realistic 3D assets (agreed by Reviewer jmWM ), dynamic properties, and annotated ground truth. Moreover, our framework is designed to be extensible, and future iterations will incorporate more complex and realistic scenarios. We will emphasize these points in the revised manuscript and discuss the extendability of our approach.

Reference

[1] Yi, Kexin, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. "Neural-symbolic vqa: Disentangling reasoning from vision and language understanding." Advances in neural information processing systems 31 (2018).

[2] Yi, Kexin, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. "Clevrer: Collision events for video representation and reasoning." arXiv preprint arXiv:1910.01442 (2019).

[3] Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14963–14973, 2023.

[4] Ding, Mingyu, Zhenfang Chen, Tao Du, Ping Luo, Josh Tenenbaum, and Chuang Gan. "Dynamic visual reasoning by learning differentiable physics models from video and language." Advances In Neural Information Processing Systems 34 (2021): 887-899.

Regarding the last concern raised in your review, we hope this can be leveraged with real video inference cases as shown in the supplementary material. Due to the constraint of annotation as explained in the previous discussion, we retrain our 3D scene parser on the Pascal3D+ dataset, which contains sufficient 3D annotations but lacks object appearance labels. The detailed results are provided in Fig 1(a) to reconstruct the 3D dynamical world states. We aim to demonstrate that our model can be trained on and infer from real-world data with proper annotations, particularly for the focused 4D dynamical properties. This is feasible if the 3D world states of objects can be successfully estimated. In Fig. (b), we show similar questions and answers as in DynSuperCLEVR in this real-world case. These questions can also be resolved with similar program executions if all properties have been estimated with the proper trained model (as shown in Fig. (c)).

I hope the review acknowledges that real-world video datasets are currently limited by the absence of necessary 3D spatial annotations. This why we start to propose the first benchmark to address this gap in VideoQA, with a focus on 4D dynamics, including velocity, acceleration, and collisions.

It would also be helpful if the reviewer could provide more detailed suggestions about which benchmark you recommend verifying with additional experiments.

1. Narrow Domain and Synthetic Nature of the Dataset

   Please refer to a detailed discussion about the synthetic nature of DynSuperCLEVR is in the above common official comment.

2. Reliance on Physics Priors

   • The physical priors used in the model reflect common-sense world knowledge (e.g., objects follow trajectories governed by physical laws) and are not specific to our dataset alone. Also, as shown in the paper, we conducted ablation studies demonstrating that our model performs well even without physics priors, albeit with some degradation in performance. This highlights the model's robustness and flexibility in scenarios where the priors may not fully apply.

   • We are working on the more abolition study, which compare the performance under different physical parameters, where the results show our results is robust to the strength physical prior choice.

3. How did you input the video frames to GPT-4o?

   We downsampled the video to 8 frames for computational efficiency. These frames were then processed and sent to the GPT-4o API according to the guidelines outlined in the OpenAI GPT-4o documentation (https://cookbook.openai.com/examples/gpt4o/introduction_to_gpt4o).

1. Limited Dynamics (Rigid Objects and Linear Motion) Please refer to the common question in the Official Comments above for the discussion on object diversity.

2. Synthetic Rendering and Real-World Challenges

   The challenges of synthetic data generalization, such as motion blur, camera shake, and lighting variability, are indeed important. Similar to widely used datasets like CLEVR and CLEVRER, our approach focuses on controlled synthetic benchmarks to establish a strong foundation for understanding 4D dynamics in vision-language models.

   To address these concerns, we have created a subset of the dataset that incorporates varying lighting conditions and motion blur. The original DynSuperCLEVR dataset already utilizes realistic ambient lighting from HDRI environment maps, including both daytime and nighttime scenes. During the rebuttal phase, we introduced additional variations by controlling the strength of ambient light to increase variability.

   To mimic motion blur, for each frame $F_i$, we created a motion-blurred frame $F_i'$ by averaging pixel values across a temporal window of random size (0–5) centered on $F_i$. This subset serves as a bridge between synthetic and real-world scenarios, enabling the evaluation and improvement of model robustness in more challenging conditions.

3. Limited Ablation Studies

   We appreciate the reviewer’s suggestion to analyze the impact of additional architectural choices and hyperparameters. While our current ablation studies focus on the importance of physics priors, we have included new experiments to evaluate the effects of different CNN backbones and variations in the physics engine parameter $\sigma$ (see the Equation (2) in main paper). Our preliminary findings suggest that the model maintains robustness across these changes, with the best performance observed at $\sigma=2$, as used in the submission. Performance across different CNN backbones is comparable (all above 77, with ResNet50 slightly lower). A summary of the results is provided below:

Questions

1. Performance on Complex Dynamics (e.g., Non-Rigid Deformation, Fluid Dynamics)

   Please refer to the Official Comments for the discussion about non-rigid objects in the dataset.

   Extending the model to handle non-rigid deformations and fluid dynamics is a meaningful direction for future work. While this paper focuses on rigid body dynamics, the core component 3D generateive model has been demonstrated the possibility in applying on deformable setting [1] and we are actively exploring these extensions.

   [1] Wang, Angtian, Wufei Ma, Alan Yuille, and Adam Kortylewski. "Neural textured deformable meshes for robust analysis-by-synthesis." In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3108-3117. 2024.

2. Addressing Real-World Challenges (e.g., Motion Blur, Camera Shake, Lighting Variability)

   As discussed above, we explored extensions to the dataset with realistic variations, including motion blur, varying lighting conditions to improve the model's robustness in real-world scenarios.

3. Impact of Architectural Choices and Hyperparameters

   Please see the discussion of weakness 1 above for these ablation studies.

We appreciate the reviewer’s insightful feedback and will incorporate these clarifications and additions in the revised manuscript.

1. Synthetic Data Limitations

   A more detailed discussion about the synthetic nature of DynSuperCLEVR is in the above common official comment.

   We acknowledge the importance of generalizing to real-world scenarios and appreciate the reviewer recognizing our efforts to enhance the realism of textures in the synthetic dataset. Regarding realism, we believe it is entirely plausible to make this dataset increasingly realistic in future developments.

2. Computational Complexity.

   We add a detailed efficiency analysis of the average cost of NS-4DPhysics model in a video clip of 120 frames, with 2 second, with 5 objects in the scenes.

   Time Efficiency: For estimating dynamical properties for each frame, the total time for the 3D scene parser is 557.85 ms, with 3D generative modeling taking 521.15 ms and physics prior calculation requiring only 21.49 ms. For each video clip, predicting static properties across the entire video takes 140.16 ms, while the question parser and program execution take 5.15 ms and 0.03 ms, respectively.

   Computation Usage: As the bottleneck of the pipeline, the peak virtual memory occupation for 3D generative modeling, dynamical properties estimation and physics prior calculation is 3179 MiB.

3. Limited Object Diversity.

   We agree that extending the dataset to include articulated or deformable objects is a meaningful direction for real-world simulation. However, our primary focus is on rigid body dynamics as a foundational step toward investigating fundamental dynamic properties in vision-language tasks. Even without deformable objects, we have made several important findings: 1) the dataset effectively reveals the limitations of existing models in understanding fundamental dynamics, and 2) explicitly representing 4D dynamic properties, as implemented in the proposed NS-4DPhysics model, significantly enhances model performance.

   Incorporating articulated or deformable objects introduces significant challenges, such as complex annotations and increased computational costs. We plan to address these challenges in future work and will note this in the paper to clarify our scope and future directions.

4. Evaluation of Real-World Applicability.

   Please refer to the third point in the common questions section of the official comments above, which discusses the generalization ability of our proposed model, NS-4DPhysics

5. Have the authors considered extending the dataset to include articulated or deformable objects?

   Please refer to the previous discussion in "3. Limited Object Diversity" above.

6. Could the authors provide an efficiency analysis of the proposed model, including resource usage and runtime under typical conditions?

   Please refer to the previous discussion in "2. Computational Complexity"

7. What modifications to the NS-4DPhysics framework would make it more efficient for real-time performance or deployment in resource-limited environments?

   As discussed above, the current model processes approximately 2 frames per second, which corresponds to handling a 2 fps video stream in real-time. To improve real-time performance at higher frame rates, we can utilize lightweight 3D representations within the neural mesh modeling framework for faster inference. Furthermore, while the current DynSuperCLEVR is setup at a frame rate of 60, the model is also applicable for lower frame rate scenarios, as the dynamic properties of the video are processed frame-wise. Additionally, our proposed physical prior shows superior performance in preserving temporal consistency, as demonstrated in Sections 5.2 and 5.3.

Here we aim to provide additional evidence demonstrating the generalization ability of our proposed model, NS-4DPhysics, to real-world videos. We consider a case study experiment, as detailed in the new supplementary material.

We consider a real-world video with the collision event. We retrain the 3D scene parser on the Pascal3D+ dataset, which includes 3D pose annotations but without object appearance labels. The qualitative reconstruction results are provided in the supplementary material. As shown in Fig.1(a), we visualize the accurate estimations of cars from 3D scenes on the real video, where the 4D dynamic properties, including velocities, accelerations and collisions, can be inferred explicitly. Although the model is not trained on object appearances, its capabilities can be further enhanced by incorporating object classifiers with proper annotations or enabling open-vocabulary recognition through pre-trained large vision-language feature embeddings (e.g., CLIP). This represents an important direction for future work. Additionally, as shown in Fig.1(b) and (c), similar types of questions can be posed for the given video, which can then be answered by executing the program step-by-step.