Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video

Thank you for your comments. We have made every effort to answer each of your questions and hope that our response can address your concerns.

Q1: Is there any reference to show the variations of predicted particle states?

Good question! It is crucial to understand the diversity in probabilistic dynamics prediction. Therefore, in Tables 1~4 in the revised paper, we predict each sequence 10 times with different $\mathbf z_t$ drawn from the prior module, and calculate the standard deviation of the errors. The results are shown below:

Our results for novel scene simulation ($t\in [2,60]$), added to Table 1:

Our future prediction results for the observed scenes ($t\in [51,60]$):

Our results on particle datasets, added to Table 3:

Our results of prediction error on generalization to unseen fluid dynamics, added to Table 4:

As we can see from the updated results, the predictions of our model yield larger variances as the prediction time horizon extends.

Note: In this context, we conduct multiple simulations to compute the standard deviation for each individual sequence. This differs from our evaluation setup for the subsequent question, where we "calculate the standard deviation of prediction errors across multiple sequences".

Q2: What's the unit of measurement in Table 3? Is it millimeter or centimeter?

All the tables in our paper, including Table 3, use millimeters as the unit of measurement.

Q3: What's the standard deviation of the prediction errors in Table 2? For example, one scene should include several sequences. The errors may vary from sequence to sequence.

First, we want to clarify that Table 2 showcases the future prediction errors in observed scenes, where only videos of a single sequence are provided for each physical parameter.

Instead, in Table 1, we do evaluate our method on 50 different sequences with diverse unseen fluid geometries and boundary conditions for each physical parameter. Below, we present the standard deviation of the simulation errors across these sequences.

We should note that there are substantial variations in simulation difficulty between these samples, resulting from diverse initial conditions (including initial positions and velocities) and boundary conditions. Consequently, we observe large standard deviations for all the compared methods. Despite these challenges, our model consistently presents significantly lower average errors and the lowest inter-sample standard deviation as well. This demonstrates its robustness in effectively handling diverse and challenging fluid scenes.

Q4: In Fig 4, is it an accurate reference of the results for the phrase "row 3-6"? It seems like the results start from row 2.

Thank you for your careful review. It's a typo and we have corrected it to "Rows 2-6" in the revision.

We appreciate your efforts to review our work. In addition to the following responses, we have also made comprehensive revisions to the paper to enhance its overall clarity.

Q1: I found the paper very dense and extremely hard to follow for someone outside of the field.

We apologize for any difficulty you experienced in reading our paper. We have tried our best to organize the writing, but found it challenging due to the complexity of the proposed method, which involves multiple training stages and many details. To improve the overall reading experience, we have made the following changes:

• We refined Figure 3 to include all network components and clarify the inputs and outputs of each module.
• We introduced Figure 7 in Appendix A.1, giving the details of each training stage.
• Moreover, Table 6 now summarizes each component's formulations, inputs, outputs, active training stages, and objective functions.

We hope these modifications will improve the readability of the paper and provide a clearer understanding of our approach. If you have any further suggestions or concerns, please feel free to let us know.

Q2: The description of the experiments and their evaluation could have been clearer including the metrics used for the evaluation.

For all experiments but the short-term predictions in Table 3, we compute the average Euclidean distance from the true particle positions ($p_t^i$) to the positions of the closest predicted particles ($\hat{p}_t$):

$\bar{d}=\frac{1}{T \times N} \sum_t \sum_i \min _{\hat{p}_t} ||p_t^i - \hat{p}_t||_2$ where $T$ is the prediction time horizon and $N$ is the number of particles. As mentioned in Section 5.1 ("Settings"), we adopt this evaluation metric from the previous work by Ummenhofer et al. (2020).

In particular, for the particle-based experiments for short-term prediction in Table 3 (the first two columns), we compute a one-to-one mapping metric, the average error of the predicted particle positions w.r.t. the true positions of corresponding particles:

$d_t=\frac{1}{N} \sum_{i} ||p_t^i - \hat{p}_t^i||_2$ where $\hat{p}_t^i$ is the predicted position for particle $i$.

Thank you for your valuable comments. We hope that the responses below can fully address your concerns. If you have any additional suggestions, please don't hesitate to inform us.

Q1: This framework includes many model processing, and it was hard for me to understand the whole structure at the first read.

(1) We apologize for any difficulty you experienced in reading our paper. We have made the following changes to improve readability:

• Following your suggestion, we refined Figure 3 to involve all network components of the model architecture, including the neural renderer. Specifically, the neural renderer follows the implementation from Guan et al. (2022), which generates separate view-independent and view-dependent encodings based on predicted particle positions. A detailed description of PhysNeRF is included in Appendix A.3.
• In Appendix A.1, we added a new figure (Figure 7) to illustrate the training details of the particle-space pretraining stage, the visual inference stage, and the prior adaptation stage, respectively. Moreover, we have improved Table 6 to give a more detailed summarization of each component's formulations, inputs, outputs, active training stages, and objective functions.

(2) Regarding "type-aware preprocessing" in the original Figure 3:

The "type-aware preprocessing" follows the implementation and the terminology from the CConv paper (Ummenhofer et al., 2020). As depicted in Figure 2 of their paper, this mechanism is realized with two separate CConv layers: One is to encode the fluid particles in the neighborhood of each particle location; The other one is to handle the virtual boundary particles in the same neighborhood. The features extracted from these processes are then concatenated to form the inputs for subsequent layers. We have provided a detailed description of how the model processes fluid and boundary particles in Appendix A.2. To enhance the clarity of Figure 3, we have omitted the "type-aware preprocessing" from the figure, without compromising the presentation of our technical contributions to the core network architecture.

Q2: It might be beneficial to describe the current state and the challenges more in the main text in addition to the information provided in the Appendix.

Thanks for the suggestion! We have incorporated a new figure (Figure 6) in the main text to provide a clearer presentation of our real-world experiments. Due to page limits, a more detailed discussion is provided in the corresponding section in the appendix.

Thank you for your comments. We hope that our responses below can adequately address your concerns regarding this paper.

Q1: I am not sure about the claim of latent intuitive physics being proposed by them first. This claim can be softened to say this is the first method to perform this for fluids using 3D exemplars.

We appreciate your suggestion, and in the revised paper, we have refined the introduction to more precisely highlight the technical contributions. Specifically, the novel aspects of the proposed "latent intuitive physics" approach now include:

• It presents a novel learning-based approach for fluid simulation, which infers the hidden properties of fluids from 3D exemplars and transfers this knowledge to a fluid simulator.
• Our approach includes the first probabilistic network for particle-based fluid simulation.

Q2: The authors report that the initial velocities are zero for initial state estimation-- how does this tally up with the initial conditions in the particle datasets considered?

(1) Clarification:

We only adopt the assumption of zero initial velocities for the observed visual scene (Table 2), while applying non-zero initial velocities for particle-based experiments (Table 3) and for the novel scenes as well (Table 1, Table 4, and Table 5).

(2) New experiments with non-zero initial velocities of the observed visual scene:

To assess the performance of our method in a more general case, that is, learning from visual observations of fluids with non-zero initial velocities, we modify the training scheme in Stage B by randomly initializing $v_{t=1}^{i=1:N}$ and treating them as trainable parameters. The optimization of $v_{t=1}^{i=1:N}$ is carried out concurrently with the optimization of the visual posteriors $\hat{\mathbf{z}}$.

Below, we evaluate the results against a model trained with true non-zero initial velocities on the observed scene. We here compare

• Future prediction errors on observed scenes (denoted by Obs.)
• Average simulation errors on novel scenes with new initial geometries and boundaries (denoted by Geom. and Bdy.):

We have included the above results in Appendix E. As we can see, both models produce comparable results, showcasing the ability of our method to infer uncertain initial velocities of fluids by treating them as optimized variables.

Q3: Is the zero initial state velocity a general assumption for all experiments performed? Does this correspond to $(\mathbf{z}\_{t=1}, \tilde{\mathbf{z}}\_{t=1})$ being zero-initialized?

(1) No, assuming zero initial velocity ($v_{t=1}^{i=1:N}$) is not a universal condition for all experiments. As shown below, we only adopt this assumption for the observed visual scene, while applying non-zero initial velocities for the novel scenes (Table 1, Table 4, and Table 5), as well as for the particle datasets (Table 3).

(2) For the latent states $(\mathbf{z}\_{t}, \tilde{\mathbf{z}}\_{t})$ in all experiments, we do initialize $\mathbf{z}\_{t=1}$ and $\tilde{\mathbf{z}}\_{t=1}$ with zero values, which is similar to the common practice for initializing the hidden states of RNNs. Notably, there is no direct connection between the initialization approaches for latent states and the velocity states.

Q4: Why is there a difference in the type of posterior used in the pre-training and the inference stages, being particle-based and visual, respectively?

As illustrated below, both particle-based posteriors ${\mathbf{z}}\_t$ and visual posteriors $\hat{\mathbf{z}}$ are employed to offer effective training supervision for the prior learner.

• Pretraining stage: The particle-based posterior module $q_\xi(\mathbf{z}\_t \mid \mathbf{x}\_{1: t}, \mathbf{z}\_{t-1})$ conditions on the prediction target $\mathbf{x}\_t$ and the previous particle states along with latent features $(\mathbf{x}\_{1:t-1}, \mathbf{z}\_{1:t-1})$. This setup enables the prior learner $p_\psi(\tilde{\mathbf{z}}\_t \mid \mathbf{x}\_{1:t-1}, \tilde{\mathbf{z}}\_{t-1})$ to generate meaningful latent features that capture hidden properties essential for accurate predictions.
• Visual inference stage: In this phase, true particle states are unavailable. Instead, we rely on visual observations and optimize a new set of variables known as visual posteriors ($\hat{\mathbf{z}}$). These visual posteriors are refined by backpropagating image rendering errors. They serve as adaptation targets for the prior learner, facilitating efficient adaptation to the specific observed fluid.

Q5:. Why is the prior learning less prone to overfitting?

In Section 4.3 of the original manuscript, we claimed that "Instead of finetuning the entire transition network as NeuroFluid (Guan et al., 2022) does, we only finetune the prior learner module, which is more efficient and less prone to overfitting." What we intended to convey is that tuning all parameters in the transition model in the visual scene (as NeuroFluid does) can lead to overfitting problems. Due to the unavailability of the ground truth supervision signal in particle space, the transition model may:

• Forget the pre-learned knowledge of feasible dynamics;
• Learn physically implausible particle transitions (as the learnable transition network has a wide variety of ways to move the input particles to the expected geometric positions, all of which are sufficient to minimize the image rendering loss).

The evidence is that NeuroFluid shows superior performance compared to our approach for the observed scene with $\rho=500,\nu=0.2$ (as presented in Table 1). However, it underperforms for corresponding novel scenes with new initial and boundary conditions (as detailed in Table 2).

Unlike NeuroFluid, our approach adapts the physical prior learner to visual scenes, without training a probabilistic physics engine. By leveraging knowledge from the pretraining stage, the transition model is less prone to overfitting on the observed scene. This not only enhances the generalization ability but also significantly reduces the training burden.

We have provided additional clarification on this statement to avoid misunderstanding in the revised paper.