MetaPhysiCa: Improving OOD Robustness in Physics-informed Machine Learning

Q4: Which two terms in each of the three dynamical systems have causal relationships? Please give an example to explain it.

A4: Thank you for the question. First, a note not entirely related to the reviewer’s question but that helps avoid misunderstandings: The causal model is a model of the dynamical system. That is, for the interventions on the initial state $X_{t_0}$ and environment changes ($W^*$ parameters), the resulting output of our model is equivalent to what would happen in the system (up to noise). This is enough to make the model robust OOD but it is not a complete causal model of the real system (interventions on variables that are not $X_{t_0}$ and $W^*$ parameters are not considered).

Now to the reviewer’s question: All terms on the RHS of the true dynamical system affect the future dynamics $dx/dt$. For example, in the damped pendulum system, the change in angular velocity is causally affected by both $sin(\theta_t)$ and the damping term. To be OOD robust in our scenarios we must be robust to changes in initial conditions $X_{t_0}$ and interventions on the parameters $W^*$.

Q5: Please conduct experiments on noisy data.

A5: Thank you for the suggestion. We had evaluated the effect of noise on MetaPhysiCa’s ability to find the true dynamical system in Appendix D.3. We have now added a pointer to this section in the main text.

In Appendix D.3., we repeat the Damped pendulum and Predator-prey experiments with increasing amounts of noise. Specifically, we add 1%, 5% and 10% Gaussian noise to all the trajectories, both in training and in test. We report the normalized RMSE for different models trained on the noisy versions of data in Figure 8.

In both tasks, the proposed method is relatively robust to small amounts of noise and outperforms the baselines. With 10% noise, MetaPhysiCa is unable to identify the dynamical system accurately, but performs comparable to the baselines. This is consistent with the impossibility results in Fajardo-Fontiveros et al. [1], where the authors show that there is a fundamental limit to learning the true model from noisy data: after a level of noise, true model is unlearnable by any model.

References

[1] Fajardo-Fontiveros, O., Reichardt, I., De Los Ríos, H.R., Duch, J., Sales-Pardo, M. and Guimerà, R., 2023. Fundamental limits to learning closed-form mathematical models from data. Nature Communications, 14(1), p.1043.

We thank the reviewer for their insightful feedback. We have addressed the reviewer questions below.

Q1: How many basis functions are used in the experiments? If there is no prior knowledge, the number of basis functions will be very large.

A1: That is a good question. In our experiments, we use the same set of 14 basis functions for all 3 dynamical systems that include sine, cosine and polynomial terms up to 3rd power. At the reviewer’s suggestion, we repeated our damped pendulum experiments with increasing numbers of basis functions ranging from 7 basis terms (sinusoidal terms, polynomial terms up to power 1) to 32 basis terms (sinusoidal terms, polynomial terms up to power 6) per output dimension in the damped pendulum system ($\theta_t$ and $\omega_t$). 

Figure 9 in Appendix D.7 shows the training loss and $||\hat{\Phi}||_1$ over epoch. MetaPhysiCa converges to the true dynamics of the damped pendulum system with 3 basis terms for all the different values of $m$ tested. However, as $m$ increases, model requires a higher number of epochs to reach convergence.

Q2: The proposed method still adopts the SINDy-like approach in which all key terms should be included in a set of basis functions. It may not work on complex equations.

A2: Thank you for the question. In Section 3.1 we discuss the importance of having appropriate basis functions in the architecture for OOD extrapolation. Without the algorithmic alignment with appropriate basis functions, the in-domain approximations of the true model may not hold when the inputs are out-of-domain (as shown in Figure 2a).

At reviewer’s suggestion, we show two cases where algorithmic alignment is more challenging to achieve:

1. Appropriate basis functions required to learn the ground truth dynamics are not present. We repeat the damped pendulum experiment without sine/cosine basis functions in Appendix D.6.
   In the absence of $\sin(\theta_t)$ term to learn the true dynamics of the damped pendulum system, MetaPhysiCa learns a truncated Taylor series approximation of this term via $\theta_t$ and $\theta_t^3$ terms:
   $\frac{d\theta_t}{dt} = W_1 \omega_t; \frac{d\omega_t}{dt} = W_2 \theta_t + W_3 \theta_t^3 + W_4 \omega_t$.
   Table 11 in Appendix D.6  shows that after test-time adaptation, this learnt model achieves $1.5\times$ to $2\times$ better OOD test NRMSE than the best baseline, but is worse than MetaPhysiCa with sinusoidal basis functions included.  

2. More expressive SCM. In Appendix D.4, we extend MetaPhysiCa to a more expressive SCM by composing the given basis functions (e.g., to obtain terms such as $sin(x^2)$, etc.). MetaPhysiCa with such an expressive SCM shows OOD performance gains on a complex ODE task (Appendix D.4), but sometimes suffers from learning stiff ODEs during optimization due to the complexity of such a 2-layer composition procedure. Better optimization techniques may help alleviate this problem. We discuss this as a potential future work in Conclusions section.

Q3: Conduct experiments on more complex systems in the CoDA paper, such as reaction-diffusion system and Navier-Stokes system.

A3: That is a good suggestion. We have started exploring fluid dynamic applications as a future direction. For now, at least, we can say that this extension is non-trivial but also very interesting. A PDE needs an expanded set of basis functions including differential operators in order to be able to OOD model behaviors like shock waves and turbulence, which pose unique OOD challenges.

Furthermore, given the nature of the PDEs, the boundary conditions, especially in OOD scenarios, become critically important. Ensuring that the model accurately represents the physics at the boundaries while generalizing well in OOD scenarios is a non-trivial task. This seems to entail developing new techniques for imposing and learning boundary conditions within the MetaPhysiCa framework. Moreover, for fluid simulations, it is important to balance the trade-offs between physical accuracy and computational efficiency. We discuss including PDEs as future work in the Conclusions section.

Q5: Why are only the task-specific parameters updated during the test-time? Why would causal structure not needed to be updated? For fraudulent detection system, it is possible that a ODE function shifts during the test time.

A5: Thank you for the interesting question. In our OOD settings, we assume we will not see enough of the test-time series in order to learn a new basis (or a modification of a basis). SINDy, for instance, tries to learn the basis under this setting and generally fails. Hence, our OOD shift is with respect to the interventions on the initial conditions $X_{t_0}$ and parameters $W^*$, while the underlying structure of the dynamical system remains the same. Thus, it is enough to update the task-specific parameters. Small OOD changes to the causal structure is an interesting future work direction, possibly requiring additional regularization terms to restrict the amount of changes to the learnt structure.

Q6: What is the theoretical guarantee that MetaPhysiCa discovers the good task specific parameters, especially for few-shot updates during the test time?

A6: This is another interesting challenge. Unfortunately, establishing theoretical bounds for task-specific parameters under arbitrary basis assignments seems quite difficult. These bounds appear intrinsically linked to the basis used in the causal-equivalent model, leading to a scenario where each application demands its unique set of parameter bounds. Although there might be potential methods to circumvent these task-dependent bounds, we are yet to find a universally applicable solution that could extend across all tasks.

Q7: Computational complexity introduced by MetaPhysiCa, especially for more larger pools of basis functions and increasing number of observations?

A7: For a single task $i$ and time $t$, computing the predicted derivatives $d\hat{X}^{(i)}_t/dt$ in Equation 3 has $O(md)$ complexity to evaluate the Hadamard product and the matrix-vector product, where $m$ is the number of basis functions and $d$ is the state dimension of the dynamical system. Since there are $M$ tasks/trajectories of maximum length $T$, the overall complexity per epoch is $O(mdMT)$. However, as Figure 9 shows, MetaPhysiCa could require a higher number of epochs to converge for higher values of $m$. We have added a paragraph in Appendix C.1 discussing the time complexity.

Q8: Would MetaPhysicCa be adapted or be integrated into PIML methods that contain different loss functions? Is that as simple as adding the additional loss terms into Equation 4?

A8: Depending on the application, other PIML loss functions could be further incorporated (e.g., energy conservation, monotonicity, etc.) while preserving the basic principles of MetaPhysiCa: Orthogonal basis functions, identifiability of the equivalent causal model for OOD robustness. However, these PIML constraints should also be enforced during test-time adaptation of task-specific parameters by adding these loss terms in Equation 5.

We thank the reviewer for their support and insightful feedback. We give detailed answers below.

Q1: it is usually the case that the the correctly specified basis function is included in the search space of f. MetaPhsysiCa’s performances in the set of incorrectly specified basis function is not known.

A1: That is a good point that we now address in the revised submission. We have added clarifying statements in the updated submission. Specifically, we emphasize in Section 3.1 that appropriate basis functions must be present in the architecture for OOD extrapolation. Without the algorithmic alignment with appropriate basis functions, the in-domain approximations of the true model may not hold when the inputs are out-of-domain (as shown in Figure 2a).

In the updated submission we perform an interesting experiment (thank you for the suggestion!). We repeated the damped pendulum experiment without algorithmic alignment, i.e., appropriate basis functions required to learn the ground truth dynamics are not present. We repeat the damped pendulum experiment without sine/cosine basis functions in Appendix D.6. In the absence of $\sin(\theta_t)$ term to learn the true dynamics of the damped pendulum system, MetaPhysiCa learns a truncated Taylor series approximation of this term via $\theta_t$ and $\theta_t^3$ terms: $\frac{d\theta_t}{dt} = W_1 \omega_t; \frac{d\omega_t}{dt} = W_2 \theta_t + W_3 \theta_t^3 + W_4 \omega_t$. Table 11 in Appendix D.6 shows that after test-time adaptation, this learnt model achieves $1.5\times$ to $2\times$ better OOD test NRMSE than the best baseline, but is worse than MetaPhysiCa with sinusoidal basis functions included.

Q2: There should be an ablation study for a relatively small pool of basis functions and a relatively large pool of basis functions.

A2: Thank you for the great suggestion! In Appendix D.7 of the revised submission, we repeat our damped pendulum experiments with increasing numbers of basis functions ranging from 7 basis terms (sinusoidal terms, polynomial terms up to power 1) to 32 basis terms (sinusoidal terms, polynomial terms up to power 6) per output dimension in the damped pendulum system ($\theta_t$ and $\omega_t$).

Figure 9 in Appendix D.7 shows the training loss and $||\hat{\Phi}||_1$ over epoch. MetaPhysiCa converges to the true dynamics of the damped pendulum system with 3 basis terms for all the different values of $m$ tested. However, as $m$ increases, model requires a higher number of epochs to reach convergence.

Q3: The authors should consider adding experiment results that shows bi-level optimization results as well as the joint optimization.

A3: We have now added a section D.2.3 in the Appendix discussing bi-level optimization vs joint optimization in more detail. We observe that bi-level optimization and joint optimization result in $\hat{\Phi}$ learning the true dynamics for all 3 tested dynamical systems. However, joint optimization is $8.2\times$ faster per epoch than the bi-level optimization (taking 90ms vs 744ms on average on one Intel(R) Xeon(R) CPU core). The final NRMSE values are the same for both as test-time adaptation is an independent step and learns the same task-specific parameters once the true structure is learnt by either optimization method. We have added a pointer to this section in the main text.

Q4. The authors have described the MetaPhysiCa primarily for ODE tasks. However, could the MetaPhysiCa be adapted to PDE setting

A4: That is an interesting question. We have started exploring fluid dynamic applications as a future direction. For now, at least, we can say that this extension is non-trivial but also very interesting. A PDE needs an expanded set of basis functions including differential operators in order to be able to OOD model behaviors like shock waves and turbulence, which pose unique OOD challenges.

Furthermore, given the nature of the PDEs, the boundary conditions, especially in OOD scenarios, become critically important. Ensuring that the model accurately represents the physics at the boundaries while generalizing well in OOD scenarios is a non-trivial task. This seems to entail developing new techniques for imposing and learning boundary conditions within the MetaPhysiCa framework. Moreover, for fluid simulations, it is important to balance the trade-offs between physical accuracy and computational efficiency. We discuss including PDEs as future work in the Conclusions section.

We thank the reviewer for their support and insightful feedback. We give detailed answers below.

Q1: Testing is required on more challenging settings e.g., 1D convection, convection diffusion, other “stiff” PDE settings?

A1: We agree that extending MetaPhysiCa to forecasting PDEs (instead of ODEs) such as 1D convection, diffusion, etc under OOD scenarios is an important future research direction; this would include an expanded set of basis functions including differential operators. Furthermore, given the nature of the PDEs, the boundary conditions, especially in OOD scenarios, become critically important. Ensuring that the model accurately represents the physics at the boundaries while generalizing well in OOD scenarios is a non-trivial task. This entails developing new techniques for imposing and learning boundary conditions within the MetaPhysiCa framework. 

We discuss including PDEs as future work in the Conclusions section.

Q2: Have authors tested on stiff ODE settings where traditional physics-informed models fail?

A2: Traditional PIML methods including SINDy typically struggle for stiff ODEs because the dynamical system contains both slow and fast varying states [2,3]. Unfortunately, MetaPhysiCa does not solve this PIML issue with stiff ODEs, rather inherits this weakness. For instance, in our experiments, it is unable to identify the stiff ODE describing Robertson's chemical reaction [1] with fast and slow dynamics. One proposed solution in the literature is to learn the ODE only for slow-varying dynamics and use a neural network for the fast dynamics [2]; however this is not guaranteed to be OOD robust due to the neural network component (as discussed in Section 3.1). We believe solving this issue while preserving OOD robustness is an important future work. 

Q3: The assumption that the collection of m possible (appropriate) basis functions are always available seems too strong to the reviewer. Are there contexts where all sub-parts of the full applicable basis might not be present in the pre-trained network? Has this been tested?

A3: That is a good point that we now address in the revised submission. We have added clarifying statements in the updated submission. Specifically, we emphasize in Section 3.1 that appropriate basis functions must be present in the architecture for OOD extrapolation. Without the algorithmic alignment with appropriate basis functions, the in-domain approximations of the true model may not hold when the inputs are out-of-domain (as shown in Figure 2a).

In the updated submission we perform an interesting experiment (thank you for the suggestion!). We repeat the damped pendulum experiment without algorithmic alignment, i.e., appropriate basis functions required to learn the ground truth dynamics are not present. We repeat the damped pendulum experiment without sine/cosine basis functions in Appendix D.6.

In the absence of $\sin(\theta_t)$ term to learn the true dynamics of the damped pendulum system, MetaPhysiCa learns a truncated Taylor series approximation of this term via $\theta_t$ and $\theta_t^3$ terms: 

$\frac{d\theta_t}{dt} = W_1 \omega_t; \frac{d\omega_t}{dt} = W_2 \theta_t + W_3 \theta_t^3 + W_4 \omega_t$. 

Table 11 in Appendix D.6  shows that after test-time adaptation, this learnt model achieves $1.5\times$ to $2\times$ better OOD test NRMSE than the best baseline, but is worse than MetaPhysiCa with sinusoidal basis functions included.

References:

[1] H. H. Robertson, The solution of a set of reaction rate equations, in J. Walsh (Ed.), Numerical Analysis: An Introduction, pp. 178–182, Academic Press, London (1966).

[2] Abdullah, Fahim, Zhe Wu, and Panagiotis D. Christofides. "Data-based reduced-order modeling of nonlinear two-time-scale processes." Chemical Engineering Research and Design 166 (2021): 1-9.

[3] Abdullah, Fahim, and Panagiotis D. Christofides. "Data-based modeling and control of nonlinear process systems using sparse identification: An overview of recent results." Computers & Chemical Engineering (2023): 108247.

We thank the reviewer for their support and insightful feedback. We give detailed answers below.

Q1: My main concern with this paper is the robustness and completeness of the empirical results. Authors developed two OOD scenarios and picked one for each dynamical system.

A1: Thank you for the question. We wish to clarify that both the OOD scenarios are tested for all 3 dynamical systems, presented in Appendix due to lack of space. Table 1 (in the main text) and Tables 2&3 (in Appendix, Pg 17) show both OOD scenarios for epidemic modeling, damped pendulum and predator-prey respectively. In both OOD scenarios for all 3 dynamical systems, MetaPhysiCa is able to obtain significantly more robust predictions than baselines.

Q2: authors use limited in- and out-distribution "pairs": only one single distribution for training, and another single distribution for testing. 

A2: Thanks for the interesting suggestion! In the revised submission we have added results for multiple in- and out-of-distribution pairs in Appendix D.5. Table 9 shows that under a variety of training ranges for initial conditions and parameters $W^*$, MetaPhysiCa is able to learn the true structure of the damped pendulum dynamical system. Given the learnt structure, Table 10 shows OOD NRMSE for different OOD ranges for initial conditions and parameters. 

Q3: Scope is limited by not accounting for interactions between basis functions… Additionally, by confining the approach to ODE-style dynamics, it may not accurately reflect the complexity of real-world systems.

A3: We thank the reviewer for the comment. The review will find Appendix D.4 interesting, where we extend MetaPhysiCa to a more expressive SCM by composing the given basis functions (e.g., to obtain terms such as $sin(x^2)$, etc.). MetaPhysiCa with such an expressive SCM shows OOD performance gains on a complex ODE task (Appendix D.4), but sometimes suffers from learning stiff ODEs during optimization due to the complexity of such a 2-layer composition procedure. Better optimization techniques may help alleviate this problem

We agree that extending MetaPhysiCa to forecasting PDEs (instead of ODEs) under OOD scenarios is an important future research direction; this would include an expanded set of basis functions including differential operators, and require careful consideration of OOD boundary conditions. In our revised submission we discuss both these limitations in the Conclusions section.

Q4: Expanding the method to incorporate prior knowledge more deeply could further improve model performance and applicability.

A4: We agree with the reviewer. In application-specific scenarios, prior physics knowledge could be further incorporated (e.g., known invariances, dimensionless variables, mesh structures for fluid simulations, etc.). The basic principles of MetaPhysiCa would still be preserved: Orthogonal basis functions, identifiability of the equivalent causal model for OOD robustness. We envision MetaPhysiCa as a starting point for OOD robustness in various application-specific scenarios.

Q5: A real epidemic does not follow the SIR model but there are some characteristics of the SIR model that are useful. How can we adapt your work to handle such real-world dynamical system?

A5: That is an interesting suggestion. In order to verify MetaPhysiCa we focused on models where we were sure there would be a unique causal structure for the train and test data (i.e., the basis functions were nearly orthogonal under the support of the training data). For more complex models, we recommend adding more complex basis functions. There is an interesting set of future works that could design task-specific basis functions and, potentially, design their associated regularization objectives for better learning the equivalent causal model for basis functions that are not orthogonal over the support of the training data.