TRENDy: Temporal Regression of Effective Nonlinear Dynamics

We would like to thank the author for these remarks. We have taken steps to clarify how our approach differs from existing methods both in premise and implementation. Crucially, unlike various state of the art approaches to solving forward problems in PDEs, including via neural operators (like the study by Li 2021 which we now cite in the revised manuscript), we are not specifically interested in fine-grained spatial prediction. Indeed, as we now show in several benchmark studies, learning a full spatial model, as with SINDyCP can lead to somewhat accurate bifurcation prediction (Fig. 2, rightmost data) but only at the cost of sensitivity to strong noise.

One of the principal benefits of learning a reduced-order model is the robustness which we have quantitatively demonstrated (Fig. 2, right panel). Further, we now show in several decoding and interpretability experiments (Figs. 3, A.5, A.6, A.7) that these latent representations of the visible dynamics still suffice to decode meaningful aspects of the input data. In short, we see our approach as one end of a robustness-prediction tradeoff, and our experiments explore how this robustness can actually be used to decode high-level properties of the underlying dynamics better than state of the art forward solving models. Naturally, future work could explore this tradeoff in greater detail.

We agree with the reviewer that details about training the neural ODE were not clear enough in the original manuscript. Training details are now included in full in the appendix (Sec. A.1.5).

We thank the reviewer for the constructive remarks. We have made numerous updates to the manuscript which address all of the reviewer’s comments. Notably, we now include several interpretability and decoding experiments which show that the TRENDy’s effective state still includes low-level information about the original state space without an explicit, learned decoder. We demonstrate that with only a few effective dimensions, we can decode pattern class at high accuracy (Fig. 3), an important problem in reaction-diffusion modeling. We also show that as the number of effective dimensions grows large, one can reconstruct the input pattern (Figs. 10-11). What’s more, we show how scattering coefficients can be used to explain spectral differences between these steady state patterns (Fig. 12). Further, we now systematically compare the scattering transform to other features, like average spatial gradients and spectral features, against which our method compares favorably.

In its revised version, following the reviewer’s comments, we have structured the paper so that the benchmarking now includes several other approaches (averaged spatial gradients, Fourier features, TRENDy with fewer coefficients) and focuses mostly on the Gray Scott case. Having benchmarked our approach in this way, the subsequent Brusselator example is meant to demonstrate specifically to evaluate our method across a huge swath of parameters (Fig. 4). The next experiments on the lizard data (and the new Ising case which we have added to the revised manuscript (Sec. A.1.5, Ising model), Figs. A.2, A.3) are meant to demonstrate yet further qualitative aspects of our approach which do not lend themselves neatly to standard quantitative benchmarking.

The reviewer’s request for different types of bifurcations is well met. In order to demonstrate both a categorically new type of system and additional interpretable aspects of our framework, we have included a new supplementary section on fitting an Ising model (see supplementary section A.1.5, Ising model). This new experiment shows how our approach can be deployed without further adjustments to systems with very different physics compared to those explored earlier.

Finally, we thank the reviewer for noting stylistic and presentation errors, which we’ve since corrected.

We thank the reviewer for acknowledging the strengths of our approach and for the numerous constructive remarks. The reviewer has rightly asked us to justify our lack of decoder as well as our use of scattering for dimensionality reduction. As we now demonstrate in the manuscript, scattering actually circumvents the need for an explicit decoder. Indeed, our premise is that often the effective dynamics are the dynamics of interest, while fine spatial details are nuisances to be abstracted away. Consider the lizard example: the goal of systems biologists studying patterning is generally not to predict the color of each individual scale, but rather to predict the general class of patterns an animal might produce (and how this process occurs).

A good effective model should (1) predict if there are different effective classes and (2) ensure that these classes are interpretable in terms of physical structures in the full state space. Our choice of the scattering transform was motivated by these two point. As we now show in several experiments, scattering coefficients can not only be used to predict bifurcations, but also can also be used to explain the difference between the spatial patterns (Fig. 3, Fig. A.7) that emerge during these bifurcations. And, in the limit of many coefficients, we also demonstrate how these coefficients can be decoded into actual images (Fig. A.6, A.6).

Crucially, this focus on effective dynamics distinguishes our work fundamentally from that of Long, Karniadakis and others. As these studies seek to accelerate the solutions of traditional forward or inverse PDE problems with data, they naturally make use of a reconstruction loss, which is often regularized with prior knowledge of physical laws. Our commitment to effective dynamics allows us to sidestep full reconstruction and thereby use no prior knowledge of underlying physics, which is often unavailable.Notably, none of these frameworks is parametric: they fit a solution to one equation and do not consider the bifurcation problem. TRENDy is designed to this end and the universal expressiveness of the scattering transform means that spatial structures in these many different parametric settings can be compactly represented with no retraining. Other frameworks which are parametric (SINDyCP) are in turn not effective. Our approach, in short, is a synthesis of the parametric and effective methods.

To emphasize these points and to answer the reviewer’s remarks regarding decoding, interpretability and the use of the scattering transform, we have added the following new experiments:

1. A classification experiment (Sec. 4, Fig. 3) whereby TRENDy’s learned state can be used to decode patterning classes of the Gray Scott model. Again, as our focus is not to reconstruct exact states but rather more qualitative patterning behaviors, it suffices to show that we can decode visually meaningful pattern classes from TRENDy’s dynamics on test data. This demonstrates:

1a. TRENDy can indeed map back to the observable space in a categorical, instead of pixelwise, sense (Fig. 3).

1b. Scattering and its hardwired features helps explain these classes in terms of specific structures (Fig. A.7). This is demonstrated with a straightforward analysis of which spatial scales and interactions between scales are predicted as a function of Gray Scott parameters.

1c. Increasing the number of scattering parameters helps facilitate an exact reconstruction of the input pattern at the cost of worsened bifurcation prediction (Fig. A.6, A.7). This tradeoff highlights effective dynamics as a theoretically different aim from exact spatial prediction.

2. A comparison between the scattering transform and new interpretable baseslines: average spatial gradients, Fourier features, and TRENDy with different numbers of dimensions (Figs. 2, 3).

3. A new experiment where TRENDy is fit to an Ising model with a varying temperature (Sec. A.1.5, Ising Model). This addresses both the reviewer’s request for new systems (Ising is stochastic and discrete-time) and the request for interpretability studies. We provide a systematic analysis of the dominant spatial scales predicted by TRENDy as temperature decreases towards the critical point (Fig. A.3).

We have also made numerous stylistic and graphical adjustments following the reviewer’s detailed remarks, notably:

1. New citations and explanations of related methods (Intro).
2. Revamped Fig. 1. 3 Notation has been cleaned overall, including all suggestions made by the reviewer.
3. Clarification that focus on the (still challenging) interpolation problem.
4. Fig. 2 is now simplified and includes benchmarks.
5. Appropriate citation and description of SINDyCP.
6. Experimental detail has now been moved to the appendix, including average compute times for training epochs, sizes of train vs test data, etc.
7. Examples of noisy data are given in Fig. A.4
8. We note potential learning of features in the discussion section.