Generalizing to New Dynamical Systems via Frequency Domain Adaptation

Thank the reviewer for the constructive feedback.

Distinction from CoDA: We highlight two main contributions distinguishing our work from CoDA: (1) we are the first to emphasize adaptation in the frequency domain, and demonstrate its potential for generalizing to unseen environments. (2) Our hierarchical disentangling via the splitter $K$ and rapid adaptation by conditioning on $c_e$ results in memory and computational efficiency, facilitating its practical use. For experiments on NS dataset, CoDA uses FNO as the backbone, while it needs to update much more parameters (0.930M) than our approach (0.096K). Moreover, CoDA requires more optimization iterations for convergence as observed in Figure 4 (c)&(d).

Comparison with feed-forward-based meta-learning: We agree that the presented feed-forward-based meta-learning approach doesn't need to update parameters during test, it adjusts the predictor implicitly using few-shot observations. We have included this paper in revised version. Here, we evaluate three Bayesian meta-learning methods introduced in this reference for inter-trajectory adaptation tasks on NS dataset:

As seen, these considered methods cannot produce accurate predictions for unseen dynamics. We conjecture that this is mainly due to two factors: (1) heavy reliance on a large meta-training set with diverse dynamics. In their original experimental setup, there are 16 environments, each equipped with 4096 trajectories for training, while we only have 5 environments with 50 trajectories each. (2) Too scarce observation for adaptation. They require 3-shot trajectories and an observed window of 5 for adaptation, while we are in a 1-shot task and the observed window is 1. As a result, their inferred initial value for the latent dynamics is unreliable.

Benefits of Fourier transform-based network over CNNs: CNNs are known to have a ``spectral bias’’ that favors low-frequency functions and limits their ability to represent the high-frequency content [Rahaman2019] which is crucial for accurate dynamics forecasting. On the other hand, Fourier transform-based networks can easily capture the higher frequency information [Tancik2020] and have demonstrated their capability to model complex PDEs [Li2021]. Furthermore, each FNO layer processes global and local information simultaneously via weight multiplication of different modes, so it can naturally incorporate our hierarchical disentangling via the splitter $K$ and rapid adaptation via conditioning on $c_e$, resulting in improved memory and computational efficiency for adaptation.

Clarifications of notations: $F_e$ is the vector field describing the ground truth dynamics, $f^e$ in Eq. 8 is the same as $F_e$. $z^{(l)} \in \mathbb{R}^{m}$ and $W_{env}^{(l)} \in \mathbb{C}^{\hat{k} \times m \times d_c}$. We have carefully revised the notations in our manuscript for clarity.

ERM-adp performs worse than ERM: ERM-adp updates all model parameters without regularization, which makes it susceptible to overfitting the observations and leads to inferior outcomes than ERM. The adaptation is performed by minimizing the loss on observations using Adam.

Comparison with using all Fourier modes: FNSD achieved only a marginal improvement over using all Fourier modes mainly attributed to the already high prediction precision, as well as the fact that the modes acquired by Fourier transform are limited for LV dataset, our model tends to employ more modes for better performance.

Using swish activation and cosine annealing schedule: In our original experiments, we kept all other settings the same for the comparison baselines, including using swish activation in backbone networks and Adam optimizer. We evaluate them with/without cosine annealing schedule on LV dataset:

Cosine annealing schedule had a negative impact on most of the comparison methods, indicating it is not a universal technique that can benefit all methods.

We have revised our manuscript accordingly and provided qualitative comparisons in the Appendix.

References

[Rahaman2019] Nasim Rahaman, et al. On the Spectral Bias of Neural Networks. ICML, 2019.

[Tancik2020] Matthew Tancik, et al. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. NeurIPS, 2020.

[Li2021] Zongyi Li, et al. Fourier Neural Operator for Parametric Partial Differential Equations. ICLR, 2021.

Thank the reviewer for the valuable questions.

Modification of FNO: We highlight our three main contributions in this submission as follows: (1) We address the generalization issue of neural learned simulators which is an essential but rather new area. The original FNOs suffer from poor generalization when facing distribution discrepancy. (2) We propose a novel hierarchical disentangling and rapid adaptation mechanism in the Fourier domain, which is lightweight, memory-efficient and computationally fast. (3) We experimentally demonstrate the superior generalization capability of our approach for various dynamical systems in a parameter-efficient and adaptation-efficient manner.

Qualitative results: Thanks for the suggestion, we have included qualitative comparisons on the GS dataset (Figure 7 and 8 in Appendix C) and NS dataset (Figure 9 and 10 in Appendix C) in our revised manuscript.

Adapting using an increasing number of trajectories: We experimentally compare the performance of CoDA with l2 regularization and our FNSDA with different numbers of trajectories for the inter-trajectory adaptation task on the LV dataset. The results are presented in the table below:

FNSDA achieves a consistent decrease in error as the number of trajectories for adaptation increases. This indicates that FNSDA can effectively leverage more data to improve its generalization ability.

The dimension of $c_e$: We examine the performance of FNSDA with different dimensions of $c_e$ ranging from {2, 5, 10, 15, 20}. The results in RMSE are shown in the table below:

FNSDA achieves the best performance when the dimension of $c_e$ is set to 10. It also shows consistent results for dim 2, 15, and 20, and slightly deteriorates for dim 5. We speculate that $c_e$, as a key component of FNSDA, learns to infer the latent code of the actual environmental parameters when generalizing to a new environment, thus its dimension is closely related to its learning ability.

Ablation on $\beta_e$: We experimentally evaluate its impact with the results in RMSE presented as follows

As seen, fixing $\beta_e$ slightly worsens the performance, while fixing $c_e$ significantly degrades the performance. This demonstrates the critical role of $c_e$ for FNSDA’s generalization capability.

Fig. 4 (c) and (d): Thanks for the suggestion, we have improved this figure for better visual clarity and presentation.

The influence of the number of modes: In our preliminary experiments, we found that reducing the number of Fourier modes would consistently lead to worse performance. This is because this operation impairs the recovery of input signals after performing inverse FFT. This motivates our introduction of a learnable splitter $K$ that allows us to preserve all Fourier modes but only update some environment-specific part of them during adaptation.

The choice of solver: We chose the solver mainly for the sake of fairness. We followed the same solver selection as the official implementations of [Yin2021] and [ Kirchmeyer2022].

The effect of $T_{ad}$: We fix the total length as $T=20s$ and vary the value of $T_{ad}$ ranging from {2.5, 5, 7.5, 10, 14} for extra-trajectory adaptation task on the LV dataset, the results are listed below

As seen, increasing the value of $T_{ad}$​ leads to a consistent improvement in prediction accuracy, which aligns with our intuition on the matter.

Time cost for adaptation: We measure the adaptation time cost by reporting the number of iterations required for convergence in Figure 4 (c) and (d). Each method takes a similar amount of time for one iteration, except for FOCA, which employs a meta-learning framework involving bi-level optimization in which each iteration consists of 10 gradient descent steps in the inner loop.

References

[Yin2021] Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, Patrick Gallinari. LEADS: Learning Dynamical Systems that Generalize Across Environments. NeurIPS, 2021.

[Kirchmeyer2022] Matthieu Kirchmeyer, Yuan Yin, Jeremie Dona, Nicolas Baskiotis, Alain Rakotomamonjy, Patrick Gallinari. Generalizing to New Physical Systems via Context-Informed Dynamics Model. ICML, 2022.

Thank the reviewer for recognizing our contribution and offering helpful writing suggestions.

The definition of environment: In our manuscript, we use the term ‘environment’ to refer to the specific vector field $F_e$ that governs the dynamics of the system. Trajectories that are generated from different initial values but the same $F_e$ are regarded as belonging to the same environment. We acknowledge that varying the boundary conditions (BC) can also produce different environments, but we follow the previous works [Yin2021] and [Kirchmeyer2022] that only investigate the generalization across different model coefficients in $F_e$. We visualize their impact on the flow dynamics by changing the viscosity in Figure 6 in the Appendix. As shown, varying the viscosity tends to shift the flow in nearby regions, but it can affect the low and high Fourier spectrum due to error accumulation. We leave the exploration of the effect of BC as future work.

More experiments on PDEs: We agree with the importance of exploring more PDEs. In this submission, we follow the setting of [Kirchmeyer2022] and [Yin2021] to explore the generalization issue for generic neural learned simulators. PDEs, including the Glycolitic-Oscillator and Navier-Stokes dynamics, are considered. We would like to claim that current research on neural learned simulators mainly focuses on the approximation capability for complex physical systems. The generalization issue, although crucial, is still in its infancy. We believe our work provides a valuable foundation for inspiring future research on neural learned simulators, especially for their practical applications.

Deviations in the model coefficients: This is an interesting experiment and we have conducted a similar setup on the LV dataset with the results shown in Fig 4. (b). In this experiment, the training environments are set as $\beta, \delta \in \\{0.5, 0.75, 1.0 \\}^2$, the interpretable test environment uses $\beta=0.625, \delta=0.625$ and the not interplatable test environment uses $\beta=1.125, \delta=1.125$. As observed, all methods perform worse in the not interpretable test environment, and our method consistently achieves a low forecast error in this challenging scenario.

Reynolds number for NS dataset: The Reynolds numbers for the five training environments are given by $\\{1250, 1111, 1000, 909, 833\\}$ and for the four test environments by $\\{1176, 1053, 952, 870\\}$.

References

[Kirchmeyer2022] Matthieu Kirchmeyer, Yuan Yin, Jeremie Dona, Nicolas Baskiotis, Alain Rakotomamonjy, Patrick Gallinari. Generalizing to New Physical Systems via Context-Informed Dynamics Model. ICML, 2022.

[Yin2021] Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, Patrick Gallinari. LEADS: Learning Dynamical Systems that Generalize Across Environments. NeurIPS, 2021.

Thank the reviewer for the detailed review and constructive comments.

Discrepancies in Fourier space when initial conditions or PDE coefficient vary: We agree with the reviewer’s suggestion that an experimental comparison of varying initial values and PDE coefficient for dynamics in the Fourier space can further enhance the clarity of our manuscript. To this end, we visualize their influence on generated trajectories by comparing their difference to a fixed trajectory on the NS dataset (Figure 5 in Appendix C). As seen, varying initial values only can change the flow dynamic immensely, along with significant changes in low and high Fourier spectrums. While varying victory tends to shift the flow in nearby regions, and it can also change the low and high Fourier spectrum due to error accumulation. In our original submission, we report the generalization results on different initial values and PDE coefficient simultaneously existing, which is a more challenging but realistic setup. The generalization results on varying initial values only (in-domain test) have been included in our revised version in Table 6 of Appendix C.

Theoretical analysis of Fourier modes for differential equations: We appreciate the reviewer’s suggestion that theoretical analyses on some simple differential systems can offer new insights for supporting our method. Generally, our work is theoretically motivated by that modeling in the Fourier domain can represent both high and low frequency content without the spectral bias of CNNs towards low frequencies [Rahaman2019]. We verify in Figure 5 that varying the environmental parameters can influence the dynamics both in low and high frequencies. Besides, these low and high frequencies, capturing the local and global information of input signals, can be separated by a customized filter [Tveito1998]. This facilitates our learnable splitter to disentangle them. Last but not least, our trainable weight tensors are direct parameterizations of one kernel [Li2021], and we can modify them to tune the kernel’s spectrum as suggested by [Tancik2020].

Limitations of FNSDA: As a neural network-learned simulator, FNSDA requires high-quality data for model training. Besides, its scalability towards much larger and more complex systems, such as those encountered in climate modeling or large-scale biological networks, is still underexplored. We discuss this in Appendix A in our revised version.

About Fig. 4: The four subfigures show the ablation of training techniques, generalization to different distribution discrepancies, and convergence speed on two adaptation tasks, respectively. We have polished them for better visual clarity and presentation.

Learning the splitting via K: The splitting is learned by minimizing the prediction error on the training environments, so it implicitly conditions on all the training environments.

Intuitive explanations for $W_e$ and $W_s$: These weight tensors are designed to modify the frequency spectra. They are direct parameterizations for the Fourier transform of a kernel function [Li2021].

The penalty on $c_e$: We constrain $c_e$​ to be close to zero to facilitate fast adaptation to new environments. We then conduct a parameter sensitivity analysis w.r.t. $\lambda$ on the LV dataset to assess its influence. The results are shown in the table below:

The results indicate that FNSDA has stable performance under different levels of penalty on $c_e$.

Data intensiveness analysis: We empirically evaluate the performance of FNSDA with different numbers of trajectories for adapting to new environments. The results are presented in the table below:

The table shows that FNSDA achieves a consistent decrease in error as the number of trajectories for adaptation increases. This indicates that FNSDA can effectively leverage more data to improve its generalization ability.

We have revised our manuscript according to the above discussion and provided qualitative comparisons in the Appendix to demonstrate the effectiveness of our method.

References

[Rahaman2019] Nasim Rahaman, et al. On the Spectral Bias of Neural Networks. ICML, 2019.

[Tveito1998] Aslak Tveito, et al. Introduction to Partial Differential Equations: A Computational Approach. Springer, New York, 1998.

[Li2021] Zongyi Li, et al. Fourier Neural Operator for Parametric Partial Differential Equations. ICLR, 2021.

[Tancik2020] Matthew Tancik, et al. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. NeurIPS, 2020.