Learning semilinear neural operators: A unified recursive framework for prediction and data assimilation.

We thank the Reviewer for the constructive suggestions which helped us to improve our paper. Regarding each of your suggestions:

• Explanation of observer design: Thank you for your suggestion, we have modified the paper in order to clarify the conditions for the observer design, which now reads:

  "By solving a nonlinear infinite-dimensional Riccati equation, one can design an operator $K(t)$ based on the PDE in equation 10 that can compute the solution $\hat{z}(t)$ over time, such that under the absence of noise and disturbances, and under mild additional conditions, the solution satisfies $||z(t)-\hat{z}(t)||_{\mathcal{Z}_x}\to0$ as $t\to\infty$ for sufficiently small initial errors (see Theorem 5.1 in [Afshar2023]). Furthermore, the authors show that under bounded disturbances $\omega(t)$ and $\eta(t)$, the estimation error can also be bounded for all $t$ (see Corollary 5.2 in [Afshar2023]). Similarly to the previous case, when $K(t)$ is strongly continuous the analytical form of a solution $\hat{z}(t)$ to (10) is given by:"

  However, we emphasize that the proposed methodology is a fully data-driven framework, thus, the conditions for the proposed framework to work are not directly comparable to those in [Afshar2022].

• Intuition for the parametrization of observer gain: To design the parametrization of the gain $K(t)$ as in Equation (15) we modified the gating intuition discussed in [Guen2020]. There, the gain works as a gate controlling the influence of the measurements in the updated states. When $K(t) = 0$ the measurement does not influence the updated states. Conversely, when $K(t)=I$, the latent dynamics is only driven by the measurements. However, this strategy does not accommodate the measurement operators $C$ and so, we modified their design. In our scenario, $K$ corrects the predicted state based on the error in the predicted measurements, thus, better approximating the observer design. Note, however, that when $C = I$, the same gating interpretation holds for the gain design in (15) for $K\equiv I$. To clarify this intuition we added the comment "To parametrize the observer gain $K$, we generalized the gating architecture used in (Guen & Thome, 2020) to cope with general measurement operators $C$ and the observer design:" when introducing (15) in the revised paper.

• Effect of warmup: We understand the Reviewer's point. Indeed, the proposed strategy uses the warmup data to improve its internal estimated states. Nevertheless, the presented experiments highlight the proposed strategy's benefits when compared with current state-of-the-art NO techniques. Also note that we performed experiments for different lengths of warmup periods $t_H$ (shown in Figure 3 of the main paper) which show that as the warmup time reduces the results of the proposed approach approach those of the standard FNO method. Nonetheless, our contribution is an observer-design-inspired framework that can be coped with different NO strategies besides FNO.

• Including colorbars on all figures: We have included colorbars on all figures in the revised paper.

An updated version of our paper incorporating the aforementioned changes will be uploaded by the end of the rebuttal period.

[Guen2020] Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11474–11484, 2020.

• NS and KdV as semilinear PDEs: For better convenience, let us recall here the expression of these PDEs, for the NS equation:

$

    \frac{\partial z(x,t)}{\partial t} + u(x,t)\nabla z(x,t) = v\Delta z(x,t) +f(x)

$

where $\nabla \cdot u(x,t)=0$, $z=\nabla \times u$;
and for the KdV equation, which we have write it into an easier-to-interpret form (compared to its form in the original paper) as:

$

    \frac{\partial z}{\partial t} = -z\frac{\partial z}{\partial x}-\frac{\partial^3 z}{\partial^3 x} 

$

In short, both the NS and KdV equations can be viewed as semilinear evolution PDEs as they contain 1) only a liner first order differential term involving the time variable, 2) linear differential terms that do not depend on the time derivatives of the solution $z$, and 3) nonlinear terms which also do not depend on the time derivatives of the solution $z$ and satisfy some regularity conditions. However, formally showing that this is the case is more laborious, thus we spare us both the work and point to some relevant references. For the NS equation, this can be found, for instance, in Section 3.3 of [Temam1982]. We have included this reference in the revised manuscript. For the KdV equation, which is known to be a third order semilinear PDE [Sonner2022], we have included a reference and also rewritten it using a clearer notation, as in the equation shown just above in this response, which we believe should make the connection to the general semilinear evolution PDE form easier for the reader to visualize.

An updated version of our paper incorporating the aforementioned changes will be uploaded by the end of the rebuttal period.

[Afshar2023] Sepideh Afshar, Fabian Germ, and Kirsten Morris. Extended Kalman filter based observer design for semilinear infinite-dimensional systems. IEEE Transactions on Automatic Control, 2023.

[Curtain2020] Ruth Curtain and Hans Zwart. Introduction to infinite-dimensional systems theory: a state-space approach, volume 71. Springer Nature, 2020.

[Temam1982] Roger Temam. Behaviour at time t=0 of the solutions of semi-linear evolution equations. Journal of Differential Equations, 43(1):73–92, 1982.

[Sonner2022] Stefanie Sonner. An introduction to partial differential equations. Radboud University, Nijmegen, IMAPP - Mathematics, 2022

We thank the Reviewer for the constructive suggestions to improve our paper. Regarding each of your suggestions:

• Derivation of equation 12: First, let us note that as explained in [Afshar2023] equation (11) is a solution to the observer system in equation (10) when $K(t)$ is strongly continuous, due to proposition 2.4 in [Afshar2023]. To improve clarity, we have made precise the need for the strong continuity of $K(t)$ for the solution in equation (11) to be valid right before equation (11) in the revised paper. Equation (12) consists of a time-discretization of equation (11). It can be derived by using the semigroup property of $T(t)$ [Curtain2020], which satisfies

$

T(t+t')=T(t)T(t')

$

for any pair $t,t'$. This way, identifying $t$ in equation (11) with the discrete time $t_k$ we can express $\hat{z}(t_k)$ as:

$

\hat{z}(t_k) ={}& T(t_k)\hat{z_0} + \int_{0}^{t_k}T(t_k-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds
\\\\
={}&  T(t_k-t_{k-1}+t_{k-1})\hat{z_0} + \int_{0}^{t_{k-1}}T(t_k-t_{k-1}+t_{k-1}-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds
\\\\
+{}& \int_{t_{k-1}}^{t_k}T(t_k-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds
\\\\
={}& T(t_k-t_{k-1})T(t_{k-1})\hat{z_0} + T(t_k-t_{k-1})\int_{0}^{t_{k-1}}T(t_{k-1}-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds
 \\\\
+{}& \int_{t_{k-1}}^{t_k}T(t_k-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds
\\\\
={}& T(t_k-t_{k-1}) \underbrace{\bigg[T(t_{k-1})\hat{z_0} + \int_{0}^{t_{k-1}}T(t_{k-1}-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds\bigg]}

\\ +{}& \int_{t_{k-1}}^{t_k}T(t_k-s)G(\hat{z}(s),s)ds + \int_{t_{k-1}}^{t_k}T(t_k-s)K(s)[y(s) - C\hat{z}(s)]ds ,

$

where we used the linearity of $T(t)$. By noting that the term inside the large brackets highlighted in the previous equation is equal to

$

T(t_{k-1})\hat{z_0} + \int_{0}^{t_{k-1}}T(t_{k-1}-s)\Big[G(\hat{z}(s),s) + K(s)[y(s) - C\hat{z}(s)]\Big]ds=\hat{z}(t_{k-1})

$

it can be seen that this equation is now in the same form as equation (12). We have included this derivation as an appendix in the revised paper, and added a remark pointing to this derivation right before equation (12) in the main body of the paper.

• Experiments with non-identity operator: We are running the experiments with NS with random full-rank $C$ matrices. We will include this extra experiment in an appendix.

• Conditions for observer design in Afshar's paper: We agree with the reviewer that the observer design could be used to inform the architecture of the observer gain $K(t)$. However, we emphasize that the proposed approach is fully data-driven, and thus, the conditions for it to work are not directly comparable to those in [Afshar2022]. Moreover, during our experiments, we were able to obtain significant improvements in the experimental results compared to the competing techniques when using the parametrization of the observer gain in (15), without need for a noiseless pre-training period. Thus, although a pretraining step seems like an interesting way of tightening the connection between the optimal observer design and the proposed data-driven strategy, this will be the focus of followup work. We modified the text in the paper above equation (11) in the paper, where we discuss the conditions under which the optimal observer design is expected to work, which now reads:

  "By solving a nonlinear infinite-dimensional Riccati equation, one can design an operator $K(t)$ based on the PDE in equation 10 that can compute the solution $\hat{z}(t)$ over time, such that under the absence of noise and disturbances, and under mild additional conditions, the solution satisfies $||z(t)-\hat{z}(t)||_{\mathcal{Z}_x}\to0$ as $t\to\infty$ for sufficiently small initial errors (see Theorem 5.1 in [Afshar2023]). Furthermore, the authors show that under bounded disturbances $\omega(t)$ and $\eta(t)$, the estimation error can also be bounded for all $t$ (see Corollary 5.2 in [Afshar2023]). Similarly to the previous case, when $K(t)$ is strongly continuous the analytical form of a solution $\hat{z}(t)$ to (10) is given by:''

We thank the Reviewer for the constructive suggestions to improve our paper. Regarding each of your suggestions:

• Scenarios where the proposed approach can have the most impact: We have discussions connecting NO literature with applications appearing in the introduction and related work sections of the manuscript. However, we understand that connecting the benefits of the data assimilation component of the proposed operator approximation framework is paramount. For this reason we added, just before stating the contributions of the paper in the introduction section, the following sentence: "This can have significant impact in practical applications including, e.g., weather and Earth surface temperature forecast (Pathak et al., 2022; Jiang et al., 2023), remote sensing imaging (Weikmann et al., 2021; Borsoi et al., 2021) and fMRI dynamics (Singh et al., 2021; Buxton, 2013)."

• How the model handles variations in the amount of measurements: The influence of the amount of measurements, defined by $\alpha$, is an important question. This is evaluated in the experiments whose results are shown in Table 3 in the main body of the paper. Specifically, this experiment evaluates the performance of NODA ranging from the case of $\alpha=0$% (with no data for assimilation) up to the case $\alpha=30$% (where one every three noisy samples is used for assimilation). It can be seen that NODA performs well compared to the baselines even when no data is available for assimilation (i.e., $\alpha=0$%), and that its performance increases gradually with the amount of data available for assimilation (with a steady reduction of approximately $0.005$ in the $\text{RelMSE}$ estimation error for every increase of 10% in $\alpha$).

• Details about computational resources and computational overhead: We would like to point the reviewer to Appendix B where we have provided the details of the computational resources used for conducting all the experiments. To compare the NODA with the competing methods with respect to the computational overhead required, we are running simulations to evaluate the timings of the different methods, and will include them in the revised version of the paper by the rebuttal deadline.

• Critical parameters or settings for achieving the robustness of the model to noise: In our practical experience the hyperparameters of NODA didn't play a huge role when considering noisy measurements. In fact we used the same hyperparameters across all experiments, which include both noiseless cases, as well as noisy data with different signal-to-noise ratios (20 and 30 decibels).

• User friendliness: Yes, we are indeed leveraging the recursive neural operator-based framework in an applied work that is being currently developed. Furthermore, we will make the source code publicly available upon acceptance of the paper. We also highlight that although the mathematical content that serves as the foundation for the proposed framework can be fairly involved, the overall implementation of the NODA algorithm is fairly straightforward since it boils down to a sequence of prediction and update steps shown in equations (13) and (14) of the paper.

An updated version of our paper incorporating the aforementioned changes will be uploaded by the end of the rebuttal period.

We thank the Reviewer for the constructive suggestions which helped us to improve our paper. Regarding each of your suggestions:

• Including timings for the experiments We are running the simulations to evaluate the timings, and will include them in the revised version of the paper.

• Including uncertainties in the experimental results: We thank the Reviewer for pointing that out. We added the uncertainties to our tables, and they are small compared with the performance gain obtained with NODA.

• Including more variety in the experiments: We note that an extra experiment in the Kortweg-De Vries (KdV) equation is present in the appendix of the submitted manuscript. Nonetheless, we are working on additional experiments for NS with $R_e\in$ {500, 5000} for FNO, MNO and NODA that will be included in the appendix of the revised manuscript by the rebuttal deadline.

• Justification for the benchmark methods: To the best of our knowledge, NODA is the first method to integrate NO learning, data assimilation, and a flexible system modeling that can also account for, e.g., measurement noise. Thus, it is indeed difficult to find baseline methods that would allow for a perfectly fair comparison. We have selected MNO and LSTM-based methods due to their recursive structure, which resembles one of the main aspects of our NODA, while FNO is used as a basic block in the architecture of NODA. We have clarified the justification for the selected baselines in the beginning of Section 4 in the revised paper, which reads:

  "The MNO and LSTM-based methods were used as baselines due to their recurrent nature, which is also one of the main aspects of NODA, while FNO was selected since it is one of the main building blocks used in the NODA architecture."

  The baselines proposed by the Reviewer are definitely relevant, and we have included them in the related work section of the revised paper. We are currently working to evaluate them on the considered datasets, but, unfortunately, due to limited time and computational resources we might not be able to include them before the end of the rebuttal period. Nevertheless, in recognizing the benefit of such as comparison, we are working on obtaining results to compare NODA with the Multiwavelet NO approach [Gupta2021]. Regarding the Bayesian approach proposed in [Magnani2022], we could not find its source code in online open repositories. For this reason, we only discuss it in the related work section.

• Including colorbars in the figures: We have included colorbars in all figures in the revised paper.

An updated version of our paper incorporating the aforementioned changes will be uploaded by the end of the rebuttal period.

[Gupta2021] Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet-based operator learning for differential equations. Advances in neural information processing systems, 34:24048–24062, 2021.

[Magnani2022] Emilia Magnani, Nicholas Kramer, Runa Eschenhagen, Lorenzo Rosasco, and Philipp Hennig. Approximate Bayesian neural operators: Uncertainty quantification for parametric PDEs. arXiv preprint arXiv:2208.01565, 2022.

Dear reviewer, we submitted the revised manuscript. In particular, we were also able to include the approach of [Gupta2021] as a benchmark method for the NS and KdV examples in our experiments: in brief, its performance stood between that of FNO and that of the proposed NODA. We believe to have addressed most of the reviewer's points and hope the revised manuscript matches your expectations. If so, we kindly ask to increase the rating provided in the first revision round.