KokerNet: Koopman Kernel Network for Time Series Forecasting

Thank you for taking the time to our work. Your valuable comments are helpful in improving our work. We will provide a point-to-point response.

Q1: What is the rationale behind this asymmetrical structure?

A1: In practice, the decoder can be any form. In KokerNet, we select MLP. That is because (1) one of the main contributions of this paper is constructing the measurement function and learning the Koopman operator to capture the dynamic patterns of the time series, which does not have any special requirements for the decoder. Therefore, we tend to select a simple model as the decoder. (2) To better optimize the proposed model. In our KokerNet, the Koopman operator learning can be simply formulated as: $\hat{\bf{x}} = \Phi_ {de}(\mathcal{K}g(\bf{x}))$, where $\bf{x}$ and $\hat{\bf{x}}$ are the input and output, respectively. $g$ denotes the kernel mapping with RFF, $\mathcal{K}$ denotes the Koopman operator, and $\Phi_ {de}$ denotes a decoder. If the decoder is selected from the RKHS, the model can be considered as a stack of the RFF mapping. The RFF mapping can be seen as the single layer of the neural network with the cosine activation. The stack of the RFF mapping with the multilayer periodic function is prone to local minima. To avoid this case, we select MLP as the decoder for this paper.

Q2: Ablation experiments in index

A2: For the time series decomposition, the process is that the Fourier transform is first performed in the original segments $\bf{X}_T = [\bf{x}_1, \bf{x}_2, \dots, \bf{x}_J ]$ to calculate the frequency spectrum and sort all frequencies by the number of occurrences. Then, the top $\alpha$ percent of the frequency spectrum are considered as the components of the stationary, while the remaining are considered as the components of the non-stationary. The difference between our KokerNet and Koopa is how to determine the value of $\alpha$. In our method, index $S_v$ can guide us to determine the value of $\alpha$. By contrast, Koopa tends to empirically determine the value of $\alpha$.

To evaluate the effect of the index $S_v$, we compare the performance under two cases, "index" and "emp", on ETTh2 and ILI datasets. "index" denotes the result guided by $S_v$. "emp" denotes the empirical case and shows the best result for different proportions of the stationary component under the decomposition case. In the "emp" case, we set the candidate range of the proportions as [10%, 20%,..., 90%] and select the most optimal result, where 10% represents that the stationary component accounts for 10%, while the non-stationary component accounts for 90%. All the results are reported in the following Table. For ETTh2 data, $S_v^{\text{ETTh2}} = 0.0865$, indicating that the proportion of non-stationary components is minimal and it does not need to conduct the decomposition. The result is better than the best result of the "emp" case. For the ILI dataset, $S_v^{\text{ILI}} = 0.2653$, meaning that it includes more non-stationary components. We will decompose the time series with $\alpha=30$. The result is equal to the best result of the "emp" case. To sum up, the index can effectively guide us to conduct the decomposition and lead to a better result.

Q3: Regarding the segmentation of non-stationary components, what criteria are used to determine the segmentation length? Is the final performance sensitive to this parameter?

A3: In our paper, the segmentation length is set as 48, the minimum forecasting length. In order to test the influence of the segmentation length on the performance, we compute the index under different segmentation lengths on ETTh2 and Exchange datasets. The results are shown in the following Table. As shown in the table, the segment length has a slight impact on the index $S_v$ but does not affect the configuration of $\alpha$. Therefore, the final performance is not sensitive to the segmentation length. In the revised manuscript, we will include it to analyze parameter sensitivity.

If you have any further questions, we will discuss them in a timely manner. That could help improve our work.

Thank you for addressing my concerns! I'd like to increase the confidence score.

Q2: Regarding the standard DMD-based methods

A2: We have briefly read the book you recommended. It primarily introduces two methods (HODMD and STKD) and their applications across various fields, while also recognizing their effectiveness in system analysis and forecasting. In the following, we will show the differences between the standard DMD-based methods and our KokerNet and the potential benefits of KokerNet over the DMD-based methods.

Difference (1) The way of learning. DMD-based methods are analysis methods based on data decomposition and do not require training. The main operation in DMD-based methods is matrix decomposition (singular value decomposition (SVD) or eigenvalue decomposition (EVD)). By contrast, our KokerNet is a neural network-based approach and requires training. Concretely, KokerNet can be formulated as: $\hat{\bf{x}} = \Phi_ {de}(\mathcal{K}g(\bf{x}))$, where $\bf{x}$ and $\hat{\bf{x}}$ are the input and output, respectively. $g$ denotes the kernel mapping with RFF, $\mathcal{K}$ denotes the Koopman operator, and $\Phi_ {de}$ denotes a decoder. We make $\mathcal{K}(\cdot)$ as the linear mapping, and thus $\mathcal{K}g(\bf{x})$ can be considered as a kernel network, stacking the non-linear kernel mapping with linear mapping. (2) We introduce a distribution module. A fundamental challenge with non-stationary time series is the time-varying distribution. Therefore, we introduce a distribution module to keep the forecasting to the distribution law of the time series or the system.

Benefits The main benefit of KokerNet over the DMD-based methods is that it reduces computational complexity, especially for large-scale problems. For DMD-based methods, SVD or EVD is commonly used, resulting in $\mathcal{O}(n^3)$ computational complexity. In contrast, our KokerNet involves only matrix multiplication and has $\mathcal{O}(nM)$ computational complexity, where $M \ll n$ is the sampling number in our paper.

Due to the difference between DMD-based methods and our KokerNet, discussed above, we just touched on the DMD-based methods a little bit in the introduction section, and do not perform the comparison with the DMD-based methods. Note that we have compared with the other Koopman operator-based methods (Koopa and KNF) that focus on learning the Koopman operator within the framework of neural networks. To include more "Koopman" based comparisons, we have added another two Koopman-based methods on Exchange and ECL datasets with the forecasting length $H=96,192$. The results are reported in the following Table. The results show that our KokerNet is comparable in most cases.

[1] Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series dynamics with koopman predictors. In Advances in Neural Information Processing Sys- tems, volume 36, pp. 12271–12290, 2023

[2] Rui Wang, Yihe Dong, Sercan O. Arik, and Rose Yu. Koopman neural operator forecaster for time-series with temporal distributional shifts. In Proceeding of the International Conference on Learning Representations, 2023

[3] Hui Wang, Liping Wang, Qicang Qiu, Hao Jin, Yuyan Gao, Yanjie Lu, Haisen Wang, and Wei Wu. Koopformer: Robust multivariate long-term prediction via mixed koopman neural operator and spatial-temporal transformer. In Proceedings of the International Joint Conference on Neural Networks, pp. 01–08, 2023.

[4] Liu Fu, Meng Ma, and Zhi Zhai. Deep koopman predictors for anomaly detection of complex iot systems with time series data. IEEE Internet of Things Journal, 2024.

If you have any further questions, we will discuss them in a timely manner. That could help improve our work.

Q1: Regarding the theorems involved in our paper

A1: Thanks for your insightful components, which are very helpful in improving the simplicity and readability of our paper. We will distill the theoretical framework, showcasing only the core insights in the revised manuscript. In the following, we will explain why the current theoretical framework is adopted.

Lemma C.1 In our work, one of the main contributions is that we tend to approximate the implicit kernel mapping by random Fourier feature (i.e., Eq. (3) in the main text), enabling learning the Koopman operators in a low-dimensional feature space. Taking into account the diversity of the reader, we present a detailed derivation of Eq. (3) in Lemma C.1.

Theorem C.1 In our work, we consider the implicit kernel mapping to be the measurement function $f$ in the Koopman framework $\mathcal{K}f(\bf{s}_ t) = f(\bf{s}_ {t+1})$ to map the data into a sufficiently high-dimensional feature space. Furthermore, we approximate the implicit kernel mapping by random Fourier feature, enabling learning the Koopman operators in a low-dimensional feature space. Theorem C.1 shows that when the sampling number meets certain conditions, the low-dimensional mapping can approach the high-dimensional mapping well. On the one hand, it shows that low-dimensional mapping can approximate the effect of high-dimensional mapping, providing strong theoretical support for our method. On the other hand, in addition to the performance, one also wants to know how many random features should be selected, such as the reviewer u2dN. Theorem C.1 indirectly provides a theoretical guide for us to choose the number of random features. As a result, we include Theorem C.1 in our work to highlight the interpretability theoretically.

Theorem C.2 (ii) We have meticulously reviewed Theorem C.2, especially Theorem C.2 (ii), and agree with your comment. We will distill (ii) in the revised manuscript to ensure the readability of our paper.

The main benefit of KokerNet over the DMD-based methods is that it reduces computational complexity,

Yes I can agree that that is a benefit. But it is hard to find it a genuinely important contribution. Most time series datasets that are worked with in practice are really quite small (especially in comparison to the huge amount data and parameters we consider in relation to langauge models, image data sets etc..). Time series problems hardly approach anything even close to that level. It is often much better to then take the O(N^3) complexity, work with a smaller overall cardinality of bespoke data that truly represents the dynamics of the underlying system, and then (as you have noticed / pointed), even without the need for training then learn the dynamics of the signal up to an incredibly high degree of accuracy.

Essentially I am coming to this from a pragmatic angle as if I were to make a recommendation to a friend / colleague / industry. In your title the word "Koopman" is used and we are aware that there a plethora of finite approximation Koopman based solutions for forecasting that have a large amount of convincing mathematical structure behind them, and which require absolutely minimal tuning for great results. I cannot forsee a situation where I would recommend using your model unfortunately over standard Koopman approximation principles already actively used in the econometrics fields, fluid flow fields, structural engineering fields (so we know they work! And for data situations much, much, much harder than what is typically explored in standard ML data set libraries in many cases).

If you could have tackled this problem head on and gave a convincing argument as to why your method is objectively better for certain (realistic) tasks in industry as opposed to other competing Koopman based I would most definitely reconsider, and that would peak further curiousity from myself. Unfortunately I dont see this being done here, little more than an argument that goes beyond "sure something like HODMD is successful, doesn't require much data, doesn't even require parameter training .... but at least we're linear". I don't believe this is the correct angle to take when working with time series data. Developing a spatio-temporal dataset of a wind tunnel fluid flow can cost upwards of 1-10M dollars. This field cannot be treated as other fields in ML where data feels endless, so the label of "big data" rings trues, and the linearity would be greatly appreciated.

Due to the difference between DMD-based methods and our KokerNet, discussed above, we just touched on the DMD-based methods a little bit in the introduction section, and do not perform the comparison with the DMD-based methods.

And there-in lies a major point. This is an important difference to comment on and address, which are you not addressing. In industry and pragmatic settings one situation is almost always favored over the other, as I re-iterate in many cases one does not come across the large data cases in time series anywhere near as those problems which can be solved via conventional DMD approaches used in the sciences and industry. In the future this point must be addressed and not simply given a passing mention.

If you want a reference for a paper I feel actually brings a lot of justice to the combination of Deep Learning and Koopman Operators without neglecting the DMD roots (which are practically what we work with in a finite world), it would be "Deep learning for universal linear embeddingsof nonlinear dynamics" by Lusch et al. As you can see in the paper a lot of effort has gone into understanding the fundemental physics of the issue which is where / how such operators are practically used a lot.

I appreciate taking my comments positively in this light (regarding to the comments on superfluous mathematics etc..). I do genuinely believe they will improve the paper for the better. Much appreciated.

Thank you for taking the time to read our rebuttal carefully and additional comments. DMD-based approaches appear closely related to our KokerNet, we will include more DMD-based approaches in the revised manuscript. In the following, we show the reason why your method is objectively better for certain (realistic) tasks and recommend it to others. In addition, we also compare our method with DMD and eDMD methods on the small datasets.

First, there is also a convincing mathematical structure behind our method. We model the temporal dependence of the time series as a kernel $k(\bf{x}_ t, \bf{x}_ {t+1}) = \langle f(\bf{x}_ t), f(\bf{x}_ {t+1}) \rangle_ {\mathcal{H}}$. Defining the integral operator $\mathcal{P}$ as $\mathcal{P}f(\bf{x}_ t) = \int k(\bf{x}_ t, \bf{x}_ {t+1}) f(\bf{x}_ {t+1})\mu(d(\bf{x}_ {t+1}))$. We have $k(\bf{x}_ t, \bf{x}_ {t+1}) = \sum_ {m=1}^M \lambda_ m \varphi_ m(\bf{x}_ t)\varphi_ m(\bf{x}_ {t+1})$, where $\lambda_ m$ is the eigenvalue and $\varphi_ m$ is the eigenfunction of the operator $\mathcal{P}$. It is known that when $M \to \infty$ the operator $\mathcal{P}$ commutes with the Koopman operator $\mathcal{K}$. An important fact here is that the space spanned by eigenfunctions is the same for commuting operators. Therefore, we approximate the eigenfunctions of the Koopman operator by approximating the kernel $k(\bf{x}_ t, \bf{x}_ {t+1})$ with the $\varphi_ m, m=1,\cdots$ (i.e., learning $g$ by the kernel network). Our theorem also proves this point.

Second, we are not limited to any one field (such as your concerns in industry, which may have a small dataset). In practical applications, there are also many forecasting tasks with large datasets, such as traffic and weather prediction. The data for these tasks is relatively easy to collect and sufficient. In these tasks, high computational complexity is unacceptable. In addition, when data is collected, distribution drift can occur for various reasons (e.g., heat generated by long-running equipment), which is also a crucial issue to consider, even for small data. As previously discussed, we introduce a distribution constraint module to tackle this issue.

Existing experiments show the superiority of our method in the large-scale dataset. To further demonstrate the performance of the proposal on the small dataset, we compare it with DMD and eDMD (The code of DMD and eDMD is from the public PyDMD codebase in GitHub). Since we do not have a small dataset that you are talking about in the industry, we take the first 500 points in time that we already have (ETTh1, ETTh2, ETTm1, ETTm2) as a standalone small data set to evaluate our method under $H=48$. The results (MSE) are shown in the following table and demonstrate that our KokerNet is superior to the DMD methods.

If you have any further questions, we will discuss them in a timely manner. That could help improve our work.

Q3: Could the authors include more insights into how they ensured their model's robustness to real-world shifts in data distribution, especially for rapidly changing non-stationary components?

A3: On the one hand, the kernel method has been proven to be a promising method to address the non-linear problem, and it can map the data into an infinite-dimensional feature space (i.e., RKHS) via implicit kernel mapping. We consider the implicit kernel mapping to be the measurement function $f$. Theorem 3.1 shows that when the number of samples meets certain conditions, the low-dimensional mapping can approach the high-dimensional mapping well. This theorem ensures that low-dimensional mapping in our KokerNet can approximate the effect of high-dimensional mapping. On the other hand, Therome 3.2 shows that the function, induced by $g$, can approximate the eigenfunction of the Koopman operator, which means that our KokerNet at the encoding stage can reveal the dynamic patterns of the time series or system.

In addition, we introduce a distribution module in our KokerNet. This module, especially for the non-stationary component, can keep the forecasting to the distribution law of the time series. We minimize the distribution loss $\mathcal{L}_ {\text{dis}}^{\text{ns}} = \mathcal{L}(\mathcal{N}(\bf{\mu}_ {\text{ns}}^i, \bf{\delta}_ {\text{ns}}^{2, i}), \mathcal{N}(\bf{\hat{\mu}}_ {\text{ns}}^i, \bf{\hat{\delta}}_ {\text{ns}}^{2,i})), i=2, \cdots, Q$, where $\mathcal{N}(\bf{\mu}_ {\text{ns}}^i, \bf{\delta}_ {\text{ns}}^{2, i})$ is the ground truth, and $\mathcal{N}(\bf{\hat{\mu}}_ {\text{ns}}^i, \bf{\hat{\delta}}_ {\text{ns}}^{2,i})$ is the prediction based on the $\mathcal{K}_ {\text{dis}}$. Furthermore, we align the predicted distribution with the distribution of forecasts by minimizing the loss $\mathcal{L}_ {\text{dis}}^{\text{alig}} = \mathcal{L}(\mathcal{N}(\hat{\bf{\mu}}_ i, \hat{\bf{\delta}}_ i^2), \mathcal{N}(\bf{\hat{\mu}}_ {\text{ns}}^i, \bf{\hat{\delta}}_ {\text{ns}}^{2,i}))$. Therefore, for the rapidly changing non-stationary, our KokerNet not only captures the evolution of the value of the data using Koopman operator learning module but also explores the dynamics of the distribution utilizing the distribution constraint module, enabling the prediction to follow the distribution patterns of the time series.

Furthermore, we also evaluate the performance of our KokerNet under different degrees of non-stationary. The non-stationary is reported in Table 6 and the performance is reported in Table 7 in Appendix E.2. All the results show that our KokerNet consistently outperforms the state-of-the-art models.

If you have any further questions, we will discuss them in a timely manner. That could help improve our work.

Thanks for your valuable comments, which have helped improve our work. We will provide a point-to-point response in the rebuttal.

Q1: Could the authors provide more detail on how the spectral kernel method integrates within the Koopman framework and specifically outline its role in enhancing efficiency?

A1: The core idea of the Koopman framework is to characterize the complicated evolution via an infinite-dimensional linear Koopman operator by acting on the measurement function. It is formulated as follows:

where $\mathcal{K}$ is the Koopman operator, $f: \mathbb{R}^d \to \mathbb{R}^D, D \to \infty$ is the measurement function.

Intuitively, the Koopman framework tends to map the data into an infinite-dimensional feature space by $f$ and characterize the non-linear complicated evolution in a linear manner by $\mathcal{K}$ within the feature space. The kernel method has been proven to be a promising method to address the non-linear problem, and it can map the data into an infinite-dimensional feature space (i.e., RKHS) via implicit kernel mapping. Therefore, we consider the implicit kernel mapping to be the measurement function $f$. Furthermore, the temporal dependence can be modeled by the kernel. In our KokerNet, we approximate the implicit kernel mapping by a low-dimensional feature mapping $g$ based on Bochner’s theorem (i.e., Eq. (3) in the main text). As a result, we integrate the spectral kernel within the Koopman framework.

Ideally, we want to map the data to a sufficiently high-dimensional feature space to linearize the complex evolution of a dynamic system. However, this would be computationally expensive. In our KokerNet, we approximate the implicit high-dimensional kernel mapping by a low-dimensional feature mapping $g$ based on Bochner’s theorem (i.e., Eq. (3) in the main text). Theorem 3.1 shows that when the number of samples meets certain conditions, the low-dimensional mapping can approach the high-dimensional mapping well. This theorem not only shows the efficiency (low-dimensional mapping can approximate the effect of high-dimensional mapping, and thus reduce the compute cost) but also provides a theoretical guide for us to choose the number of random features. In addition, we compared the model efficiency in three aspects, including forecasting performance, training time, and memory footprint in section 4.3. The results are reported in Appendix E.4, showing that our KokerNet has better forecasting performance with less training time and memory footprint.

Q2: Why is multiresolution DMD, which appears closely related, not discussed as part of the literature? Could the authors clarify the differences and potential benefits of KokerNet over multiresolution DMD?

A2: The differences between multiresolution DMD and our KokerNet are: (1) The way of learning. Multiresolution DMD is an analysis method based on data decomposition and does not require training. The main operation in multiresolution DMD is matrix decomposition. By contrast, our KokerNet is a neural network-based approach and requires training. Concretely, KokerNet can be formulated as: $\hat{\bf{x}} = \Phi_ {de}(\mathcal{K}g(\bf{x}))$, where $\bf{x}$ and $\hat{\bf{x}}$ are the input and output, respectively. $g$ denotes the kernel mapping with RFF, $\mathcal{K}$ denotes the Koopman operator, and $\Phi_ {de}$ denotes a decoder. We make $\mathcal{K}(\cdot)$ as the linear mapping, and thus $\mathcal{K}g(\bf{x})$ can be considered as a kernel network, stacking the non-linear kernel mapping with linear mapping. (2) We introduce a distribution module. A fundamental challenge with non-stationary time series is the time-varying distribution. Therefore, we introduce a distribution module to keep the forecasting to the distribution law of the time series or the system.

The main benefit of KokerNet over the multiresolution DMD is that it reduces computational complexity. For multiresolution DMD, singular value decomposition (SVD) or eigenvalue decomposition (EVD) is commonly used, resulting in $\mathcal{O}(n^3)$ computational complexity. In contrast, our KokerNet involves only matrix multiplication and has $\mathcal{O}(nM)$ computational complexity, where $M \ll n$ is the sampling number in our paper.

We admit that multiresolution DMD appears closely related to our KokerNet. Due to the difference, discussed above (1), we just touched on the DMD-based method a little bit in the introduction section. We will include more multiresolution DMD approaches in the revised related works.

Thanks for your feedback. Basically, I can understand the contribution and organization of this paper more clearly now. Thus, I will keep the original rating.

Dear Reviewer SgkE,

We just want to let you know that we updated the comparison for the DMD-based methods. We think this modification is related to your comments about the DMD. Thank you again for your time and valuable comments.

We take the first 500 data points that we already have (ETTh1, ETTh2, ETTm1, ETTm2) as a standalone small data set to evaluate our method under . The results (MSE) are shown in the following table and demonstrate that our KokerNet consistently outperforms the others.

[1] Meanti, G., Chatalic, A., Kostic, V., Novelli, P., Pontil, M., & Rosasco, L. (2024). Estimating Koopman operators with sketching to provably learn large scale dynamical systems. Advances in Neural Information Processing Systems, 36.

[2] Dimitrios Giannakis, Amelia Henriksen, Joel A. Tropp, and Rachel Ward. Learning to forecast dynamical systems from streaming data. SIAM Journal on Applied Dynamical Systems, 22(2): 527–558, 2023

Thank you for taking the time to our work. Your valuable comments are helpful in improving our work. We will provide a point-to-point response in the rebuttal.

Q1: When is the constraint used?

A1: In the following, we will provide the detailed process and show it in the following Algorithm 1.

Specifically, for the stationary component $\bf{X}_ {\text{s}}$, we assume the distribution $\mathcal{N}(\bf{\mu}_ {\text{s}}, \bf{\delta}_ {\text{s}}^2)$ is constant. We just need to let the distribution of the forecasting $\hat{\bf{X}}_ {\text{s}}^{\text{fore}}$ closed to the real distribution $\mathcal{N}(\bf{\mu}_ {\text{s}}, \bf{\delta}_ {\text{s}}^2)$ by minimizing the distribution loss $\mathcal{L}_ {\text{dis}}^{\text{s}} = \mathcal{L}(\mathcal{N}(\bf{\mu}_ {\text{s}}, \bf{\delta}_ {\text{s}}^2), \mathcal{N}(\hat{\bf{\mu}}_ {\text{s}}, \hat{\bf{\delta}}_ {\text{s}}^2))$, where $\mathcal{N}(\hat{\bf{\mu}}_ {\text{s}}, \hat{\bf{\delta}}_ {\text{s}}^2)$ is the distribution of the forecasting $\hat{\bf{X}}_ {\text{s}}^{\text{fore}}$.

For the non-stationary component $\bf{X}_ {\text{ns}}$, similar to the local Koopman operator learning in section 3.2, we explore the variation of the distribution by introducing the distribution Koopman operator $\mathcal{K}_ {\text{dis}}$ (Eq. (13)). In this section, we minimize the distribution loss $\mathcal{L}_ {\text{dis}}^{\text{ns}} = \mathcal{L}(\mathcal{N}(\bf{\mu}_ {\text{ns}}^i, \bf{\delta}_ {\text{ns}}^{2, i}), \mathcal{N}(\bf{\hat{\mu}}_ {\text{ns}}^i, \bf{\hat{\delta}}_ {\text{ns}}^{2,i})), i=2, \cdots, Q$, where $\mathcal{N}(\bf{\mu}_ {\text{ns}}^i, \bf{\delta}_ {\text{ns}}^{2, i})$ is the ground truth, and $\mathcal{N}(\bf{\hat{\mu}}_ {\text{ns}}^i, \bf{\hat{\delta}}_ {\text{ns}}^{2,i})$ is the prediction based on the $\mathcal{K}_ {\text{dis}}$. Furthermore, we align the predicted distribution with the distribution of forecasts by minimizing the loss $\mathcal{L}_ {\text{dis}}^{\text{alig}} = \mathcal{L}(\mathcal{N}(\hat{\bf{\mu}}_ i, \hat{\bf{\delta}}_ i^2), \mathcal{N}(\bf{\hat{\mu}}_ {\text{ns}}^i, \bf{\hat{\delta}}_ {\text{ns}}^{2,i}))$.

Q2: Global and local Koopman operators and loss function

A2: For the global Koopman operator, we set $\bf{X}_ {\text{s}} = [\bf{X}_ {\text{s}}^{\text{back}}, \bf{X}_ {\text{s}}^{\text{fore}}]$. We first encode $\bf{X}_ {\text{s}}^{\text{back}}$ into the feature space *i.e., \bf{Z*_ {\text{s}}^{\text{back}} = g(\bf{X}_ {\text{s}}^{\text{back}})}, and then forecast by the global Koopman operator $\bf{Z}_ {\text{s}}^{\text{fore}} = \mathcal{K}_ {\text{s}}\bf{Z}_ {\text{s}}^{\text{back}}$. Finally, we decode the forecasting by a decoder: $\hat{\bf{X}}_ {\text{s}}^{\text{fore}} = \Phi_ {\text{de}}(\bf{Z}_ {\text{s}}^{\text{fore}})$. In this framework, the operation $\bf{Z}_ {\text{s}}^{\text{fore}} = \mathcal{K}_ {\text{s}}\bf{Z}_ {\text{s}}^{\text{back}}$ is a linear mapping and is learned by minimizing the loss $\mathcal{L}_ {\text{fore}}^{\text{s}} = \ell(\bf{X}_ {\text{s}}^{\text{fore}}, \hat{\bf{X}}_ {\text{s}}^{\text{fore}})$.

For the local Koopman operator, we first divide the non-stationary component $\bf{X}_ {\text{ns}}$ into $Q$ segments $\bf{X}_ {\text{ns}} = [\bf{x}_ 1, \bf{x}_ 2, \dots, \bf{x}_ Q]$, and encode them into the feature space *i.e., \bf{z*_ i = g(\bf{x}_ i), i=1,\dots, Q}. Then, we construct $\bf{Z}^{\text{back}} = [\bf{z}_ 1, \dots, \bf{z}_ {Q-1}]$ and $\bf{Z}^{\text{fore}} = [\bf{z}_ 2, \dots, \bf{z}_ {Q}]$, and compute $\mathcal{K}_ {\text{ns}} = (\bf{Z}^{\text{back}}) (\bf{Z}^{\text{fore}})^\top$. Moreover, the computed $\mathcal{K}_ {\text{ns}}$ is used to forecast $\hat{\bf{z}}_ i = \mathcal{K}_ {\text{ns}}\bf{z}_ {i-1}$, and a decoder is applied to decode the forecasting $\hat{\bf{x}}_ i = \Psi_ {\text{de}}(\hat{\bf{z}}_ i), i=2,\dots, Q$. In this process, $\mathcal{K}_ {\text{ns}}$ is learned by minimizing the loss $\mathcal{L}_ {\text{fore}}^{\text{ns}} = \sum_ {i=2}^Q \ell(\bf{x}_ i, \hat{\bf{x}}_ i)$.

In our KokerNet, the loss function consists of three parts, including the forecasting loss $\mathcal{L}_ {\text{fore}} = \mathcal{L}_ {\text{fore}}^{\text{s}} + \mathcal{L}_ {\text{fore}}^{\text{ns}}$, the distribution loss $\mathcal{L}_ {\text{dis}} = \mathcal{L}_ {\text{dis}}^{\text{ns}} + \mathcal{L}_ {\text{dis}}^{\text{alig}}$, and the reconstruction loss $\mathcal{L}_ {\text{rec}} = \ell(\bf{X}_ T, \hat{\bf{X}}_ {\text{s}} + \hat{\bf{X}}_ {\text{ns}})$, $\hat{\bf{X}}_ {\text{s}} = [\Phi_ {\text{de}}(\bf{Z}_ {\text{s}}^{\text{back}}), \Phi_ {\text{de}}(\bf{Z}_ {\text{s}}^{\text{fore}})]), \hat{\bf{X}}_ {\text{ns}} = [\Psi_ {\text{de}}(\bf{z}_ 1), \cdots, \Psi_ {\text{de}}(\bf{z}_ Q)]$ are the reconstruction by the decoder for the stationary and non-stationary components, respectively.

Q6: Previous papers

A6: Thank you for providing us with references that let us know more about the work of learning koopman operators based on kernel methods. We have already read the two of papers listed. The main idea of these two papers and our work is incorporating the kernel method in the Koopman theory and further learning the Kppman operator in a finite-dimensional measurement space by kernel approximation.

"Estimating Koopman operators with sketching to provably learn large-scale dynamical systems" approximates the kernel using the Nystrom method, which depends very much on the selection of induction points and does not take all the data into account. Compared to the Nystrom method, the random Fourier feature (RFF) can make a "good" approximation of a kernel using all the data. Both "Online Estimation of the Koopman Operator Using Fourier Features" and our paper utilize RFF to approximate the kernel and construct a finite-dimensional measurement space. The main difference between our work and this one is that the learning of Koopman operator is realized through the learning of the kernel network in our work. Concretely, the Koopman operator learning can be formulated as: $\hat{\bf{x}} = \Phi_ {de}(\mathcal{K}g(\bf{x}))$, where $\bf{x}$ and $\hat{\bf{x}}$ are the input and output, respectively. $g$ denotes the kernel mapping with RFF, $\mathcal{K}$ denotes the Koopman operator, and $\Phi_ {de}$ denotes a decoder. We make $\mathcal{K}(\cdot)$ as the linear mapping, and thus $\mathcal{K}g(\bf{x})$ can be considered as a kernel network, stacking the non-linear kernel mapping with linear mapping.

If you have any further questions, we will discuss them in a timely manner. That could help improve our work.