KooNPro: A Variance-Aware Koopman Probabilistic Model Enhanced by Neural Process for Time Series Forecasting

Thank you for your insightful feedback and thought-provoking questions.

[W1] Improvememt of Presentation.

[A] We have modified the first paragraphs of Sections 4.1 and 4.2 in the manuscript to strengthen the connection between the details and the overall framework shown in Fig.1. Additionally, links to Fig.1 are added in the first paragraph of Section 4 to clarify the explanation of the framework. These modifications in the manuscript are now,

In Section 4, "... Initially, NP captures the discrete spectrum of dynamics governing the entire time series which is shown by the downward arrows in Fig.1. Additionally, inspired by the concept of pseudospectra, we utilize the probabilistic deep Koopman model to refine these dynamics, obtaining a variance-aware continuous spectrum for prediction which is demonstrated by the shadowed box in Fig.1"

In Section 4.1, "The proposed model first identifies the distribution of the latent variable in Eq.1 through an embedding
which is presented in the left part of Fig.1. The latent variable integrates the underlying dynamics present in the time series, which allows the model to be more reactive to global features of the time series ..."

In Section 4.2, "The probabilistic deep Koopman model shown in the gray block of Fig.1 concentrates on explaining local characteristics of time series, namely intricate temporal dynamics, with a variance-aware continuous spectrum ..."

[W2] Better analysis of each component of the model.

[A] We substitute Neural Process with Attention Neural Process and Gaussian Process and test their predictive results in Appendix E.

[W3] Performance on non-stationary time series or time series tasks with a much larger prediction length.

[A] The KPSS test (Kwiatkowski–Phillips–Schmidt–Shin test) is a classical statistical method used to determine whether a time series is stationary. We apply the KPSS test to each dataset to assess its stationarity. The results reveal that ETTs are stationary time series, while solar, electricity, traffic, taxi, and cup are non-stationary. The superior performance of KooNPro across these five non-stationary time series demonstrates its effectiveness in handling non-stationary scenarios. It can be found in Appendix G.

We compare each model in the longer prediction length, the details of each dataset, and the results in Appendix F.

[W4] Performance about Robustness.

[A] To evaluate the robustness of KooNPro's predictive performance, we test it under varying Signal-to-Noise Ratio (SNR) conditions (20dB/40dB/60dB). During training, KooNPro is provided with ground truth data. At the testing stage, Gaussian noise is added to the input data, and the predictions are compared against the ground truth. As presented in Tab. 8, the performance degradation with decreasing SNR remains within acceptable limits, demonstrating KooNPro's robust predictive capability across varying noise levels. It can be found in Appendix H.

[W5] Details on training baselines.

[A] We train baselines by open code which is reported in corresponding papers, and follow the default setting. It can be found in Appendix J.

[Q1] Definition of spectral pollution.

[A] We apologize for the lack of clarity in explaining the term “spectral pollution.” In the context of Koopman operators, “spectral pollution” refers to the phenomenon where the computed eigenvalues (or spectra) of an approximation to the Koopman operator contain spurious or extraneous components. These unwanted components can arise due to numerical inaccuracies, poor choice of observables, or insufficient resolution in the discretization of the state space. These spurious eigenvalues can distort the true dynamics of the system and make it harder to extract meaningful insights from the system’s behavior.

In simpler terms, when applying techniques like Dynamic Mode Decomposition (DMD) or other spectral methods to approximate the Koopman operator, we expect to recover the true modes and eigenvalues that describe the underlying dynamics of the system. However, in practice, numerical methods can introduce errors that lead to additional, unphysical modes or eigenvalues that do not correspond to the true dynamics. This ``pollution’’ of the spectrum can complicate the interpretation of the system’s modes and can reduce the accuracy of predictions or the stability of the model.

[Q2] Long-term Prediction.

[A] The choice of context length is informed by the ablation study results, as presented in Figure 3 and detailed in the Appendix ablation study. The study indicates that increasing the context length does not necessarily improve prediction performance, highlighting the importance of selecting an optimal context length for effective modeling.

Thank you for your thoughtful and detailed feedback. Your comments have greatly helped us improve the clarity and depth of our paper.

[W1-W10 & W13] Improvement suggestions.

[A] We have revised the introduction to include more background on probabilistic forecasting and its applications, cited the recommended works, and improved the overall writing quality.

The primary reason for choosing Neural Processes (NPs) as the probabilistic modeling framework is their ability to learn global latent representations through an encoder-decoder structure, which is essential for our variance-aware continuous spectrum modeling. In contrast, diffusion models, while powerful for generating probabilistic forecasts, are not inherently designed to function as encoders that can extract such latent representations.

[W11] Novelty.

[A] The motivation for this work stems from capturing different granularities of temporal dynamics. The dot spectrum provides a method to capture the stable characteristics of a time series. To leverage this, we use NP to learn a dot spectrum embedding that governs the entire series. While when facing unstable temporal dynamics, this method may be collapsed. Thus, we employ an auxiliary neural network to manage unstable dynamics, enabling a more detailed representation of temporal dynamics through the continuous spectrum. However, numerical issues often lead to spectrum pollution in the continuous spectrum. To address this, we adopt a variance-aware approach to learn Pseudospectra, offering improved robustness and performance. To some extent, our approach aligns with [1], which analyzes time series at multiple granularities. However, we focus on treating time series from a dynamics-based perspective.

[W12] Multivariate time series forecasting problem definition.

[A] A multivariate time series forecasting problem predicts future values of multiple variables from past observations. The input is a time series matrix of shape $[T, N]$ ($T$: time steps, $N$: variables), aiming to forecast $H$ future steps while capturing temporal patterns and variable dependencies.

[W13] Comparisons to other baselines.

[A] We compare with DeepAR, and MQ-CNN and show the predictive results in Appendix I. As for the MQ-transformer, it's regret that we haven't found an open code.

[W14] The case study in Dimension 6 and 8.

[A] We acknowledge that KooNPro does not perform optimally in capturing the peaks of certain dimensions, likely due to the large range of peak values (spanning from 100+ to 400+). However, the key takeaway from Figure 4 is KooNPro's ability to learn the temporal dynamics of the data. Specifically, when values transition from the bottom to the peak and reverse from the peak to the bottom, KooNPro demonstrates low error and variance. From a data processing perspective, it is characterized by continuous peaks and troughs. Therefore, we believe that the higher variance observed in both the peaks and troughs is reasonable. We believe that modeling diverse probabilistic behaviors with a single model is challenging. This represents an important direction for exploration, and we plan to enhance KooNPro to address this aspect in future work.

[Q1] Generalize past Gaussian distributions to modeling arbitrary distributions.

[A] NPs utilize Gaussian distributions as their foundational building block. Their capacity to model arbitrary distributions stems from the flexibility of deep neural networks. This flexibility is further enhanced during the training procedure, which can be summarized as follows:

• Individual distributions are Gaussian, while they can represent different means and variances for different inputs, offering adaptability to data.
• The splitting of context and target set, and the learning to aggregate the context adaptively serve as a prior for the prediction at new data points.
• The combination of the latent variable $S$ and the observed data allows the model to capture complex dependencies, effectively mimicking arbitrary distributions.

[Q2 & Q3] Test with ANP and GP.

[A] We substitute Neural Process with Attention Neural Process and Gaussian Process and test their predictive results in Appendix E.

[Q4] Performance of TimeGrid on the electricity dataset.

[A] The properties of the electricity dataset—high dimensionality, short temporal intervals, heterogeneity, and multivariate dependencies—are well-suited to TimeGrad's design. By combining autoregressive RNNs with diffusion probabilistic models. While in Appendix G we test the predictive performance of each model. The result shows in the longer-term prediction case, our model shows better performance than TimeGrid.

Reference

[1] Xinyao Fan, Yueying Wu, Chang Xu, Yuhao Huang, Weiqing Liu, and Jiang Bian. MG-TSD: Multi-Granularity Time Series Diffusion Models with Guided Learning Process, March 2024.

Thank you for your valuable feedback.

[W1 & W2] Model clarity and component motivation.

[A] We have modified the first paragraphs of Sections 4.1 and 4.2 in the manuscript to clarify the advantage of such a design that considering both the local and global dynamics helps achieve better predictions. The lines at the beginning of Sections 4.1 and 4.2 in the modified manuscript are,

"The proposed model first identifies the distribution of the latent variable $S \in\mathbb{R}_s$ in Eq.1 through an embedding $\boldsymbol \tau$ which is presented in the left part of Fig.1. The latent variable integrates the underlying dynamics present in the time series, which allows the model to be more reactive to global features of the time series ..."

"The probabilistic deep Koopman model shown in the gray block of Fig.1 concentrates on explaining local characteristics of time series, namely intricate temporal dynamics, with a variance-aware continuous spectrum ..."

[W3] Comparison with other Koopman-based methods.

[A] We have updated the introduction to explicitly distinguish KooNPro from existing Koopman-based and probabilistic time series models in lines 053 to 061. Specifically, we have detailed the limitations of current methods, such as spectrum pollution and fixed state transition assumptions, and how KooNPro addresses these issues through variance-aware continuous spectrum modeling and integration with Neural Processes.

[W4] Other probabilistic forecasting methods.

[A] Actually, in our experiments, all of our comparison methods are multivariate probabilistic forecasting models, many of which explicitly or implicitly model latent factors in time series. We have thoroughly discussed these methods in the Related Work section and demonstrated through experiments that KooNPro outperforms these probabilistic forecasting models across multiple datasets. We have included a detailed introduction of the comparison methods in Appendix I.

Among the methods, we did not directly compare our method with state-space models, but we emphasized that GP-Copula and TimeGrad can be conceptually related to state-space models (SSMs). GP-Copula can capture dependencies in time series in a way similar to how SSMs use latent states to represent system dynamics. Gaussian Processes (GPs) can be used to approximate latent dynamics, akin to the latent state modeling in SSMs. TimeGrad leverages diffusion processes, which conceptually align with continuous-time state-space models. Diffusion models often involve latent representations of system states evolving over time, similar to SSMs.

Deep State Space Models excel at modeling local latent transitions, while KooNPro uses Neural Processes and Koopman operator for global dynamic representation and fine-grained local dynamics to provide a more comprehensive modeling framework.

[Q1] The utility of the neural process latent variable $S$ at a high level.

[A] As you mentioned, $S$ is indeed an $s$-dimensional latent variable that encodes the entire time series into a subspace, but it doesn't have to be low-dimensional. At a high level, $S$ captures time-invariant features that represent the global characteristics of the time series. $S$ encapsulates the shared structure and underlying patterns across the entire time series, which at a high level, allows it to effectively model the inter-correlations between different time series.

Therefore, $S$ can indeed be thought of as an encoding of the whole time series, not only summarizing individual series but also capturing the relationships and correlations between them implicitly.

[Q2] The advantage of the proposed approach over methods that model time series as state space models.

[A] You are correct that many state space models, such as those cited, involve low-dimensional latent decomposition. Similarly, our approach leverages this concept by extracting global time-invariant features and local time-varying dynamics, with distinct advantages brought by our Koopman-based modeling.

While state space models excel in certain scenarios, their assumptions about linear or fixed dynamics can limit their capacity to handle complex systems. KooNPro's use of Koopman operators for global time-invariant feature extraction, coupled with variance-aware modeling and integration of Neural Processes for local dynamics, offers a more robust and flexible framework for probabilistic time series forecasting.

Methods, such as Deep Factors, Deep State Space Models, and Hierarchically Regularized Forecasting, face limitations in handling nonlinear and non-stationary dynamics due to their reliance on linear assumptions, stationary transitions, or hierarchical priors.

We have added a comparative discussion with these three papers in the introduction section in lines 046-050.

Thank you for these insightful comments.

[W1] Questions about challenges in implementation, and the computational complexity.

[A] With advancements in probabilistic deep neural networks, such as variational autoencoders and diffusion models, the calculation of the Evidence Lower Bound (ELBO) has become both stable and efficient. Moreover, KooNPro integrates lightweight MLP architectures, which reduce computational overhead compared to transformer-based and diffusion-based methods.

[W2] Questions about the design of the encoder and decoder networks.

[A] That's an insightful question. We have also considered employing different encoder-decoder architectures tailored to specific scenarios. For example, CNNs may be more suitable for image-related tasks, transformers for natural language processing, and GNNs for graph-structured data. Adapting the architecture to the nature of the data ensures better performance and efficiency in diverse applications.

[Q1] Questions about the model's ability to capture complex dynamics and the validity of the linearity assumption in formula (6).

[A] The proposed probabilistic deep Koopman model captures complex temporal dynamics via the Koopman operator framework, which allows for a linear representation of nonlinear dynamical systems in a latent space. At its core, the Koopman operator provides a way to track the evolution of observables (functions of the state variables) over time, even in systems with highly nonlinear behavior. In our approach, we extend this framework using deep learning techniques to approximate the Koopman operator in a probabilistic manner. The probabilistic aspect allows the model to capture the inherent uncertainty and variability in the system's dynamics. Rather than assuming a rigid, deterministic evolution, the model learns a distribution over possible future states, which helps capture the stochastic nature of complex temporal dynamics that may arise due to noise, unobserved variables, or inherently uncertain processes.

Additionally, the probabilistic formulation provides a way to quantify the uncertainty in the predictions, which is important when dealing with real-world systems where uncertainty is an inherent characteristic. This not only helps in making more accurate forecasts but also provides a framework for analyzing and understanding the system's temporal behavior under varying conditions. The assumption in formula (6)—that the latent space possesses linear characteristics—is grounded in the principles of Koopman theory, which links nonlinear dynamical systems to linear representations in an augmented (or extended) space of observables. In Koopman theory, the original nonlinear system is mapped to an infinite-dimensional space where the dynamics are represented by a linear operator acting on the space of observables. This mapping allows us to model the evolution of nonlinear systems using linear techniques, despite the underlying system being nonlinear. More specifically, the "latent space" referred to in our formulation can be understood as a discretization of the observable space, where the dynamics of the system are approximated by a set of chosen observables.

Thus, the validity of the assumption is justified by Koopman theory, which provides a theoretical foundation for understanding the behavior of nonlinear systems in a hidden linear space.

Thank you for your feedback. We appreciate your time and efforts in reviewing our manuscript.

We sincerely thank the reviewer for the constructive feedback, which has greatly helped us refine and improve our work.

[W1] Formatting, notation, language, section naming, and figure presentation.

[A] We have addressed the issues related to formatting, notation, language, section naming, and figure presentation in the revised manuscript. Additionally, $𝑒$ refers to "scientific notation," commonly used in programming languages. There is no overlap between the training and test sets.

[W2] Definition of target and context sets and the splitting of $𝑧$.

[A] This method forms the core idea of the Neural Process (NP) introduced by [1], aiming to enhance generalization when handling unseen data. The context can be viewed as a prior that improves generalization. The splitting of $𝑧$ separates the training dataset into context and target sets, a method that is independent of point or distribution estimation. Previous studies, such as [2] and [3], have employed variants of NP for predicting climate change through point estimation.

[Q1] Provide additional metrics, contextualize findings, and compare with probabilistic forecasting approaches.

[A] Utilizing the Monte Carlo method to sample the predictive distribution offers a valuable approach for evaluating predictive performance. To assess predictions comprehensively, we consider the two metrics (CRPS/NRMSE) that strongly relate to MSE/MAE. NRMSE measures the normalized root mean square error between the ground truth and the predicted means. CRPS, a generalization of MAE, quantifies the error between the predictive probability density function (PDF) and a step function at the truth value. In this context, MAE can be viewed as a special case of CRPS, where the PDF is replaced by a step function at the predicted value.

[Q2] Run multiple times.

[A] To ensure performance stability, we conduct multiple runs, averaging results over 10 independent runs. Each table presents the mean and standard deviation of the performance metrics, while Figure.3 includes error bars representing the standard deviation, also estimated from the 10 independent runs.

[Q3] Variance modeling accuracy, outliers in dimensions (#3, #6), overestimated nightly variance in the Solar dataset, and uniform variance concerns.

[A] We acknowledge that KooNPro does not perform optimally in capturing the peaks of certain dimensions, likely due to the large range of peak values (spanning from 100+ to 400+). However, the key takeaway from Figure 4 is KooNPro's ability to learn the temporal dynamics of the data. Specifically, when values transition from the bottom to the peak and reverse from the peak to the bottom, KooNPro demonstrates low error and variance. From data processing perspective, it is characterized by continuous peaks and troughs. Therefore, we believe that the higher variance observed in both the peaks and troughs is reasonable. We believe that modeling diverse probabilistic behaviors with a single model is challenging. This represents an important direction for exploration, and we plan to enhance KooNPro to address this aspect in future work.

[Q4] The number of parameters.

[A] We check the number of parameters of each learned model, and we show the table in Appendix I.

[Q5] Forecast fractional steps.

[A] It is a great idea. We evaluate KooNPro on functional Magnetic Resonance Imaging (fMRI) data with a temporal resolution of 0.72 seconds, where it continues to demonstrate exceptional predictive capability. What's more interesting, the prediction variance provides insights into the fluctuations of different brain regions. For instance, the left and right cortexes exhibit similar properties, while both are markedly distinct from the subcortical regions. The visualization of predicted results and variance of different brain areas is shown in Appendix E.

Reference

[1] MartaGarnelo, DanRosenbaum, Christopher Maddison, TiagoRamalho, David Saxton, Murray Shanahan, YeeWhyeTeh, DaniloRezende, and S.M.AliEslami. ConditionalNeuralProcesses. In Proceedings of the 35th International Conference on Machine Learning, pp.1704–1713.PMLR, July 2018a.

[2] AndrewFoong, WesselBruinsma, JonathanGordon, YannDubois, JamesRequeima, and Richard Turner.Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems, 33:8284–8295,2020.

[3] Wessel P Bruinsma, Stratis Markou, James Requiema, Andrew YK Foong, Tom R Andersson, Anna Vaughan, Anthony Buonomo, J Scott Hosking, and Richard E Turner. Autoregressive conditional neural processes. arXiv preprint arXiv:2303.14468, 2023.