FLDmamba: Integrating Fourier and Laplace Transform Decomposition with Mamba for Enhanced Time Series Prediction

W2. The experimental comparison focuses primarily on Transformer-based models and Mamba-based methods. Inclusion of more diverse SSM-based baselines, such as those based on S4 or other recent advances, would strengthen the evaluation.

Thanks for your point. We have conducted new Mamba-based methods including SST and Bi-Mamba+. Results are shown in the following table:

W1. While the paper explains the intuition behind using the Laplace transform to capture transient dynamics, it lacks a deeper theoretical exploration of how exactly the inverse Laplace transform contributes to performance improvements in the context of the model.

Q1. Can you provide more details on how the inverse Laplace transform is computed in practice within your framework? Given that inverse Laplace transforms can be numerically challenging, how do you ensure stability and efficiency in this component?

Thanks for your comment. We provide theoretical explanation in Appendix 6.2, which we summarize here. Transient dynamics are characterized by exponential decaying amplitudes w.r.t. time $t$. Thus, a time series variable $u(t)$ exhibiting transient dynamics can then in general be decomposed by

$u(t)=\sum_{n=1}^M A_n e^{-\xi_n t}\cos(\omega_n t + \varphi_n)$ (Eq. 20)

where $\xi_n, n=1,2,...$ are the decaying rates, $\omega_n, n=1,2,...$ are the corresponding periodic frequencies (can be 0 for non-periodic signal), $A_n, \varphi_n, n=1,2,...$ are the amplitudes and phases, respectively.

When we are performing prediction on time series, we are essentially learning an operator (mapping between functions) that maps a segment of time series $u(t), t\in[t_0,t_1]$ in the past, to a segment of time series $u(t), t\in[t_1,t_2]$ in the future. Thus, the above $A_n$, $\lambda_n$, and $\omega_n$ are in general functions of the past time series $u(t), t\in[t_0,t_1]$. Below, we show how our modeling of inverse Laplace transform exactly captures the above transient dynamics (Eq. 20).

As was explained in Appendix 6.2 in the original submission, we model the operator which maps an input function $v(t)$ to an output function $u(t)$ as

$u(t)=(\kappa(\phi)*v)(t)=\int_D \kappa_\phi(t-\tau)v(\tau)d\tau$

Performing Laplace transform on both sides, we have

$U(s)=K_\phi(s)V(s)$

where $K_\phi(s)=\mathcal{L}\{\kappa_\phi(t)\}$ and $V(s)=\mathcal{L}\{v(t)\}$, $U(s)=\mathcal{L}\{u(t)\}$. Based on the Residue Theorem in complex analysis, the poles (singularities) in the complex plane determines its behavior in the original space. Therefore, we assume $K_\phi(s)=\sum_{n=1}^N \frac{\beta_n}{s-\mu_n}$ in the Laplace space, where $\beta_n\in \mathbb{R}$ and $\mu_n\in \mathbb{C}$ are learnable parameters. Also, performing Fourier series expansion on $v(t)$, we have $v(t)=\sum_{l=-\infty}^{\infty}\alpha_l \exp{i \omega_l t}$, which results in $V(s)=\sum_{l=-\infty}^{\infty}\frac{\alpha_l}{s-i\omega_l}$. Mapping back into the original space, we have

$u(t)=\sum_{n=1}^N\gamma_n \exp(\mu_n t)+ \sum_{l=-\infty}^{\infty}\lambda_l \exp(i\omega_l t)$ (Eq. 19 in Appendix 6.2)

Here $\gamma_n$, $\lambda_l$ are derived parameters from $\beta_n$, $\mu_n$, $\omega_l$ and $\alpha_l$, the former two depending on the kernel $\kappa(\phi)$, and the latter two depending on $v(t)$.

Note that $\mu_n\in\mathbb{C}$ is a complex number, whose real part and imagery part represent decaying and periodic behaviors, respectively. If we truncate the number of Fourier series terms $l$, the above Eq. (19) reduces to

$u(t)=\sum_{n=1}^M A_n e^{-\sigma_n t}\cos(w_n t + \varphi_n)$ (Eq. 21)

In our work, we directly parameterize the above $A_n$, $\sigma_n$, $w_n$, and $\varphi_n$ as learnable functions of the output of the previous layer, which in turn are functions of the history time series. We see that this equation (Eq. 21) exactly matches the above Eq. 20 which characterizes transient dynamics. Therefore, our parameterization of the inverse Laplace transform via Eq. 21 can learn transient dynamics accurately. Furthermore, in contrast to performing inverse Laplace transform which involves integration in the complex plane where the integrand has poles, we see that our parameterization in Eq. 21 has better efficiency and stability.

We have updated Appendix 6.2 to include the above analysis.

(continued)

The results indicate that our method outperforms other Mamba-based baselines. This improvement is attributed to the incorporation of FFT and ILT, which effectively capture multi-scale periodicity and transient dynamics. Results are also added in Table 1 in revised version. Please refer to the revised version.

W1. Incomplete Justification for RBF Kernel: Although the RBF kernel is presented as an effective data-smoothing technique, its choice is not empirically validated. A comparison with other kernel functions or a focused ablation study would help verify this choice and ensure that RBF is the optimal choice.

Thanks for your comment. To address your concern, we have conducted experiments by replacing the RBF kernel with Laplacian and Sigmoid kernels. And the results are shown in the following table:

From above results, we observe that RBF kernel achieves the best performance on time series prediction than Laplacian and Sigmoid kernels. This can be attributed to the inherent ability of the RBF kernel to capture the nonlinear and complex patterns in the time series data more effectively.

W2. Unclear Necessity of FFT-IFFT Sequence: The FMamba block employs an FFT followed by an IFFT without a clear explanation of any specific frequency-domain manipulations before reconstructing the signal in the time domain. If this process is meant to filter specific frequencies or reduce noise, the details of such operations should be specified. Otherwise, the sequence could appear redundant, as it may be feasible for the neural network to approximate frequency characteristics without explicitly embedding FFT.

Thanks for your comment. To address the concern regarding the necessity of the FFT-IFFT sequence in the FMamba block, it is important to clarify the intended purpose of this process. The sequence of applying the FFT followed by the IFFT is designed to facilitate specific frequency-domain manipulations that are crucial for enhancing the model's performance. (1) The FFT allows us to analyze the frequency components of an input signal by transforming it from the time domain to the frequency domain using $\Delta' = \text{FFT}(\Delta)$. We then apply $\Delta_F = \text{IFFT}(\tilde{W} \cdot \Delta')$ to return to the time domain. Here $\tilde{W}$ is the Fourier transform of the kernel $\tilde{K}$. In this paper, we treat $\tilde{W}$ as a learnable parameter matrix. This process helps us identify and filter out specific frequencies that may introduce noise, ultimately enhancing the signal and improving forecasting accuracy [1]. (2) The FFT-IFFT sequence can be effectively employed to isolate and mitigate unwanted frequency components. This process ensures that the reconstructed signal retains the essential features while minimizing the influence of noise, leading to more robust predictions. (3) While neural networks can learn to approximate frequency characteristics, explicitly embedding the FFT provides a structured approach to feature extraction. This allows for a more interpretable representation of the data, highlighting important frequency components that may not be as easily captured through learning alone. (4) The FFT-IFFT sequence serves as a complementary mechanism that enhances the neural network's ability to learn complex temporal patterns. By combining explicit frequency analysis with the neural network's learning capabilities, we create a more powerful model that effectively captures both global and local dynamics in the data.

The FFT-IFFT sequence is a deliberate choice aimed at enabling specific frequency-domain manipulations, such as filtering and noise reduction, which ultimately enhance the model's performance. We will clarify these points in the revised version to ensure a better understanding of its necessity.

[1] Li, Zongyi, et al. "Fourier neural operator for parametric partial differential equations."

W1. My biggest concern about this paper is their evaluation metric. I believe using R2 score or Pearson correlation is more suitable for the task. However, this paper only considers the MSE and MAE error, while the MSE and MAE seems to be lower than all other baselines, I still have some doubts on the models ability to capture informative time series patterns.

Thanks for your comment. We performed calculations of the Pearson correlation coefficient and included the results for several update-to-date baselines (S-Mamba and iTransformer) alongside ours on all datasets due to time constraints, shown in the following table. We also add it in Section 6.6 in the revised manucript. Please refer to the revised manuscript for details.

From the above results, we observe that measured by Pearson correlation coefficient, our method outperforms other baselines in most cases on all datasets. This again verifies that the better performance of our method.

Dear Authors,

Thank you for your response. While I believe the improvement in Pearson correlation is relatively marginal, I still think the proposed model performs better overall compared to other baselines. However, I noticed that one baseline [1] appears to be missing. Specifically, the zero-shot performance of Moirai seems to outperform the proposed method on several datasets. Considering that Moirai functions as a universal forecaster with competitive results, what distinct advantages does the proposed method offer over Moirai?

[1] Unified Training of Universal Time Series Forecasting Transformers (ICML)

W1. This paper claims that FLDmamba theoretically achieves faster inference than Transformer-based models, which could be partially demonstrated by the experiments on training time. However, there is no experiment to directly validate this claim.

Thank you for your comments. We have conducted experiments to evaluate the computational overhead during training, which is presented in Figure 12 of the Appendix. Additionally, we assessed the inference times for various models, including Vanilla Mamba, Vanilla Mamba + FFT, Vanilla Mamba + Inverse Laplace Transform (ILT), our method, S-Mamba, iTransformer, AutoFormer, and Rlinear, using a lookback length of 96 on the Electricity dataset. Results are shown in the following table:

The results show that our methods maintain comparable computational overhead to the others while achieving the best performance. We also added it in the revised version in Appendix 6.11.

W2. The discussions in ablation study are thorough, but the conclusion is a little confusing and inconsistent with the experimental results.

Thanks for your comment. We have revised the conclusion section to make it more clear and consistent with the experimental results. Please refer to the revised manuscript.

Questions:

Q1. Does other transforms in frequency domain provide similar benefits as the Fourier and Laplace Transform? Could you provide some insights into this?

Thanks for your question. Other transforms in the frequency domain may offer similar benefits as the Fourier and Laplace Transforms, each with its own advantages and applications. Here are some insights into this:

(1) Wavelet Transform: The Wavelet Transform is known for its ability to represent signals in both time and frequency domains simultaneously. This feature makes it particularly useful for analyzing signals with non-stationary and transient characteristics, such as in time series data with varying trends and periodicities.

(2) Short-Time Fourier Transform (STFT): The STFT divides a signal into shorter segments and performs a Fourier Transform on each segment. This allows for the analysis of how the frequency content of a signal changes over time, making it useful for capturing time-localized frequency information.

(3) Discrete Cosine Transform (DCT): The DCT is commonly used in data compression and image processing. In signal processing, it is known for its energy compaction properties, which make it efficient for representing signals in a smaller number of coefficients while retaining important frequency information.

(4) Z-Transform: While typically used in the context of discrete-time signals and systems, the Z-Transform can also be applied to analyze the frequency content of signals in the complex plane. It is useful for studying system dynamics and stability in the frequency domain.

Each of these transforms has its own strengths and weaknesses, and the choice of transform depends on the specific characteristics of the signal being analyzed and the objectives of the analysis. Experimenting with different transforms and understanding their properties can help in selecting the most suitable transform for a given signal processing task.

W1. The authors' characterization of multi-scale periodicity, transient dynamics, and data noise as challenges specific to the Mamba-based model is inappropriate. These three challenges are faced by all models, not just those based on Mamba. Furthermore, among the proposed improvements to address these challenges, only the FLDMAMBA module appears to be model-specific; the others seem to be model-agnostic. The paper lacks experiments demonstrating the integration of these strategies into other methods. Additionally, it is unclear whether the authors are making improvements to the Mamba architecture or proposing a collection of strategies to address these three challenges.

Thanks for your comments. We have uploaded the revised manucript. We appreciate the reviewer’s feedback regarding our characterization of multi-scale periodicity, transient dynamics, and data noise. All models face these challenges but few of them can catpure the effective features to solve it. In this paper, our intention was to highlight how these challenges particularly impact the performance of the promising Mamba, given its novel architectural design and powerful performances. To address your concern, we also combine RBF and Inverse Laplace Transform (ILT) with classifical Transformer architecture such as Autoformer on datasets like Etth1 and Etth2. Results are shown in the following table:

From results, we find that combination RBF and ILT with other method like Autoformer do not bring positive impacts on performance. The reason can be attributed to the redudant attention mechanism which can not show its superiority on the frequency domain. We have incorporated the above table in section 6.10 in Appendix in the revised manuscript. Please refer to the revised version.

Besides, we have conduced experiments of case study about illustration of adddressing challenges of multi-scale periodicity, transient dynamics, shown in Figure 6, Figure 7 and Figure 13, shown in the revised version. Compared our method with S-Mamba, this verifies that our method can capture multi-scale periodicity and transient dynamics. Meanwhile, we also conducted experiments on addressing the challenge of data noise, shown in Figure 4. Compared ours with S-Mamba and iTransformer, we find that our method has robust performance than that of S-Mamba and iTransformer.

W2. The authors need to provide details on computational overhead. One motivation for introducing Mamba is its lower time complexity compared to Transformer models. However, on one hand, the authors employ parallel FMamba and Mamba modules, which significantly increase the model's parameters and computational overhead. On the other hand, it is uncertain whether FFT and IFFT will become computational bottlenecks, especially for datasets with a high number of channels, such as "electricity," which has 321 channels.

Thanks for your comments. We have conducted experiments on computational overhead during training period for each epoch, which is shown in Figure 12 in Appendix 6.7 in the original submission. Meanwhile, we have also conducted experiments during inference period for vanilla Mamba, Vanilla Mamba+ FFT, Vanilla Mamba+ Inverse Laplace transform (ILT), Ous, S-Mamba, iTransformer, AutoFormer and Rlinear on each batch when the lookback length is set as 96 on Electricity, as shown in the following table. We see that our method has comparable computational overhead w.r.t. other baselines with the best performance.

We also added this table in Appendix 6.11 in the revised manuscript.

Issues like multi-scale periodicity, transient dynamics, and data noise are common. Why did the authors specifically focus on the Mamba structure? Is your proposed method only applicable to Mamba? Evaluating broader effectiveness would improve the paper's quality.

Regarding ablation studies: Please conduct thorough ablation experiments with dual/multiple modules. I suggest using tables rather than figures to thoroughly clarify the main sources of the method's performance. Since your improvement over Mamba is minimal, the proposed methods appear to be merely tricks.

Less importantly, we encourage the authors to experiment with datasets having larger numbers of channels to validate efficiency.

Q1. Issues like multi-scale periodicity, transient dynamics, and data noise are common. Why did the authors specifically focus on the Mamba structure? Is your proposed method only applicable to Mamba? Evaluating broader effectiveness would improve the paper's quality.

Q2. Regarding ablation studies: Please conduct thorough ablation experiments with dual/multiple modules. I suggest using tables rather than figures to thoroughly clarify the main sources of the method's performance. Since your improvement over Mamba is minimal, the proposed methods appear to be merely tricks.

Thanks a lot for your further comments. Firstly, we would like to clarify the scope of our method. Our method, FLDmamba, consists of both Mamba as its main architecture, and seamlessly integrating Fourier and Laplace Analysis, etc. Both aspects are indepensable aspects of the full architecture. Mamba offers computational efficiency and long-term prediction capability through its state-space architecture, while the Fourier and Laplace transformations overcome the shortcomings of Mamba and enhance its predictive capabilities by specifically targeting the challenges of multi-scale periodicity, transient dynamics, and data noise. This strategic integration of frequency and Laplace analysis within the Mamba structure enables FLDmamba to better handle these complex aspects of time series data, thereby improving its performance in long-term predictions.

Through ablation study, we have validated the effectiveness of our method, demonstrating that each component is indispensable. For the Mamba component, we show in Fig. 5 and Fig. 11 that vanilla Mamba architecture offers better long-term prediction capability. Fig. 12 and Table 8 demonstrate that Mamba-based methods offers significant speedup compared to transformer-based architectures like AutoFormer. Furthermore, we provide tables below showing quantitive results of the ablation study for each other component. For w/o FT: This variant excludes the Fourier transform for the parameter $\Delta$, allowing us to assess the impact of frequency domain analysis. w/o FM: This variant removes the FLDmamba component, leaving only the Mamba architecture, enabling us to evaluate the contribution of the frequency-domain enhanced Mamba. w/o Ma: This variant eliminates the Mamba component, retaining only FLDmamba, allowing us to assess the impact of the frequency-domain modeling. w/o RBF: This variant omits the Radial Basis Function (RBF) kernel, enabling us to evaluate the impact of data smoothing on performance. w/o ILT'': This variant disregards the inverse Laplace transform, allowing us to assess the impact of the time-domain conversion.

From results, we observe that each component make positive contributions to the accuracy prediction performance. By conducting these experiments and evaluating the performance of each model and its components on both datasets, the study aimed to demonstrate the positive impact of each component on enhancing time series prediction performance. The results provided insights into how the decomposition of Fourier Transform, Inverse Laplace Transform, and other components with mamba can improve the mamba-based model's ability to handle challenges such as multi-scale periodicity, transient dynamics, and data noise in time series data, ultimately leading to more accurate and robust predictions. We also revised the paper and added the above table in Section 6.14 in Appendix. Please refer to the revised version.

While the proposed method is tailored to work with the Mamba structure, we conducted experiments on 9 datasets and our method has shown the best performance in most of cases, which shows that the effectiveness of our method.

We appreciate your bringing this point on evaluating its broader effectiveness beyond Mamba could indeed enhance the paper's quality. Future work could involve assessing the adaptability of the FLDmamba framework to other state-of-the-art time series modeling architectures to demonstrate its versatility and effectiveness across different models and datasets. Such an evaluation could further validate the generalizability and robustness of the proposed approach in addressing the challenges of multi-scale periodicity, transient dynamics, and data noise in time series prediction tasks.

We have uploaded the revised manuscript. Please refer to the revised version.

Thanks for your comments. We have incorporated the revised parts into our revised manuscript. Please feel free to continue the discussion if anything remains unclear.

W1. The improvement proposed in the paper are largely orthogonal to Mamba algorithm, which makes the story less coherent. For example, I think RBF kernel and inverse Laplace transformation are mostly agnostic of the model struce, and can be applied to other forecasting model such as MLP or transformer.

Thanks for your comment. We have conducted experiments on RBF and ILT. And results are shown as following:

The results indicate that the combination of RBF and ILT with other methods, such as Autoformer, does not yield positive improvements in performance. This can be attributed to the redundant attention mechanism, which fails to demonstrate its advantages in the frequency domain. We have incorporated the above table in section 6.10 in Appendix in the revised manuscript.

Q1. Page 6 line 270 says $\tilde{W}$ denotes the Fourier transform of the kernel $\tilde{\mathcal{K}}$, but I don't see where the kernel $\tilde{\mathcal{K}}$ is defined in the paper. Then in Algorithm 2, there is $\Delta'=FFT(\Delta), \Delta_F=IFFT(\Delta')$. Doesn't this implies $\Delta=\Delta_F$, and therefore nothing is done?

Thanks for your comments. In the revised manuscript, We have added the definition of the kernel $\tilde{\mathcal{K}}$ in Definition 1 in section 3.1.2 and also improved the writing to make it more clear.

As for the $\Delta_F$, as indicated in the Eq. (4) in the original submission, it is calculated as $\Delta_F=IFFT(\tilde{W}\cdot \Delta')$, where $\tilde{W}$ is the Fourier Transform of the kernel $\tilde{\mathcal{K}}$, and $\Delta'=FFT(\Delta)$. Therefore, $\Delta_F$ is different from $\Delta$ due to the filtering effect of $\tilde{W}$. In the Algorithm 2 in the original submission, we have a typo and missed the $\tilde{W}$, and we have corrected it in the revised manuscript.