TimeKAN: KAN-based Frequency Decomposition Learning Architecture for Long-term Time Series Forecasting

Many thanks to Reviewer BGvS for providing thorough detailed comments.

Q1: "Section 3.2 appears somewhat disorganized. While the overall logic is clear, the expression could be refined for clarity. Additionally, more mathematical details and background should be provided, which can be included in the appendix."

Thank you for your valuable suggestions. We have reorganized and refined the content as follows: The sequence preprocessing part originally included in Section 3.2 Cascaded Frequency Decomposition has been relocated to a newly created section, Section 3.2 Hierarchical Sequence Preprocessing. We have also revised Section 3.3 Cascaded Frequency Decomposition to provide a clearer explanation of the step-by-step process used to obtain each frequency component in a cascaded manner. Furthermore, we have added detailed mathematical discussions on the Kolmogorov-Arnold Network and the Fourier Transform in Appendix A.

Q2: "If possible, please add more data to Table 5 in Section 4.3. Supplement it with the performance of other methods in Table 1 on parameters (params) and MAC across these six datasets."

Thank you for your valuable suggestion. We have added the performance of the remaining methods from Table 1 in terms of parameters (params) and MACs across these six datasets. The complete comparison of parameters and MACs is shown below (Params|MACs).

As can be seen, except for DLinear, our TimeKAN consistently demonstrates a significant advantage in both parameter count and MACs compared to any other model. DLinear is a model consisting of only a single linear layer, which makes it the most lightweight in terms of parameters and MACs. However, the performance of DLinear already shows a significant gap when compared to state-of-the-art methods. In this case, the advantage in parameters may not be a major consideration for researchers. Therefore, in the main text, we compare the parameters and MACs of the current state-of-the-art methods: TimeKAN, TimeMixer, iTransformer, and PatchTST, and the complete comparison results are provided in Appendix B.2. Hope you can understand this arrangement.

We would like to sincerely thank Reviewer 8Zzz for providing a detailed review and insightful suggestions.

Q1: "Depth in Explanation: The methodology section could offer more detail on complex components like Kolmogorov-Arnold Networks for greater accessibility."

Thanks to your valuable suggestions, we have added mathematical details about how Kolmogorov-Arnold Networks perform forward propagation as well as refined the description of the Chebyshev KAN in Section 3.4, please see the red text in Section 3.4.

Q2: "What if we split the frequency band to more layers (more than 3 for example ). Will it increase performance ?"

In fact, the number of frequency bands is a hyperparameter that we manually set to accommodate the frequency domain characteristics of different datasets. To explore the impact of the number of frequency bands on performance, we set the number of frequency bands to 2, 3, 4, and 5. The effects of different frequency band divisions on performance (MSE|MAE results under 96-to-96 setting) are shown in the table below:

As we can see, in most cases, dividing the frequency bands into 3 or 4 layers yields the best performance. This aligns with our prior intuition: dividing into two bands results in excessive frequency overlap, while dividing into five bands leads to too little information within each band, making it difficult to accurately model the information within that frequency range. We have added the above discussion to Appendix B.5.

Logically, more frequency bands could potentially improve performance by allowing for finer modeling. However, in real-life time series, noise is more prevalent in the high-frequency range. When the number of bands exceeds a certain threshold, some high-frequency bands may contain only useless noise, which harms prediction accuracy.

One solution is to use an uneven frequency division, modeling only those bands with effective content. However, since the frequency distribution varies across different time series datasets, manually dividing the frequency for each specific dataset would be impractical and hinder model generalization. Therefore, we divide the frequency uniformly into multiple bands and limit the number of bands to a reasonable range (e.g., 3-4 bands), ensuring each band contains useful information for prediction while maintaining the model's generalization ability. In this way, we can apply band-specific modules to extract effective frequency data, improving the prediction performance. Our results show that 3 or 4 bands is an optimal threshold, beyond which the impact of noise on prediction is amplified, leading to a decline in performance.

We hope our response addresses your question, and we would be happy to answer any further questions before the rebuttal ends.

Q6: "For Table 2, why not include the electricity dataset?"

Thanks to your valuable reminder, we have added the results of the missing electricity dataset in Table 2, which is shown in the following table:

Q7: "For KAN, you mentioned that the Kolmogorov-Arnold representation theorem states that any multivariate continuous function can be expressed as a combination of univariate functions and addition operations. Could you explain more about how it can capture multivariate correlations?"

When using KAN to perform supervised learning tasks, the correlations among multivariate are implicitly reflected in the learnable univariate activation functions. Each variable in KAN is regulated by a learnable univariate activation function. If multiple variables exhibit consistent behavior, KAN tends to reduce redundancy by allowing only a small number of activation functions to remain effective, while deactivating those corresponding to redundant variables. Through supervised training, KAN learns and captures the implicit correlations among variables and uses the learnable activation functions to regulate their behavior. If explicit direct relationships among variables are needed, unsupervised learning can be employed to find a non-zero function $g$ such that $g(x_{1}, \cdots, x_{n})\approx0$. After training, the explicit relationships between each variable and the remaining variables can be constructed by extracting the expressions of the corresponding learnable activation functions. In this scenario, the univariate refers to one element in the input sequence and does not represent a variate in the multivariate time series. In TimeKAN, KAN is used to model the latent representation of time series, i.e., learning the hidden dimension representation. And we adopt a variate-independent strategy for multivariate processing, also known as the channel-independent strategy in PatchTST.

Q5: "Could you provide a short theoretical analysis about why in some cases in time series forecasting, KAN is better than MLP?"

Time series typically consist of components from multiple frequencies. Suppose we divide the time series into $K$ frequency bands from low to high frequency, where each frequency band $f_k$ contains $N$ frequencies. The complexity $C_{f_k}$ of the $k$-th frequency band $f_k$ can be defined as its spectral entropy, i.e., $C_{f_k}=-\sum_{i=1}^N P(f_i) \log P(f_i)$ where $P(f_i)$ is the normalized power spectrum. In most cases, the time series contains a small number of concentrated high-amplitude frequencies in the low-frequency region, resulting in lower complexity. On the other hand, the distribution in the high-frequency region is relatively uniform, leading to higher complexity. Therefore, the complexity of each frequency band usually satisfies $C_{f_1} < C_{f_2} < \cdots < C_{f_K}$. To perform more accurate time series prediction, we assume that for each frequency band $f_k$, its representation in the latent space is $F_k$. Furthermore, assume there exists an perfect network $M_k$ that perfectly fits $F_k$ in the latent space, such that $M_k(f_k) = F_k.$ Thus, the complexity of the network should be proportional to the complexity of the corresponding frequency band, satisfying $C_{M_1} < C_{M_2} < \cdots < C_{M_K}$. Our goal is to find the network $\hat{M}\_k$ such that the following fitting error is minimized: $(\hat{M}\_k(f_k) - M_k(f_k))$ for each $f_{k}$. To achieve this, we first need to minimize the complexity error betweem $\hat{M}\_k(f_k)$ and $M_k(f_k)$. We define the complexity error for the $k$-th frequency band as $\Delta C_k = C_{\hat{M}\_k} - C_{M_k}$, where $C_{\hat{M}\_k}$ represents the complexity of the fitted network $\hat{M}\_k$. When using MLP as the fitting network, the complexity $C_{\hat{M}\_k}$ is fixed. Since the complexity of the MLP cannot be adjusted, there can be at most one $C_{\hat{M}\_k}$ that satisfies: $\Delta C_k = 0$, because the $C_{M_k}$ is monotonically increasing with $k$. Therefore, when using the fixed-complexity MLP as the fitting network, there will inevitably be an error accumulation, and the total error satisfies $\sum_{k=1}^K \Delta C_k \geq \epsilon$, where $\epsilon$ is a lower bound for the error. In contrast, when using KAN, we can flexibly control the complexity by adjusting the polynomial degree of the KAN's internal kernel. In the optimal case, each frequency band can satisfy $\Delta C_k = 0$, resulting in a total error of zero: $\sum_{k=1}^K \Delta C_k = 0$. This shows that the KAN can achieve a better theoretical performance upper bound when fitting multiple frequency components in the latent space. Furthermore, by accurately fitting the frequency representations, KAN can provide more reliable foundations for precise time series prediction.

Q3: "Lack of comparison between TimeKAN and other methods regarding look-back windows."

Thank you for your valuable suggestion. We have added the performance of TimeMixer, iTransformer and PatchTST with varing look back windows for comparison in Figure 2. You can also see the table ( MSE results of 96 step forecasting horizon in Ettm2 dataset) below.

As we have seen, TimeKAN can clearly benefit from longer look back windows compared to other models and consistently performs optimally in most cases.

Q4: "For baseline methods, it is better to choose more frequency-based (such as FreTS) methods since frequency decomposition is a key contribution."

Thanks to your valuable suggestions, we have added two representative frequency methods FiLM and FreTS in Table 1. A comparison between TimeKAN and these two frequency methods is shown in the following table (Average MSE|MAE of all forecasting horizons):

It can be seen that TimeKAN significantly outperforms FiLM and FreTS on most of the datasets, which shows the effectiveness of our designed frequency Decomposition-Learning-Mixing architecture.

Many thanks to Reviewer gWX6 for providing thorough detailed comments.

Q1: "The author does not explain why TimeKAN does not perform well in the Electricity dataset. It would be helpful if the authors could provide potential reasons or hypotheses for why TimeKAN underperforms on the Electricity dataset specifically."

Our TimeKAN is a model based on frequency analysis. We infer that its poor performance on the electricity dataset is due to the overly short look-back window ($T=96$), which cannot provide sufficient frequency information. To verify this, we compare the average number of effective frequency components under a specific look-back window. Specifically, we randomly select a sequence of length $T$ from the electricity dataset and transform it into the frequency domain using FFT. We define effective frequencies as those with amplitudes greater than 0.1 times the maximum amplitude. Then, we take the average number of effective frequencies obtained across all variables to reflect the amount of effective frequency information provided by the sequence. When $T=96$ (the setting in the paper), the average number of effective frequencies is 10.69. When we extend the sequence length to 512, the average number of effective frequencies becomes 19.74. Therefore, the effective frequency information provided by 512 time steps is nearly twice that of 96 time steps. This indicates that $T=96$ loses a substantial amount of effective information. To validate whether using $T=512$ allows us to leverage more frequency information, we extend the look-back window of TimeKAN to 512 on the electricity dataset and compare it with the state-of-the-art methods, TimeMixer and MOMENT. The results are shown in the table below (results of MSE|MAE):

Although TimeKAN performs significantly worse than TimeMixer when $T=96$, it achieves the best performance on the electricity dataset when the look-back window is extended to 512. This also demonstrates that TimeKAN can benefit significantly from richer frequency information. For a fair comparison with other methods, we have adopted the common 96-step lookback window setting, which actually compromises the advantages of TimeKAN in long sequence. We have added the above discussion to Appendix B.4.

Q2: "From Table 4, I do not see a huge increase with KAN compared to MLP models. Generally, if these results are similar, mostly, KAN is much slower than MLP. It would be good to see runtime comparisons between KAN and NLP implementations. Additionally, if KAN is slower than MLP in practice, it would be beneficial for authors to discuss more reasons why we prefer KAN over MLP."

One significant reason why the improvement of KAN over MLP is not particularly pronounced lies in the well-designed nature of our Frequency Decomposition-Learning-Mixing architecture, which enables even MLP to demonstrate competitive performance. In TimeKAN, we use Chebyshev polynomials as the basis functions for the learnable activation functions, denoted as ChebyshevKAN. Assuming each layer of the neural network contains $N$ neurons, the computational cost per layer for ChebyshevKAN with a maximum degree of $K$ is $N^2K$, while for an MLP, the computational cost is $N^2$. Therefore, the computational cost of ChebyshevKAN is $K$ times that of an MLP. Notably, in the comparison presented in Table 4 of our paper, we ensure that the computational costs of the MLP and ChebyshevKAN are similar. Under comparable computational costs, we observe that the inference speeds (iter/s) of the two methods are not significantly different, as shown in the table below:

Therefore, under similar computation consumption, KAN achieves relatively better performance, demonstrating its capability in dynamic frequency representation. With the emergence of more KAN variants, we can anticipate further improvements in the computational efficiency and forecasting performance of TimeKAN.

Q4-part3: "The absence of data on inference speed and computational cost undermines its practical applicability."

In fact, the computational cost of TimeKAN has been presented in Table 5 using the MACs metric. Compared with the three other best-performing models (including MLP-based and Transformer-based methods), TimeKAN requires significantly less computational cost. For inference time (s/iter), we have added a comparison of TimeKAN and other models under the same settings. Due to the computational cost of multiple Fourier Transforms, TimeKAN’s inference speed is relatively slower but remains in the same order of magnitude as other methods except time series foundation model Time-FFM, and significantly outperforms Time-FFM.

Q5-part1: "Compared to one-fit-all solutions like LLM-based time series methods or time series foundation methods, approaches focusing solely on improving time series forecasting may have limited value. Besides, TimeKAN may not be able to compete with established frameworks like Transformers in terms of generalization and adaptability."

Our TimeKAN is carefully designed for time series forecasting. As shown in the table above, TimeKAN significantly outperforms the current state-of-the-art LLM-based time series methods Time-FFM and time series foundation models MOMENT. Furthermore, as recent studies indicate: LLM-based time series forecasters perform the same or worse than basic LLM-free ablations, yet require orders of magnitude more compute [3]. This suggests that the effectiveness of current LLMs for the time series forecasting task remains questionable. Besides, the state-of-the-art time series foundation model MOMENT requires pretraining on over 13 million time series datasets, which consumes approximately 404 GPU hours and has more than 20000 times the number of parameters as TimeKAN. However, it still fails to outperform TimeKAN on typical time series forecasting tasks. Therefore, whether achieving superior performance in time series forecasting requires building large models following trends in other fields or designing lightweight, task-specific models remains an open question.

Based on the current evidence, large models trained on massive datasets have not surpassed our lightweight TimeKAN in time series forecasting tasks. While such models perform well in generalizing across multiple tasks, they are yet to achieve the best results on single forecasting task. In conclusion, we argue that the future of time series forecasting should not be confined to scaling up models. Designing high-performance, lightweight models that incorporate domain knowledge for single tasks presents a promising alternative pathway.

Q5-part2: "The improvements reported in the experimental results are relatively modest, and without statistical significance testing, it remains unclear if these gains are truly meaningful or could simply be attributed to random variation. "

To evaluate the robustness of TimeKAN, we repeated the experiments on three randomly selected seeds and compared it with the second-best model (TimeMixer). We report the mean and standard deviation of the results across the three experiments, as well as the confidence level of TimeKAN's superiority over TimeMixer. The results are averaged over four prediction horizons (96, 192, 336, and 720).

As shown in the table, in most cases, we have over 90% confidence that TimeKAN outperforms the second-best model and demonstrates good robustne of TimeKAN.

[3] Tan et al., Are Language Models Actually Useful for Time Series Forecasting?, NeurIPS 2024.

Q3: "The foundation of TimeKAN is KAN model, which has limited validation and uncertain theoretical correctness. If KAN is found to have flaws, it could undermine overall reliability of TimeKAN."

In fact, the foundation of TimeKAN is Decomposition-Learning Mixing architecture, which decompose time series into multiple frequency bands represented in the time domain and leverage time-domain processing techniques to learn representations for specific frequencies. We have demonstrated the effectiveness of this framework, showing that even when KAN is replaced with a traditional MLP, competitive performance is still maintained. To better represent various frequency components in a more fitting manner, we further developed the Multi-order KAN to enhance the Decomposition-Learning Mixing architecture. In addition, based on the Decomposition-Learning Mixing architecture, the performance of the well-designed Multi-order KAN surpasses that of the direct use of KAN, indicating that a carefully designed Multi-order KAN can more accurately model different frequency components. As better KAN variants emerge in the future, our framework can benefit further. Therefore, from a practical perspective, our current efforts have shown that the design of the Multi-order KAN is an effective component of our framework.

From a theoretical standpoint, although KAN is still a relatively new member of the deep learning family, the Kolmogorov-Arnold Network theorem it relies on has been established for decades. The Kolmogorov-Arnold Network theorem can be described as: any multivariate continuous function can be represented as a combination of univariate functions and addition operations. In our Multi-order KAN, we primarily utilize univariate functions of varying complexities to form networks with different fitting capabilities, thus representing different frequency components, which is essentially the core concept of the Kolmogorov-Arnold Network theorem. Therefore, we believe that the use of Multi-order KAN can be fundamentally attributed to the well-established Kolmogorov-Arnold Network theorem, making the design of our TimeKAN consistent with foundational theory.

Q4-part1: "Lack of in-depth analysis of the actual computational efficiency between TimeKAN and mainstream time series models."

Thank you for your valuable feedback. We have added an analysis of the computational complexity of TimeKAN in Appendix B.1. Given a multivariate time series with $M$ variables and $L$ time steps, the computational complexity of TimeKAN is approximately $\mathcal{O}(ML\ logL)$. This indicates that the complexity scales linearly with the number of variables and grows logarithmically with the number of time steps. The term $L\ logL$ arises from the computational complexity of the Fast Fourier Transform. To further reduce the computational complexity with respect to the number of time steps, we can employ a pre-selected set of Fourier bases for quick implementation, reducing the complexity to $\mathcal(ML)$. In comparison, the computational complexity of PatchTST is $\mathcal(L^{2})$, iTransformer has a complexity of $\mathcal(M^{2})$, and TimeMixer exhibits a complexity of $ML^{2}$.Therefore, theoretically, TimeKAN achieves a better balance between the number of variables and input length when compared to these state-of-the-art methods. Notably, the actual computational speed may vary due to differences in parameter settings. For instance, while the computational complexity of PatchTST scales quadratically with input length, its computation speed can be significantly improved by selecting larger patch sizes.

Q4-part2: "There is no quantification of the computational cost associated with KAN’s multi-order polynomial calculations when handling long-sequence data. "

In TimeKAN, the role of the Multi-order KAN is to learn representations at different frequencies, i.e., learning the latent space representation. For a single KAN, assuming the highest degree of the Chebyshev polynomial is $k$ and the hidden dimension is $D$, the computational cost for channel modeling is $M\*L\*K\*D^{2}$, which increases linearly with the sequence length $L$. Therefore, it does not result in excessive computational costs for long sequences.

Q2: "Lack of comparison between TimeKAN and current state-of-the-art LLM-based time series models as well as time series foundation models in terms of predictive accuracy or computational efficiency."

Thanks for your valuable suggestion. Through comparisons, our TimeKAN outperforms the state-of-the-art LLM-based time series model and time series foundation model in the majority of cases across six datasets. We have added a state-of-the-art LLM-based time series model TimeFFM [1], which have the same look-back window ($T=96$) setting as TimeKAN. Additional comparative results are shown below:

Additionally, we also compare our TimeKAN with the state-of-the-art time series foundation model MOMENT [2], which is pretrained on time series dataset of size 13M. Notably, the look back window of MOMENT is set to 512. Therefore, we extend the look window of TimeKAN to 512 to ensure a fair comparison. The result of the comparison is as follows:

As shown in the two tables above, our TimeKAN outperforms the state-of-the-art LLM for time series models and time series foundation models in the majority of cases across six datasets, demonstrating its effectiveness in time series forecasting. Furthermore, Time-FFM has 90M parameters, and MOMENT has 385M parameters, while TimeKAN has an average of only 21K parameters. Therefore, TimeKAN is a simple yet effective model, achieving superior performance with just 0.025% (or even less) of the parameters required by these foundational models.

[1] Liu et al., Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting., NeurIPS 2024.

[2]Goswami et al., MOMENT: A Family of Open Time-series Foundation Models., ICML 2024.

We would like to sincerely thank Reviewer XpEF for providing the valuable feedback.

Q1: "This work appears more like a combination of existing technologies, such as frequency decomposition and mixing, rather than a genuinely innovative study."

In fact, the core contribution of TimeKAN lies in re-examining the time series forecasting task from the perspective of multi-frequency decomposition learning. To the best of our knowledge, TimeKAN is the first method to decompose time series into multiple frequency bands represented in the time domain and leverage time-domain processing techniques to learn representations for specific frequencies, which effectively handle the challenges arising from complex frequency mixing. As a result, TimeKAN fundamentally differs from prior methods in frequency domain learning (e.g., FreTS, FEDformer) and decomposition-based learning (e.g., TimeMixer, MICN) in the following ways:

(1) Frequency-Domain Learning: FreTS, FEDformer, and other frequency-domain based methods focus on learning the representation of frequency signals directly in the frequency domain. However, current evidence suggests that methods learning directly in the frequency domain generally perform worse than those learning directly in the time domain. In contrast, we propose using Frequency Upsampling to cascade different frequency components’ representations in the time domain, which allows TimeKAN to directly process different frequency components in the time domain, avoiding the difficulty of learning frequency domain signals directly. To our knowledge, no previous work has adopted this paradigm.

(2) Decomposition-Based Learning: Decomposition-based methods like TimeMixer and MICN mostly decompose sequences into subcomponents from a time-domain perspective, while we decompose time series from a frequency-domain perspective. Frequency domain perspective enables a more comprehensive analysis of time series, encompassing both periodic long-term variations and sharp short-term fluctuations. By extracting the sequence from the frequency domain and learning in the time domain, our TimeKAN is able to accurately model the subcomponents of the time series.

Furthermore, the incorporation of the Multi-order KAN enhances the framework’s capability to adaptively learn specific frequency components. TimeKAN has been demonstrated state-of-the-art performance across multiple long-term time series forecasting tasks and showcasing its lightweight and parameter-efficient architecture.

Dear Reviewer XpEF, 

We would like to express our sincere gratitude once again for your valuable feedback. We fully understand your concerns about the effectiveness of KAN as a novel neural network, but with less than 24 hours remaining until the rebuttal period concludes, we would like to kindly remind you that the vanilla KAN has already received all positive evaluations from five reviewers on OpenReview, and its theoretical contributions have been acknowledged by the community. This demonstrates the potential of KAN for widespread adoption in the deep learning community. In TimeKAN, we have carefully designed the Multi-order KAN to align the variable characteristics of KAN with the multi-frequency nature of time series, and demonstrated its effectiveness through experiments. Therefore, we sincerely hope that you might reconsider your score. Finally, we would like to reiterate our appreciation for your thoughtful comments, which have significantly improved the quality of our paper. 

Best regards, 

The Authors