CMamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting

We sincerely thank Reviewer Wmo3 for providing thorough and insightful comments. Here are our responses to your concerns and questions.

• W1: Lack of experiments under very long-term horizon (longer than 720).

Thanks for your insightful suggestions. We follow the common setting of long-term multivariate time series forecasting, i.e., predicting the future for 96, 192, 336, and 720 steps, so we do not consider other prediction lengths in our paper. However, it is important and meaningful to make ultra-long-term forecasts. Hence, we add experiments under 960 future steps in the following table, where CMamba ranks top 1 in 10 out of 14 settings:

The above results demonstrate the powerful ultra-long time series prediction capability of CMamba, which further proves the effectiveness of our modeling of cross-time dependencies and cross-channel dependencies.

• W2 & Q1: Generalization of CMamba in other fields (like financial or biological datasets).

Thanks for your valuable suggestions. We have added experiments on the Exchange (the daily exchange rates of eight different countries ranging from 1990 to 2016) and ILI (the weekly recorded influenza-like illness (ILI) patients data in the United States between 2002 and 2021) datasets. The look-back length is 96 and the horizons are set to the commonly used lengths (96, 192, 336, 720 for Exchange and 24, 36, 48, 60 for ILI). The results show that our model achieves the best average performance, which further proves the versatility of Cmamba.

• Q2: Computational analysis

Thanks for your suggestions. We have added the analysis of model efficiency, including model size and training time, in Appendix C.2 Efficiency Analysis in the new updated version. Here we show the results on the Weather dataset, with more details presented in Appendix C.2.

We can find that CMamba is the most lightweight model on average except for the Linear-based model (e.g., RLinear and TimeMixer). This indicates that CMamba’s excellent performance stems more from its capacity to capture historical time series patterns than from an increase in parameters. However, CMamba's running speed is not outstanding, which is primarily due to the suboptimal parallelism inherent to state space models (SSMs) compared to attention-based architectures. Nevertheless, CMamba's strong performance and lightweight model size still make it a highly efficient model, effectively balancing accuracy and computational efficiency.

• W5: Ablation study for M-Mamba is not convincing.

Due to space limitations, our discussion in M-Mamba is indeed insufficient. We have added the ablation results of vanilla Mamba and M-Mamba in the newly updated version (Appendix C.1). Here we briefly explain its content. As shown in the following table, where bold means improved performance (we only modify the backbone, so GDD-MLP and Channel Mixup are still adopted).

M-Mamba achieves performance gains in 58 out of 70 scenarios, with improvements becoming increasingly prominent at longer prediction horizons. This phenomenon is consistent with our architectural adjustments: removing the convolution operation and adopting a feature-independent matrix $\mathbf{A}$, which deemphasizes local temporal dependencies, allowing Mamba to better capture long-term dependencies. Recent studies, including iTransformer and ModernTCN, underscore the importance of a global receptive field and long-term dependency modeling for effective long-term time series forecasting. Specifically, iTransformer utilizes self-attention to achieve this, while ModernTCN leverages large convolution kernels. By removing modules emphasizing local dependencies, we enable Mamba to prioritize global dependency capture, as evidenced by the significant improvements in experimental results. These findings validate the effectiveness of our modifications in enhancing Mamba's capacity for long-term forecasting.

• Q1: Why MLP is optimal?

Thanks for the valuable question. We think MLP is optimal for its efficiency and potential to capture both correlations and irrelevances. For instance, in the extremely special case where there is no relationship between channels, MLP can be transformed into an identity module through learning. The universal approximation theorem also illustrates the potential of MLP. However, the original MLP cannot achieve good generalization effects because it is susceptible to distribution shifts. To cope with this, we propose GDD-MLP and Channel Mixup, which greatly enhances the generalization ability of MLP, as shown in Figure 4 in our paper. What's more, the recent NeurIPS 24 paper SOFTS[1], which proposes a centralized MLP architecture, demonstrates the strong ability of MLP in capturing cross-channel dependencies. The following table compares SOFTS with our CMamba, which shows that our model still outperforms it.

[1] Han, Lu, et al. "SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion." NeurIPS 2024.

We sincerely thank Reviewer ah3a for providing thorough and insightful comments. Here are our responses to your concerns and questions.

• W1: Inaccurate description of the current status and problems of Mamba-based research in time series.

Thanks for your valuable suggestions. We have elaborated on the current status and existing problems of Mamba in time series in the latest uploaded version, where Lines 53-59 describe the shortcomings of existing Mamba research in the time series and our motivation:

1. We point out that some current Mamba-based models in time series, e.g., S-Mamba, SiMBA, focus on replacing attention with Mamba, but ignore the study of Mamba's components.
2. We point out that Mamba lacks cross-channel dependency modeling capabilities. The current approach of using the SSM kernel to model channel dependencies is too complex and ignores the fact that channels are not sequences.

Hopefully, this will clear up your concerns.

• W2: The description of the Transformer in Lines 61-64 is ambiguous.

Thanks for your careful reading. The Transformer we are talking about actually refers to the entire Transformer encoder block, including the multihead self-attention module and the FFN module. Therefore, the MLP we mentioned in our paper refers to the FFN module. We are sorry for the ambiguity. We have corrected this in the latest uploaded version (Line 65-69).

• W3: Experimental results should not be included in the introduction.

Thanks for your suggestions. The Fig. 2 in the introduction section, which exhibits some experimental results is removed in the newly updated version.

• W4: The description of the current status of Mamba in time series in related work is too general.

Thanks for the valuable suggestions. We have elaborated the Section 2.1 in the latest uploaded version (Line 127-139):

1. We have cited and elaborated on more studies about state space models.
2. We introduce the latest state-space models for time series, e.g., S-Mamba, SiMBA, Time-SSM, and Bi-Mamba+. We describe their methods and point out their shortcomings.
3. We emphasize that our research focuses on studying the effectiveness of Mamba components and modeling cross-channel dependencies.

Hopefully, this will clear up your concerns.

It seems that you misunderstood our experimental settings. We highlight in our response to W5 that: we only modify the backbone, so GDD-MLP and Channel Mixup are still adopted, which means that the only difference between Mamba and M-Mamba is our modification, that is, Mamba here also takes patch-wise inputs (i.e., it is PatchMamba).

In terms of the cross-channel approach, we have proved in Fig. 4 that the vanilla MLP does not work well. Our approach, GDD-MLP and channel mixup, though simple, does enable the MLP architecture to achieve good results in capturing cross-channel dependencies.

We sincerely thank Reviewer Z3ph for providing thorough and insightful comments. Here are our responses to your concerns and questions.

• W1 a: Unclear description of contributions and motivation.

Thanks for your advice. The modeling of dependencies across time variables is both our motivation and purpose. Our description follows the following pipeline:

1. There are relationships between different time variables (shown in Fig. 1) but Mamba does not model cross-variate dependencies (Motivation )
2. --> We want to design a module to model cross-variate dependencies (Purpose) --> How?
3. --> The success of iTransformer: data-dependent and global receptive field (Motivation)
4. --> GDD-MLP and channel mixup. We hope that the above pipeline is more clear and solves your concerns.

• W1 b: The model does not consider the influence of exogenous variables.

Thanks for your advice. Nowadays, there are two mainstream directions in multivariate time series prediction:

1. predicting by better modeling the relationship (cross-time and cross-channel) between endogenous variables;
2. introducing exogenous variables to assist prediction.

In this paper, we mainly focus on the first direction. Both Table 1 and Table 3 have shown the effectiveness of CMamba in modeling these dependencies. We believe that exogenous variables are important and useful for time series prediction. We can introduce them into the model through attention, concat, or even feature addition. However, as we said in Appendix C.6 Limitations, this will be a direction we can explore in the future. In addition, in some real-world scenarios, it is much more difficult to obtain exogenous variables than endogenous variables. Therefore, we believe that how to rely solely on endogenous variables to perform time series forecasting is a more universal topic, and the corresponding results can also be well applied to scenarios that consider exogenous variables.

• W2: The improvement is not significant after considering cross-channel dependencies.

Thanks for the comments. If I understand correctly, you are referring to cross-channel (variable) dependencies rather than cross-time dependencies. The following descriptions are provided from the perspective of cross-channel dependencies:

Different models have different design focuses. For instance, TimeMixer focuses on cross-time dependency modeling and simply utilizes the channel-independent strategy. It achieves this with season & trend decomposition and multiscale mixing. However, our model does not employ these techniques. Hence, I think it is unfair to deny the necessity of cross-channel dependency modeling by comparing the improvement rate of CMamba relative to other models (like, TimeMixer). TimeMixer focuses more on cross-time modeling while we focus on cross-channel modeling. As shown in the following table, modeling cross-channel dependencies do improve CMamba's performance.

Admittedly, different datasets have different improvement rates because the complexity of the variable relationships in different datasets differs. For instance, some variables in the ETT dataset have obvious proportional relationships (e.g., MULL (Middle UseLess Load) is roughly equivalent to half of HULL (High UseLess Load)). Such dependencies are easy to model, and even a linear layer can simulate this kind of relationship, i.e., the input is proportional, so the output after linear projection is naturally proportional. This is why Linear-based models can always achieve sota performance on the ETT dataset. In more complex datasets, such as Weather and Electricity, our method shows advantages.

Dear Reviewer-Z3ph,

We appreciate Reviewer-Z3ph's feedback and would like to take this opportunity to further clarify two key contributions of our work:

• Firstly, we simplify the vanilla Mamba block. We propose a simplified Mamba block by removing the conv1d layer and simplifying matrix A, as these components were found to negatively impact long-term dependency modeling. The improvements achieved with the simplified design are particularly notable for longer prediction horizons, as illustrated in the table below. Given that removing these components enhances model performance, albeit modestly, we advocate for the adoption of this streamlined version of Mamba, aligning with the principle of model simplicity and efficiency.

• Secondly and more importantly, we focus more on cross-channel dependency modeling. The core innovation of our work lies in capturing cross-channel dependencies effectively through the use of GDD-MLP and Channel Mixup, which retain the MLP architecture and provide substantial improvements. Our experiments (refer to Fig. 4 in the main text) demonstrate that neither vanilla MLP nor generic mixup techniques work as effectively. In the following table, we replace the Mamba backbone in our model with MLP. We name the resulting model CMLP. As illustrated below, CMLP significantly outperforms vanilla MLP and even achieves comparable or superior results to iTransformer (and other benchmarks compared in our paper), with dramatically fewer parameters. This underscores the versatility and efficiency of GDD-MLP and Channel Mixup.

We hope this addresses your concerns and emphasizes the contributions of our work.

We sincerely thank Reviewer 86ct for providing thorough and insightful comments. Here are our responses to your concerns and questions.

• W1: CMamba does not involve the influence of exogenous variables.

Thanks for pointing out the weakness. We have discussed the limitations of our model in Appendix C.6 Limitations, where both exogenous variables and limited MAE improvement on the Traffic dataset are discussed. In this work, we mainly focus on cross-time and cross-channel dependencies modeling with endogenous variables, which is a well-built topic[1][2]. Both Table 1 and Table 3 show the effectiveness of CMamba in modeling these dependencies.

Admittedly, some recent work[3] has demonstrated that exogenous variables can improve the prediction accuracy of endogenous variables. However, exogenous variables are not the focus of this paper, so we only regard them as a direction for future exploration. In addition, in some real-world scenarios, it is much more difficult to obtain exogenous variables than endogenous variables. Therefore, we believe that how to rely solely on endogenous variables to perform time series forecasting is a more universal topic, and the corresponding results can also be well applied to scenarios that consider exogenous variables.

[1] Liu, Yong, et al. "itransformer: Inverted transformers are effective for time series forecasting." ICLR 2024.

[2] Nie, Yuqi, et al. "A time series is worth 64 words: Long-term forecasting with transformers." ICLR 2023.

[3] Wang, Yuxuan, et al. "Timexer: Empowering transformers for time series forecasting with exogenous variables." NeurIPS 2024.

• Q1: What are the specific advantages of the Mamba-based model compared to the MLP-based model?

Thanks for the question. Unlike other fields, Linear/MLP-based models have been proven to be very powerful models in time series prediction[4][5]. The advantage of Mamba-based or Transformer-based models over Linear/MLP-based models is their data-dependent ability. This enables these models to cope with more diverse scenarios. For instance, Transformer-based models can adaptively adjust the weights of feature fusion through attention scores to cope with different data inputs, while the Linear/MLP-based models can only use constant weights. Mamba-based models also have characteristics similar to those of Transformer-based models.

In the following table, we replace the M-Mamba backbone of CMamba with MLP. We name the resulting model CMLP. The results show that the Mamba backbone, with fewer parameters, performs better than the MLP backbone. What's more, CMLP largely outperforms pure MLP and iTransformer, which demonstrates the effectiveness of our modeling of cross-channel dependencies.

[4] Zeng, Ailing, et al. "Are transformers effective for time series forecasting?." AAAI 2023.

[5] Wang, Shiyu, et al. "Timemixer: Decomposable multiscale mixing for time series forecasting." ICLR 2024.

• Q2: MAE/MSE of some baseline models appear to differ from those reported in the original article (e.g., ModernTCN).

Thanks for your carefulness. Some results differ from those reported in the original article for two reasons:

1. These models use look-back windows of different lengths as input in the original paper. For instance, ModernTCN takes look-back length as a hyperparameter, and PatchTST adopts a look-back length of 336 or 512. Following the setting of most recent models, e.g., iTransformer, and TimesNet, we unify the look-back length as 96.
2. The drop_last bug stemming from Informer. This bug comes from an incorrect implementation of the dataloader, where the test dataloader uses drop_last=True, which may exclude a significant portion of test data, particularly with large batch sizes, leading to unfair model comparisons. It has influenced many models, e.g., PatchTST, ModernTCN, and TimeMixer. We fix the bug in our implementation. Therefore, the MAE/MSE of the models affected by the bug is different from the original results.