Dear Authors:

Thank you for your rebuttal!

A1: The author also claims that this is the property of stability of model after some factors are affected, so according to the description of the property of the neural network in the statistical learning book, this should be called stability. This is mainly because robustness is too easily understood to mean that the model performance does not degrade after being disturbed by input noise.

A2: If the data set size is small (i.e., ETT dataset), then the proposed method seems to have under-fitting on small-scale tasks, which will be an obstacle for the model to solve small-scale time series forecasting tasks.

I was generally satisfied with the rest of the responses. Thus, I'm willing to increase the score to 6.

W1. Definition of robustness

We understand that robustness refers to the stability of results or performance despite variations in certain factors, and is not solely related to the context of adversarial attacks, as the reviewer mentioned. As noted in the L17, the robustness we discuss specifically pertains to 'robustness to channel order' in TS, particularly regarding the channel order that Mamba receives as input

W2. Performance of ETT datasets

It is worth noting that (1) CD models, which capture dependencies between channels, typically have high capacity but lower robustness, whereas CI models exhibit the opposite [A,B]; and (2) the ETT datasets have the relatively small size among the benchmark datasets, as shown in Appendix A.1.

Due to these factors, CI models (e.g., PatchTST [C], RLinear [D]) generally outperform CD models (e.g., iTransformer [E], S-Mamba [F], SOR-Mamba [ours]) on smaller datasets, including the ETT datasets. We believe that the relatively lower performance on the ETT datasets is not due to regularization strategy, but rather to the smaller dataset size, where CI models tend to be more suitable.

[A] “The Capacity and Robustness Trade-off: Revisiting the Channel Independence Strategy for Multivariate Time Series.” TKDE (2024)

[B] Nie, Tong, et al. "Channel-aware low-rank adaptation in time series forecasting." CIKM (2024)

[C] Nie, Yuqi, et al. "A time series is worth 64 words: Long-term forecasting with transformers." ICLR (2023)

[D] Li, Zhe, et al. "Revisiting long-term time series forecasting: An investigation on linear mapping." arXiv (2023)

[E] Liu, Yong, et al. "iTransformer: Inverted transformers are effective for time series forecasting." ICLR (2024)

[F] Wang, Zihan, et al. "Is mamba effective for time series forecasting?." arXiv (2024)

W3. Training and inference time

We appreciate the reviewer's interest in the practical efficiency of our approach. However, we have already conducted an analysis regarding the efficiency in terms of (1) the number of parameters, (2) complexity and memory, and (3) computational time (for both training and inference) using the Traffic dataset, which has a large number of channels (C=862), as shown in Table 13.

W4. Misunderstanding of the term “sequence order”

Thank you for pointing this out. The varying sampling frequencies and periodic change rules for different channels in Figure 1 were intended solely to illustrate our motivation in representing distinct channels. We agree that this may be confusing, so we will add further explanations to clarify this point.

W1. Why Mamba for capturing CD?

As noted in Line 39–41, our focus is on Mamba’s role in capturing CD, in line with recent work [A] that advocates for using complex attention mechanisms for CD while employing simple multi-layer perceptrons (MLPs) for TD. Consequently, recent works [B,C] have also explored using Mamba as an alternative to attention for capturing CD, aligning with our direction. In addition, we present further experiments on using Mamba to capture temporal dependency (TD) in Table 11. These results are consistent with previous studies, suggesting that complex architectures like attention or Mamba are more effective for capturing CD than TD.

[A] Liu, Yong, et al. "iTransformer: Inverted transformers are effective for time series forecasting." ICLR (2024)

[B] Wang, Zihan, et al. "Is mamba effective for time series forecasting?." arXiv (2024)

[C] Ma, Shusen, et al. "Fmamba: Mamba based on fast-attention for multivariate time-series forecasting." arXiv (2024)

W2. Longer lookback window ($L$)

Thank you for your insightful comment.

We set $L=96$ to align with the experimental protocols used in previous works (iTransformer [A], S-Mamba [B]). However, as the reviewer suggested, we conducted an additional ablation study on input length across four datasets (Traffic, Electricity, PEMS04, ETTm1), testing various lookback lengths $L \in \\{48, 96, 192, 336, 720\\}$, with forecast horizon of $H= 12$ for PEMS04 and $H = 96$ for the other datasets, in accordance with previous works [A,B]. The results, shown in Appendix L, indicate that performance remains robust to the choice of $L$ for some datasets and even improves with a larger $L$ for others.

[A] Liu, Yong, et al. "iTransformer: Inverted transformers are effective for time series forecasting." ICLR (2024)

[B] Wang, Zihan, et al. "Is mamba effective for time series forecasting?." arXiv (2024)

W3. Calculation of (global) correlation

Thank you for pointing this out.

We agree that the description of calculating the (global) correlation could be clearer. The global correlation, as mentioned in L369--370, refers to the correlation between channels across the entire "training" TS dataset, excluding the validation and test datasets. Details on the train-validation-test split protocol are described in Appendix A.

Thanks for the author's responses to my questions. I still think the motivation of using Mamba to capture cross-channel dependencies is unconvincing. Additionally, it would be beneficial to evaluate the impact of the lookback window on other datasets, such as ETTH1. I will keep my original score and discuss with other reviewers.

2. Comparison with PatchTST

We believe your statement, "(I find that) the performance of SOR-Mamba is worse than the results of PatchTST", is inaccurate, as shown in Table 2, where our method outperforms PatchTST on 11 out of 13 datasets.

It is important to note that our work uses the same $L$ for all baseline methods for fair comparison, along with previous works. To clarify, we present five representative SOTA algorithms, listed chronologically by their release dates, along with the number of citations (as of 2024.12.02) and their input length ($L$):

• [A] PatchTST (ICLR 2023, 1031 citations): $L = 512$ (for self-supervised setting) and $L = 336$ (for supervised)
• [B] Crossformer (ICLR 2023, 523 citations): $L \in \\{24, 48, 96, 168, 336, 720\\}$ (tuned as a hyperparameter)
• [C] RLinear (arXiv 2023, 74 citations): $L = 336$
• [D] iTransformer (ICLR 2024, 432 citations): $L = 96$
• [E] S-Mamba (arXiv 2024, 19 citations): $L = 96$
• [Ours] SOR-Mamba: $L = 96$

These works use different $L$ values, and aligning them to a unified setting (along with an ablation study of varying $L$) is a common practice. Furthermore, it is important to note that our setting follows the experimental settings and analyses established in recent SOTA work [D,E], which is widely used as a backbone for recent methods [F, G], and the comparison with PatchTST is also done within this context.

We respectfully believe that claiming our method underperforms PatchTST overlooks the experimental settings and analyses established in recent SOTA work [D], which are commonly used in contemporary methods [F, G].

[A] Nie, Yuqi, et al. "A time series is worth 64 words: Long-term forecasting with transformers." ICLR (2023)

[B] Zhang, Yunhao, and Junchi Yan. "Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting." ICLR (2023)

[C] Li, Zhe, et al. "Revisiting long-term time series forecasting: An investigation on linear mapping." arXiv (2023).

[D] Liu, Yong, et al. "iTransformer: Inverted transformers are effective for time series forecasting." ICLR (2024)

[E] Wang et al. "Is mamba effective for time series forecasting?." arXiv (2024)

[F] Dong, et al. "TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling." ICML (2024)

[G] Gao, Shanghua, et al. "Units: Building a unified time series model." NeurIPS (2024)

3. Performance of PatchTST on small-scale datasets

It seems that the reviewer is particularly interested in the ETT datasets (as the reviewer requested an additional ablation study with ETTh1, and PatchTST (CI model) is one of the SOTA models for the ETT datasets). However, for other datasets, CD models (including Ours) generally perform better, demonstrating the effectiveness of these algorithms.

To be precise, according to Table 10 of iTransformer [A] and Table 3 of S-Mamba [B], both results show that, for the ETT series (ETTh1, h2, m1, m2), iTransformer and S-Mamba, which are CD models, perform worse than both RLinear [C] and PatchTST [D], which are CI models.

However, it is worth noting that

• (1) CD models typically have high capacity but lower robustness, whereas CI models exhibit the opposite [E, F]; and
• (2) ETT datasets are relatively small compared to other benchmark datasets, as shown in Appendix A.1.

We note that this issue regarding the ETT datasets, has already been addressed, which was also raised by Reviewer Cgt6.

[A] Liu, Yong, et al. "iTransformer: Inverted transformers are effective for time series forecasting." ICLR (2024)

[B] Wang et al. "Is mamba effective for time series forecasting?." arXiv (2024)

[C] Nie, Yuqi, et al. "A time series is worth 64 words: Long-term forecasting with transformers." ICLR (2023)

[D] Li, Zhe, et al. "Revisiting long-term time series forecasting: An investigation on linear mapping." arXiv (2023).

[E] Han, L., Ye, H. J., & Zhan, D. C. “The Capacity and Robustness Trade-off: Revisiting the Channel Independence Strategy for Multivariate Time Series.” TKDE (2024)

[F] Nie, Tong, et al. "Channel-aware low-rank adaptation in time series forecasting." CIKM (2024)

We hope the three responses above address your concerns and provide clarification.

Q1. Hyperparameter lambda and the effect of regularization

As the reviewer mentioned Table I.1 illustrates that the inclusion of the regularization loss significantly improves the performance regardless of the value of lambda.

This phenomenon is reasonable, as shown in Figure 9 and Figure G.1, where the regularization loss converges very quickly early in the training process for our proposed methods. This rapid convergence suggests that the regularization effectively aligns the two reversed-order vectors with ease, yielding consistent performance improvements regardless of the value of lambda.

Q2. Motivation of (1) single unidirectional Mamba and (2) removal of 1D-conv.

(1) Motivation of single unidirectional Mamba

Previous research used two separate Mambas to capture channel dependencies from both directions [A, B] in an effort to address the channel order bias. However, we found that using bidirectional Mamba does not always yield the best performance (Table 1) and can be inefficient (L51--52). By using a single Mamba with regularization loss, we effectively address this bias, achieving superior performance (Table 2,4,5) and improved efficiency (Table 13) compared to using two Mambas."

(2) Motivation of removal of 1D-conv

The 1D-conv in Mamba [C] was originally designed to capture local information in sequential data (L66--68), making it less suitable for non-sequential data (i.e., channels in TS). However, previous works have overlooked the need to remove the 1D-conv when using Mamba to capture channel dependencies (CD) [D,E,F], where channels do not have sequential order. This has motivated us to remove it from the Mamba block. Further details on the removal of the 1D-conv are provided in Appendix D.

[A] Wang, Zihan, et al. "Is mamba effective for time series forecasting?." arXiv (2024)

[B] Liang, Aobo, et al. "Bi-Mamba4TS: Bidirectional Mamba for Time Series Forecasting." arXiv (2024)

[C] Gu, Albert, and Tri Dao. "Mamba: Linear-time sequence modeling with selective state spaces." arXiv (2024)

[D] Weng, Zixuan, et al. "Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting." arXiv (2024)

[E] Cai, Xiuding, et al. "MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting." arXiv (2024)

[F] Ma, Shusen, et al. "Fmamba: Mamba based on fast-attention for multivariate time-series forecasting." arXiv (2024)

Q3. Lack of details regarding CCM

We apologize for the lack of detail regarding the proposed pretraining task, CCM. To address this, we have included pseudocode for the task in Appendix J and provided additional guidance for readers in L244.

W1. Motivation for regularization

It is important to clarify that the proposed regularization was not specifically intended to enhance the capture of channel correlation, but to address the sequential order bias.

As shown in Table 1 and Figure 7, previous methods employing bidirectional Mamba to handle sequential order bias still exhibited sensitivity to channel order, both in terms of performance and qualitative results. This motivated us to develop a more effective approach to address the bias: the regularization strategy. This approach yields a more consistent representation that is robust to changes in channel order, as demonstrated in Table 10, Appendix H.1 and Figure 7.

W2. Longer lookback window

We set $L=96$ to align with the experimental protocols used in previous works (iTransformer [A], S-Mamba [B]). However, as the reviewer suggested, we also conducted an ablation study on input length across four datasets (Traffic, Electricity, PEMS04, ETTm1), testing various lookback lengths $L \in \\{48, 96, 192, 336, 720\\}$, with forecast horizon of $H= 12$ for PEMS04 and $H = 96$ for the other datasets, in accordance with previous works [A,B]. The results, shown in Appendix L, indicate that performance remains robust to the choice of $L$ for some datasets and even improves with a larger $L$ for others.

[A] Liu, Yong, et al. "iTransformer: Inverted transformers are effective for time series forecasting." ICLR (2024)

[B] Wang, Zihan, et al. "Is mamba effective for time series forecasting?." arXiv (2024)

W3. Clarity of sentences

Thank you for your feedback.

We understand that the excessive use of abbreviations without clear explanations can make the manuscript and experimental result tables difficult to follow.

Below is a list of the abbreviations used:

• CI/CD: channel independence/channel dependence
• TD: temporal dependencies
• LTSF: long-term TS forecasting
• 1D-conv: 1D-convolution
• SSL/SL: self-supervised learning / supervised learning
• FT/LP: fine-tuning / linear probing,
• MSE/MAE: mean squared error / mean absolute error

We believe these abbreviations are commonly used in previous works. However, we will replace the abbreviations (TD, FT, LP) with full expressions, as they are not widely recognized, and expand (LTSF), as it appears infrequently in our manuscript. We will revise the text to provide clearer definitions and reduce abbreviation use to improve readability.

W1. Complexity of proposed architecture

Thank you for your feedback regarding the complexity of our approach. We believe the proposed method is both simple and effective, as noted by reviewer 9Yaz.

(1) Simple architecture

The removal of the 1D-conv is straightforward and does not require any additional changes to the remaining architecture, as detailed in Appendix D. For the proposed regularization, we utilize embeddings from the existing architecture without introducing any additional modules or embeddings, as shown in Appendix C. This approach preserves simplicity and effectiveness while minimizing computational overhead.

(2) Simple pretraining task

For the CCM pretraining task, only the correlation between output vectors needs to be computed, with no need for extra modules. To clarify the process, we have included pseudocode for CCM in Appendix J and further guidance in L244, highlighting the task’s simplicity.

To facilitate replication and ensure accessibility, we have also provided the code in the supplementary material.

W2. Trade-off in removing 1D-conv

It is important to clarify that our primary motivation for removing the 1D-conv is not efficiency, but rather to address general TS datasets, where channels typically do not have a sequential order. We believe it is reasonable to design a model architecture suited to these general cases.

As shown in Table 7 and Figure 5, removing the 1D-conv of Mamba in datasets where channel order has inherent structure (e.g., traffic data) does not always lead to performance benefits. However, this finding further supports our approach, indicating that the 1D-conv captures sequential relationships between tokens (channels). When channels have an inherent order, which is identifiable based on the dataset domain or type, removing the 1D convolution may not be desirable.

Further details on the rationale and process for removing the 1D-conv are provided in Appendix D.

W3. Hyperparameter sensitivity

Thank you for pointing this out. We have, however, already discussed the robustness of our proposed method to hyperparameters across various datasets, as follows:

• Robustness to hyperparameter $\lambda$: Appendix I.1
• Robustness to distance metric (for regularization): Appendix K.1 (as guided in L207)
• Robustness to distance metric (for CCM): Appendix K.2 (as guided in L224--255)

Q1. Robustness to distance metric (for regularization)

Q1-a. Relationship between the distance metric $d$ and the performance

As discussed in Appendix K.1, the performance gains from regularization loss are robust across different choices of the distance metric $d$. However, we did not observe a consistent relationship between the choice of metric and the specific characteristics of the TS datasets.

Q1-b. Relationship between the effectiveness of regularization and the characteristic of dataset

While Figure 4 shows that the degree of bias varies with the number of channels and channel correlation, we believe that the relationship between performance gains (i.e., the effectiveness of the regularization) and bias reduction is influenced by various inherent factors.

Q2. Effect of pretraining tasks by dataset characteristics

We agree on the importance of analyzing the performance gains of different pretraining tasks and dataset characteristics, particularly in terms of the number of channels. For this reason, we have already discussed this in Figure 6, which demonstrates that the proposed pretraining tasks outperform both MM and reconstruction tasks across datasets with both large and small numbers of channels. Additionally, as shown in the right panel of Figure 4, the correlation between channels varies across datasets, with the ETT datasets exhibiting low correlation. Despite this, our method performs well on these datasets, demonstrating its robustness and effectiveness even in cases with weak channel correlations.

W1. The regularization strategy is overly simplistic

We believe that the regularization strategy should be simple and efficient to effectively replace the bidirectional Mamba for capturing CD. Our proposed regularization approach is intentionally straightforward and efficient, leveraging embeddings directly from the existing architecture without requiring additional modules or embeddings. This design minimizes computational overhead while providing effective regularization within the model.

W2. Justification of removal of 1D-conv

It is important to note that we remove 1D-conv primarily for general TS datasets, where channels do not inherently have a sequential order. We believe it is reasonable to design a model architecture suited to these general cases.

As shown in Table 7 and Figure 5, removing the 1D-conv of Mamba in datasets where channel order has inherent structure (e.g., traffic data) does not always lead to performance benefits. However, this finding further supports our approach, indicating that the 1D-conv captures sequential relationships between tokens (channels). When channels have an inherent order, which is identifiable based on the dataset domain or type, removing the 1D convolution may not be desirable.

Further details on the rationale and process for removing the 1D-conv are provided in Appendix D.

Q1. Comparison with other (Order-invariant) Mamba architectures

As mentioned in Section 2. Related Works, several concurrent studies have utilized Mamba to capture CD [A,B,C,D], and these methods generally address the sequential order bias by employing the bidirectional Mamba [A,B,C,D]. However, aside from our baseline method [A], most of these works have not released code for reproducibility, making direct comparisons with our approach currently infeasible. We look forward to conducting comparisons when these implementations become available.

[A] Wang, Zihan, et al. "Is mamba effective for time series forecasting?." arXiv (2024)

[B] Liang, Aobo, et al. "Bi-Mamba4TS: Bidirectional Mamba for Time Series Forecasting." arXiv (2024)

[C] Weng, Zixuan, et al. "Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting." arXiv (2024)

[D] Behrouz, A., Santacatterina, M., & Zabih, R. “Mambamixer: Efficient selective state space models with dual token and channel selection" arXiv (2024)

Q2. Definition of sequential order bias

As mentioned in L322--333, the sequential order bias is quantified by measuring the performance difference (average MSE across four horizons) when the channel order is reversed, using SOR-Mamba without regularization.

Q1. Channel order shuffling method (fixed shuffling method vs. shuffled randomly each time)

We use a fixed shuffling approach for channel order, with results averaged over five different random seeds. Using a random shuffling each time would impair the model’s ability to distinguish between channels, leading to poor performance, as discussed in Table 12, Figure 9, and L466–470.

Q2. Incorrect calculation formula (Last row of Table 1)

Thank you for pointing this out. We initially reversed the sign of the last row of Table 1 to indicate the degree of improvement, as a lower MSE indicates better performance. However, following the reviewer’s feedback, we corrected the sign to avoid any confusion.

W1-a) Reverse modeling is widely used

We acknowledge that reverse modeling (bidirectional Mamba) is widely used, as noted in L46--51. However, we highlight its inherent limitations in Table 1, where the bidirectional Mamba does not consistently yield the best performance and, in some cases, underperforms compared to a single unidirectional Mamba. Additionally, the bidirectional Mamba uses two Mambas for tuning, doubling the computational burden compared to a single Mamba. In light of these limitations, we propose an alternative to reverse modeling (bidirectional Mamba): a single unidirectional Mamba with regularization, which demonstrates improved performance and efficiency, as shown in Table 2,5 and Table 13, respectively.

W1-b) Removal of 1D-conv is trivial

While removing the 1D-conv might seem minor, many studies in the TS domain often adopt architectures directly from NLP or vision tasks without accounting for TS-specific properties. Since the 1D-conv in Mamba [A] was originally designed to capture sequential patterns (L66--68), applying it to non-sequential TS channels is an unnatural choice, yet it has been commonly used without adaptation. We believe our approach is non-trivial, as it specifically accounts for the distinctive characteristics of TS datasets that have been overlooked in other Mamba-based methods. Furthermore, the proposed method has the potential for application in other domains involving non-sequential data (e.g., tabular data).

Further details on the removal of the 1D-conv are provided in Appendix D.

[A] Gu, Albert, and Tri Dao. "Mamba: Linear-time sequence modeling with selective state spaces." arXiv (2024)

W2. Performance improvement is not significant.

It is important to clarify that our primary goal is not to achieve substantial performance gains over the baseline, but rather to demonstrate that a single unidirectional Mamba with a regularization strategy can effectively replace the widely used bidirectional Mamba, yielding comparable or even better performance with greater efficiency, as shown in Table 5 and Table 13. Additionally, applying the regularization strategy to the bidirectional Mamba also results in performance gains, as shown in Table 6.

While the optimal model structure (and hyperparameters) may vary between the unidirectional and bidirectional Mamba setups, we used the original hyperparameters from the bidirectional Mamba for the unidirectional Mamba (see L265--266), allowing us to evaluate results on the same basis. Optimizing these hyperparameters specifically for the unidirectional Mamba configuration could likely yield additional performance gains.

W3. Necessity of removing 1D-conv

It is important to note that we remove 1D-conv primarily for general TS datasets, where channels do not inherently have a sequential order. We believe it is reasonable to design a model architecture suited to these general cases.

As shown in Table 7 and Figure 5, removing the 1D-conv of Mamba in datasets where channel order has inherent structure (e.g., traffic data) does not always lead to performance benefits. However, this finding further supports our approach, indicating that the 1D-conv captures sequential relationships between tokens (channels). When channels have an inherent order, which is identifiable based on the dataset domain or type, removing the 1D convolution may not be desirable.

Further details on the rationale and process for removing the 1D-conv are provided in Appendix D.

W4. Comparison with other Mamba models.

As mentioned in Section 2. Related Works, several concurrent studies have utilized Mamba to capture CD [A,B,C,D], and these methods generally address the sequential order bias by employing the bidirectional Mamba [A,B,C,D]. However, aside from our baseline method [A], most of these works have not released code for reproducibility, making direct comparisons with our approach currently infeasible. We look forward to conducting comparisons when these implementations become available.

[A] Wang, Zihan, et al. "Is mamba effective for time series forecasting?." arXiv (2024)

[B] Liang, Aobo, et al. "Bi-Mamba4TS: Bidirectional Mamba for Time Series Forecasting." arXiv (2024)

[C] Weng, Zixuan, et al. "Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting." arXiv (2024)

[D] Behrouz, A., Santacatterina, M., & Zabih, R. “Mambamixer: Efficient selective state space models with dual token and channel selection" arXiv (2024)