TimeBridge: Non-Stationarity Matters for Long-term Time Series Forecasting

Thanks for your constructive feedback and valuable insights into our work. Below, we address your concerns:

Q1: The loss function lacks sufficient discussion on its contribution to performance improvements.

A1: We conducted additional experiments where TimeBridge was trained using the MSE loss, denoted as "TimeBridge (MSE loss)", and compared the results with the best baseline, DeformableTST. Regardless of the loss function, TimeBridge outperforms DeformableTST in most datasets (See Table A), highlighting the effectiveness of our framework. We will incorporate the full results in the revision of our paper.

Table A: Average results of different prediction length {$96, 192, 336, 720$} with MSE Loss and Hybrid MAE Loss

Q2: The font size in Table 17 could be increased for better readability.

A2: Thank you for the suggestion. We will adjust the font size of the large tables in the revised version to enhance readability.

Q3: Could the authors provide further insights into the rationale behind their loss function choice and its comparative effectiveness? If all methods were evaluated under the same loss function, would the proposed approach still achieve significant improvements? Clarifying this point is crucial to distinguishing whether the performance gains stem from handling non-stationarity or from the loss function itself.

A3: From Table A above, it is clear that the hybrid MAE loss improves TimeBridge's performance on four ETT datasets. The ETT datasets have relatively few channels ($C=7$), which limits TimeBridge’s ability to utilize the Cointegrated Attention module to capture cointegration information. In fact, we only use the Integrated Attention to model ETT datasets (see Table 8 in our paper). Additionally, the ETT data exhibits a high degree of random fluctuation, which is better modeled using a loss function that weighs both time and frequency components. Hence, the hybrid MAE loss strengthens the modeling of short-term dependencies in such datasets. On datasets with more channels (e.g., Electricity, Traffic, Solar), the choice of loss function has minimal impact, as both losses yield similar results. We will incorporate this analysis in the revised version of our manuscript.

We are deeply grateful for your recognition of our work and your constructive feedback. If you have any further questions or suggestions, we would be more than happy to address them.

Thank you for your feedback and insightful comments on our work. Below, we address each of your concerns and questions:

Q1: As indicated in Table 6, the sequential selection strategy of CI and CD significantly impacts prediction performance on Electricity and Traffic datasets, yet this phenomenon remains insufficiently explained in the text.

A1: We believe that modeling short-term dependencies first is crucial because these local temporal features are essential for accurate short-term forecasting. By using Integrated Attention (CI) to capture short-term dependencies before Cointegrated Attention (CD) models long-term relationships, we ensure that important immediate patterns are not lost, which significantly improves performance. A similar "local first, global later" modeling strategy is also used in other works [1].

Q2: The manuscript lacks computational complexity analysis, making it difficult to assess the model's efficiency and memory requirements.

A2: We provide a detailed comparison of the theoretical complexity of our method with other mainstream Transformer-based models. The table below summarizes the theoretical computational complexity, where $C$ is the number of channels, $I$ and $O$ are the input and output lengths, and $S$ is the patch length:

Moreover, theoretical complexity alone cannot fully capture real-world performance due to implementation differences. We tested on 2 NVIDIA 3090 GPUs, measuring training (1 epoch) and inference times for three datasets of increasing size, with results averaged over 5 runs. The results in the table below indicate that TimeBridge outperforms most Transformer-based models.

Q3: While the study represents an effort in modeling cointegration through deep learning, the architectural design of the proposed modules appears relatively simplistic and could benefit from further sophistication.

A3: We would like to clarify that our method was intentionally designed with a simple, intuitive structure to address the challenges of non-stationarity and dependency modeling effectively:

• Integrated Attention focuses on capturing short-term dependencies while mitigating spurious regressions, using a streamlined approach that normalizes the attention map to remove non-stationary components.
• Cointegrated Attention preserves non-stationarity to model long-term cointegration relationships between variables.
• Patch Downsampling bridges the two blocks, allowing the Integrated Attention block to process aggregated short-term information, which feeds into the Cointegrated Attention block for long-term modeling.

We further provide a theoretical analysis of the modeling rationale behind these design choices, which can be found in Reviewer 1BTh’s Answer 1 (A1).

Q4: I am particularly interested in whether the financial experiments have been implemented in real-world trading scenarios? Could the authors provide some insights into the practical deployment?

A4: Thank you for your interest in the practical deployment of our model. Our financial experiments were conducted using the Qlib [2] platform for historical backtesting, with simulated trading costs to approximate real-world scenarios. While real-world deployment would need to address challenges such as latency and liquidity, we have not yet integrated with live trading systems due to data privacy and regulatory constraints.

[1] MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting (ICLR 2023)

[2] https://github.com/microsoft/qlib

Thank you for your valuable feedback. If you have any further questions or need clarification, feel free to follow up.

Thank you for your feedback and insightful comments on our work. Below, we address your questions:

Q1: It omits direct comparisons with seasonal-trend decomposition methods.

A1: We add comparison with seasonal-trend decomposition methods (Autoformer [1] and TDformer [2]). The table below shows the average prediction results across all datasets. Full results will be included in the revised paper.

Q2: The definitions of stationary and non-stationary in the paper seem to be mere rebranding of trend and seasonal components.

A2: While trend and seasonal components can be viewed as corresponding to non-stationary and stationary elements, previous works have mainly focused on modeling them within individual channels. They haven’t fully addressed the need to preserve non-stationarity across channels to capture long-term cointegration. Our paper’s core insight is that short-term dependencies can be modeled within stationary channels, while long-term cointegration requires preserving the non-stationary relationships between variables. Thus, we analyze non-stationarity’s distinct impacts on both intra- and inter-variable modeling, rather than just decomposing the sequence within channels as previous methods [1,2] have done.

Q3: The improvements are modest and don't definitively prove TimeBridge's superiority over existing methods.

A3: As shown in Table 1, TimeBridge consistently achieves superior performance in long-term forecasting, reducing MSE and MAE by 1.85%/2.49%, 5.56%/4.12%, and 13.66%/7.58% compared to DeformableTST, ModernTCN, and TimeMixer, respectively. Additionally, our model demonstrates exceptional performance in financial forecasting (Table 3), highlighting its ability to capture complex cointegration relationships in financial markets. These results confirm the effectiveness of our approach.

Q4: The methodology and structure of TimeBridge bear strong resemblance to several prior works, including [3,4,5,6].

A4: We design our model with a unique focus on non-stationarity in both short-term stability and long-term dependencies. Integrated Attention stabilizes short-term dependencies by removing non-stationary features from the Query-Key pairs, while Cointegrated Attention emphasizes the necessity of preserving non-stationarity for modeling long-term cointegration relationships between channels. In contrast, PatchTST [5] does not address non-stationarity and focuses solely on the temporal dimension, missing the rich cointegration relationships between variables. Crossformer [3] and UniTST [6] model short-term dependencies across and within channels but are theoretically susceptible to spurious regressions and lack explicit consideration of long-term cointegration. Fredformer [4], though utilizing frequency-domain attention to capture long-term signals, struggles with fine-grained temporal trends in the time domain, limiting its ability to model short-term dependencies effectively.

Q5: A more detailed ablation study is needed to clarify the individual contributions of Integrated and Cointegrated Attention in mitigating non-stationarity.

A5: We have conducted an ablation study on the impact and order of Integrated Attention and Cointegrated Attention in Table 5 in our paper. The results indicate that both attention mechanisms must be used together, with Integrated Attention applied first, followed by Cointegrated Attention.

Q6: The theoretical grounding of Integrated and Cointegrated Attention should be more explicitly justified within the context of existing time series modeling literature.

A6: We have provided a more explicit theoretical justification for Integrated and Cointegrated Attention in Reviewer 1BTh’s Answer 1 (A1).

Q7: Would a decomposition-based or frequency-domain approach offer comparable performance with potentially greater computational efficiency?

A7: We compared TimeBridge with methods such as TimeMixer (decomposition-based), TimesNet (frequency-domain), and the newly added Autoformer and TDformer in A1, all of which demonstrate that TimeBridge consistently outperforms them. While decomposition-based or frequency-domain methods may offer comparable performance, we currently do not have conclusive evidence to support this.

We appreciate your insightful feedback. If you have any further questions or need clarification, feel free to follow up.

Thanks for your insightful comments on our work. Below, we address each of your questions:

Q1: A more detailed theoretical exploration could strengthen the framework's validity.

A1: Thank you for your valuable comments. Below we provide a concise theoretical explanation grounded in classical stochastic processes, specifically Brownian motion, to clarify the rationale behind our design.

Proposition 1: Spurious Attention from Non-Stationary Inputs

Consider a standard Brownian motion $X_t \sim I(1)$, where:

$X_t = X_{t-1} + u_t,\quad u_t \sim \mathcal{N}(0, \sigma^2)$.

Then we have:

$\text{Mean}(X_t)=0, \quad \text{Var}(X_t) = t\sigma^2,\quad \text{Cov}(X_{t_1}, X_{t_2}) = \min(t_1, t_2)\sigma^2$

Let two input patches of length $S$ be:

$p_i = [X_{t+i+1}, \dots, X_{t+i+S}],\quad p_j = [X_{t+j+1}, \dots, X_{t+j+S}]$

Their attention score $\text{score}(p_i, p_j)$ is approximated as:

$\text{score}(p_i, p_j) \propto p_i p_j^T \propto \sum_{s=1}^{S} (X_{t+i+s} X_{t+j+s}) \propto \sum_{s=1}^{S} \text{Cov}(X_{t+i+s}, X_{t+j+s})$

$\propto \sigma^2 \left(S\min(i, j) + \frac{S^2 + 2St + S}{2}\right)$

This score grows with both the time index $t$ and the square of the patch length $S$, leading to spurious attention caused by global trends rather than genuine short-term dependencies. As shown in Figure 4(a), many patches tend to exhibit high attention scores.

To mitigate this, we perform patch-wise detrending:

$p_i' = \text{Detrend}(p_i) = [\Delta X_{t+i}, \dots, \Delta X_{t+i+S}] \sim I(0), \quad \Delta X_t = X_t - X_{t-1}$

In this case, the variance becomes stable:

$\text{Var}(\Delta X_t) = \sigma^2,\quad \text{score}(p_i', p_j') \propto S\sigma^2$

This makes the attention mechanism focus on true short-term patterns, unaffected by long-term drift.

---

Proposition 2: Importance of Non-Stationarity in Capturing Cointegration

Let $X_t, Y_t \sim I(1)$ be two non-stationary time series with a cointegration relationship:

$Z_t = X_t - \beta Y_t \sim I(0)$

where $\beta$ is a constant coefficient. If we remove non-stationarity via detrending:

$\Delta X_t = \text{Detrend}(X_t),\quad \Delta Y_t = \text{Detrend}(Y_t)$

Then:

$Z_t = \Delta X_t - \beta \Delta Y_t = \epsilon_t$

which becomes a random noise sequence. This destroys the cointegration signal, making it impossible for attention mechanisms to capture long-term equilibrium relationships. Figure 4(b) illustrates that removing non-stationary components eliminates the vast majority of cointegration information.

---

These two propositions together explain our architectural choice:

• For short-term modeling, we eliminate non-stationarity to avoid spurious regressions.
• For long-term modeling, we preserve non-stationarity to retain meaningful cointegration.

We will incorporate this theoretical analysis in greater detail in the revision of our paper.

Q2: Highlighting distinct advantages or improvements over these methods would clarify the contribution of this work.

A2: We design our model by focusing on the distinct characteristics of non-stationary data, specifically addressing short-term stability and long-term dependencies. This insight leads to the adoption of simple yet effective strategies that align with the nature of non-stationary data. Unlike other attention methods that directly model short-term dependencies and are prone to spurious regressions, Integrated Attention stabilizes short-term modeling by removing non-stationary features from the Query-Key pairs. In contrast, Cointegrated Attention emphasizes the necessity of retaining non-stationary features for capturing long-term cointegration relationships, setting it apart from recent channel-dependent Transformer models. These models either fail to consider long-term cointegration [1] or overlook it entirely [2].

[1] Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting

[2] Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling

Q3: ARIMA is a strong baseline for short-term forecasting; the authors should compare its performance to highlight the advantages of their framework.

A3: Since ARIMA is for univariate forecasting and our PeMS datasets are multivariate, we compared TimeBridge with VARIMA, which is more appropriate for multivariate forecasting. The table below shows that TimeBridge outperforms VARIMA in most cases. This comparison will be updated in the revised paper.

We sincerely appreciate your constructive feedback. If you have any further questions or need clarification, please feel free to reach out.