Correlated Attention in Transformers for Multivariate Time Series

We are really appreciative of your acknowledgement of our work's contribution. Per your request, we have also included additional experiments in Appendix C in the revised manuscript on long-term forecasting for three common datasets spanning various domains. The results demonstrate the benefit of our CAB even in forecasting. Given this and if the following concerns of yours are addressed, we hope you may consider increasing the score for our work.

$**Q1:**$ "Can you provide a more detailed evaluation of the Correlated Attention Block (CAB) on prediction tasks, such as the MLTSF dataset? It would be interesting to see how CAB performs compared to other models that utilize inter-variable correlations in prediction tasks. This analysis would shed light on the effectiveness of CAB in different scenarios and provide a better understanding of its potential advantages."

$**Response:**$ As mentioned above, we have included additional experiments for Multivariate Long-term Time Series Forecasting in Appendix C in the revised manuscript. As shown therein, when paired with a strong baseline for long-term forecasting, CAB results in competitive performance. Specifically, in the dataset like Exchange in our test, CAB gives significant performance boost for longer series sequence. With the flexibility of CAB, we look forward to its deployment on top of many existing Transformers for efficiency improvement.

$**Q2:**$ "In Table 2 and Table 3, it is observed that in some cases, the performance of CAB+Transformer is not as good as Nonstationary, and in some cases, Nonstationary+CAB even leads to worse results. Can you provide an explanation for these observations? What factors contribute to the varying performance of the proposed method in different settings? Understanding the limitations and potential trade-offs of the proposed method would provide valuable insights for future improvements."

$**Response:**$ For the anomaly detection task in Table 3, we note that the main metrics within interest is usually the F1 score, which is the harmonic mean of precision (P) and recall (R), thereby giving balance between the two. In terms of F1, CAB actually provides consistent improvement when plugged into both Transformer and Non-stationary Transformer, where decrease in F1 score is only in one dataset for each base model. For the imputation task in Table 2, this task differs from other tasks in that it involves missing data. This can make the representation learning in CAB harder, as the lagged cross-correlation filtering step is sensitive to time step data. For example, if a time lag with high correlation is yet missing, CAB would not have data to detect such potentiall lagged cross-correlation.

$**Q3:**$ How were the hyperparameters determined in the experiments? Specifically, can you provide more details on the selection of hyperparameters such as the initial value of lambda (\lambda) and the values of k and c? Understanding the rationale behind these choices would help in reproducing the results and provide insights into the sensitivity of the proposed method to hyperparameter settings.

$**Response:**$ We kindly refer the reviewer to our Appendix A in the paper for details on implementation and hyperparameter setting, which would provide information to all of your question.

$**Q4:**$ "Non-compliance with ICLR submission requirements: The paper does not follow the submission requirements of ICLR by placing the appendix together with the main text. It would be better to separate the appendix from the main text, following the formatting guidelines specified by the conference."

$**Response:**$ We thank you for that, and will separate the main text from the supplementaries in later versions including our revised manuscript now.

---

We thank you again for your meaningful comments. If there is any concern left, we would be pleased to answer in the remaining time of the discussion period. If our positive results on forecasting as well as asnwers could have properly addressed your main concerns, we hope that you could consider raising the score for our work.

We thank the reviewer very much for the insightful comments as well as the acknowledgement of our proposed method's benefits. Please find our revised manuscript taking into account your suggestions and providing additional experiments in the appendix to address the reviewer's concerns on computational complexity of CAB and CAB's performance on predictive tasks. Given the new positive experimental results and if we can properly address all of your concerns, we really hope that you can increase the score for our work.Now, we hereby would like to address the main concerns of the reviewer.

$**Q1:**$ "It would be beneficial to provide more clarity on the specific aspects of CAB's design that contribute to this efficiency."; "Could you provide a detailed breakdown of the distinct contributions of each component within CAB, both in terms of performance enhancement and efficiency gains?"

$**Response:**$ We kindly refer the reviewer to the ablation studies in Appendix B for performance analysis of specific aspects of CAB. In the revised manuscript, we added blue text further highligting the most crucial components of the CAB. For a summary, from the empirical findings, the lagged cross-correlation filtering component (even when we hard-fix the learnable parameters as constants) resulted in highest increase in the accuracy of 1.39% from the basic CAB model without lagged cross-correlation. Then, given that the lagged-cross correlation filtering is used, letting the parameter $\beta$, which is in Eq. (6) of our paper and balances the effect of instantaneous and lagged cross-correlation, be learnable would result in the highest increase in accuracy of 1.81%.

$**Q2:**$ "Could you elaborate on the primary challenges or obstacles preventing the integration of CAB into encoder-decoder models for conducting multivariate time series forecasting?"; "It's important to acknowledge that the design of CAB is limited to encoder-only models and does not support time series forecasting. While this limitation is briefly mentioned, expanding on the reasons behind this constraint and discussing potential avenues for future work or extensions to address this limitation would add depth to the paper."

$**Response:**$ Before providing detailed explanations, we would like to note that:

• We provide new experiments in Appendix C of the revised manuscript demonstrating that the CAB, even when naively plugged into the decoder architecture of base architectures, indeed consistently improves the performance of such Transformer-models over three long-term forecasting datasets spanning different domains!
• We believe that the integration of CAB into the decoder can be done even better in the future with proper design of masking mechanism, as discussed in details below.

First, we recall the main difference between the decoder and the encoder is that the decoder has two multi-head attention (MHA) blocks: one with masking and one without masking (i.e. conventional MHA). The masking is meant to prevent information to preserve the auto-regressive property, so that the forecasting of the next time step should depend only on the time steps strictly before it. In typical self-attention mechanism, this is done by masking out (setting to $-\infty$) all values in the lower triangular of the scoring matrix $Q K^T\in \mathbb{R}^{T \times T}$ that is input to softmax. As you can see, $Q K^T$ can be thought of as the attention/correlation matrix in time (with dimension $T\times T$), so masking out the triangular part corresponds to "blinding" the history in making predictions; one the other hand, in CAB, the $Roll(K, l)^T Q \in \mathbb{R}^{d_k \times d_k}$, which is input to softmax, is correlation matrix across feature with dimension $d_k \times d_k$ instead of time, so the same masking strategy, even if naively applicable, would not have intuitive or physical phenomena as in vanilla self-attention, thereby being more of a heuristic approach. In the experiments in Appendix C, we only replace the non-masked MHA with our mixture-of-head attetions comprised of temporal attentions and our CAB, while keeping the masked MHA of the base Transformer. As you can see from the results, the CAB consistently improves performance of Non-stationary Transformer across all the tested datasets and resulting in the model that outperforms the previous SOTA Transformers for long-term forecasting.

If you have further concerns regarding our answers, we are more than happy to respond to them during this discussion period. Otherwise, we would really appreciate your acknowledgement that you concerns indeed have been cleared.

Again, we thank you very much for your time and reviews!

Dear Reviewer,

We kindly remind that the discussion period is coming to the end. Thus, please provide us with any concern you may still have. Otherwise, we would really appreciate your acknowledgement that you concerns indeed have been cleared and hope that you consider raising the score for our work, especially given the additional experiments having been added supporting our method.

Again, we thank you very much for your time and and hard work reviewing our paper!

Authors

We thank you the reviewer for all the comments and suggestions, and would like to hereby address your questions. If we could answer all of your concerns, we hop taht you can consider increasing the score for our work.

$**Q1:**$ " “CrossFormer deploys a convoluted architecture, which is isolated from other prevalent Transformers with their own established merits in temporal modeling and specifically designed for only MTS forecasting, thereby lacking flexibility” I am a bit confused, could you explain more in detail this?"

$**Response:**$ We just want to discuss that CrossFormer's architecture is very unconventional from other prevalent Transformer with simple multi-head attention that yet has been shown to be very good in temporal modeling. By deviating from the very base architecture of vanilla Transformer, Crossformer may not utilize such advantages; for your information, Crossformer deploys a Hierarchical Encoder-Decoder architecture with Dimension-Segment-Wise embedding, which devides time into segmants, Two-Stage Attention and segment merging, which merges back the time. We thank you for the suggestion, and will re-write it for more clarity in the updated versions in the future.

$**Q2:**$ "It is confusing how the value of k in equation 6 is defined, do you get the value of c first and then calculate k with the topK operation? It is not very clear to me why not directly take top k, for instance, top 5, lagging value, and use it instead of getting a value k by using the topK operator? Any motivation behind?"

$**Response:**$ We would like to clarify that such $k = c \log(T)$ is indeed the input $k$ to TopK operation, instead of getting a value $k$ out of topK operator. So yes, you would choose $c$ first, which is a hyperparameter, and then use $k = c \log(T)$ as the input to TopK. Also, motivation for doing $k = c \log(T)$ is that even if one uses FFT of $O(T \log(T))$ for fast computation, your final computational cost will still depend on $O(d^2 T \cdot k )$ for summing up $k$ terms in Eq. (6), so setting $k = c \log(T) = O(\log(T))$ would allow you to maintain such $O(T \log(T))$ complexity.

$**Q3:**$ "I am not sure how to go from equation 7 to the result they got in the section below, maye some proof?"

$**Response:**$ We would like to explain it in more details here. First, please note that Eq (7) $S_{X, Y}(f) = F(X_t) F^*(Y_t)$ should be interpreted as $S_{X, Y}(f) = F(X_t)(f) \cdot F^*(Y_t)(f)$. Note that in the two equations in Eq Eq (7), the variable $f$ corresponds to frequency domain and $l$ corresponds to time domain. While we write $S_{X, Y}(f)$ and $(X \star Y)(l)$ in point-wise form as in the paper, which demonstrates the Cross-correlation Theorem and how you can compute $(X \star Y)(l)= \sum_{t=1}^{T} X_{t-l} Y_t$ via FFT and inverse FFT, the actual implementation using Pytorch's FFT library would let you input, for example in case of $d=1$ feature, the vector $X_t$ of size $T$ in time domain (i.e. a vector of all $l$ stacked together), and perform FFT in $O(T \log(T))$ to return you the vector $F(X_t)$ of size $T/2+1$ in frequency domain (i.e. a vector of all $f$ stacked together). From there, you can compute $S_{X, Y}(f) = F(X_t) F^*(Y_t)$, which is again a vector of all $f$ stacked together. From the second equation in Eq (7), you can now feed such vector into the inverse FFT to get the vector $(X \star Y)(l)= F^{-1}( S_{X, Y}(f) )$ of all $l = 0\to T-1$ stacked together. We hope that this can clarify your concern.

$**Q4:**$ "It seems even with FFT, the computational complexity is still quite high for a time series with a large feature dimension."

$**Response:**$ We kindly refer the reviewer to our new experiments on run-time of the baselines in Appendix D of the manuscript. It shows that the overhead incurred by our CAB to the base models is quite minimal, so the resulting models still maintain around the same order of magnitude in computation as other SOTA Transformers. We do agree with you CAB will have another multiplicative factor of $d$ in its complexity, yet CAB has advantage in $T$, which has been usually the real computational bottleneck to be addressed in the literature of Transformers' computational complexities. Further acceleration in $d$ would be interesting future direction.

We thank the reviewer for the comments, and believe that there can be some misunderstanding on our work contribution that we would be pleased to hereby clarify. We really hope that if your concerns can be properly addressed in the rebuttal and the position of our contribution is clarified, you can positively re-evaluate our work. Before giving our detailed answers, we kindly summarize the following points:

• Our architecture has many differences from Autoformer, which itself was very specifically designed to capture auto-correlation in long-term forecasting, in order to capture lagged cross-correlation. To this end, we have provided a paragraph (highlighted in blue) at the end of Section 3.2.2 in the revised manuscript to emphasize the differences.

• We have included the performance of Autoformer in all of our experiments in the revised manuscript. The results confirm that Autoformer is not as competitive as other baselines and our models; this is quite expected as Autoformer was designed for forecasting, and also a reason why we did not include it in our original experiments focusing more on encoder-only tasks.

• For completeness, in Appendix C of the revised manuscript, we also provide new experiments on long-term forecasting, which Autoformer excels in, on three datasets spanning different domains. In these tests, the our Non-stationary+CAB model achieves the best performance among Transformer on two out of three datasets. While Autoformer still has very competitive results, it generally lags behind Fedformer, which is one of the latest and SOTA Transformers for long-term forecasting.

$**Q1:**$ "The novelty of the paper is limited. The proposed correlated attention is basically a extension for Autoformer, which only captures auto-correlation. "; "The design of CAB is based on autoformer. Why is there no comparison between the effects of CAB and autoformer?"

$**Response:**$ We believe that our proposed CAB (correlated attention block) has many differences from the Autoformer, and have provided a paragraph (highlighted in blue) at the end of Section 3.2.2 in the revised manuscript to emphasize the differences. We would like to next re-iterate our discussion in the paper here in our response for your convenience.

Since the CAB aims to capture the lagged cross-correlation,which is relevant to yet more generalized than the auto-correlation module in Autoformer, we believe it is crucial to emphasize the main differences. First, Autoformer overall is a decomposed encoder-decoder architecture proposed for long-term forecasting, so its auto-correlation module is specifically designed to work with series seasonalities extracted from various series decomposition steps of Autoformer. On the other hand, CAB ensures flexibility with any input series representation by deploying normalization step and learnable temperature coefficient $\tau$ reweighting the correlation matrices. Second, while Autoformer computes purely auto-correlation scores and aggregates their exact values for TopK , CAB computes cross-correlation matrices and aggregates the absolute values of such entries for TopK in Equation 5 (as correlation can stem from either positive or negative correlation). Finally, to facilitate robustness to different input series representation, CAB adopts learnable weights $\lambda$ in TopK operation, which balances between auto-correlation and cross-correlation, and $\beta$ in sub-series aggregation, which balances between instantaneous and lagged cross-correlation.