Temporal Query Network for Efficient Multivariate Time Series Forecasting

Thank you for your detailed and thoughtful review! We must apologize for the concise response earlier due to length limits.

Summary: TQNet combines a single attention layer to handle multi-time-steps interactions and an MLP to handle multi-channel interactions.

Sorry for any unclear points in our previous version. In fact, it is the TQ-enhanced attention layer that handles multi-channel interactions, followed by an MLP that handles multi-time-step interactions. We will improve the clarity of this statement in the revised paper.

Claims: The main evidence which is lacking is how good the model is at using multivariate information.

Thank you for pointing this out. Figure 5 only indirectly demonstrates TQNet's ability to utilize multivariate information but does not directly show how effective it is in leveraging such information. To address this, we have added experiments on several large-scale multivariate datasets. In these experiments, the goal is to predict the target channel using multiple variables, ranging from not utilizing any additional variables to fully leveraging all available variables. The results show that incorporating covariate information significantly improves the prediction accuracy of TQNet for the target channel.(even for Solar dataset) This provides direct evidence that TQNet can effectively utilize multivariate information.

M1. This makes it plausible that the results' quality is solely due to the improved time-wise interactions.

Our goal is to explore a more elegant mechanism for modeling multivariate correlations, for which we designed TQ-MHA. Time-wise interactions are not the primary focus of our study, so we adopted a simple two-layer MLP for temporal dependency modeling, which has been shown in recent studies to be sufficiently capable of capturing temporal dependencies. As shown in Figure 2, TQNet's major improvements are particularly evident in large-scale multivariate datasets.

M2. It is not mentioned whether any of the baselines have also been given this information.

Many baselines have indeed utilized temporal (time) information. For example, CycleNet explicitly models periodic sequences, while iTransformer and TimeXer use timestamp sequences as tokens. In fact, how to leverage this information to build better forecasting models remains an open research question, and TQNet provides an elegant solution in this regard.

M3 (also mentioned in Essential References Not Discussed). Classical and older models are absent from the baselines, such as ETS.

We have added comparison experiments with ETS, which clearly demonstrate the superiority of TQNet. Please refer to this link for the results: https://anonymous.4open.science/r/TQNet-ETS-CD4F.

Other Comments: Please add the hyperparameters used for TQNet.

Yes, we have detailed the hyperparameters of TQNet in Appendix A.2, where they are set relatively consistently accross datasets. The results for other baselines are sourced from their official results or the iTransformer paper to ensure reliability.

Q1 (also mentioned in Supplementary Material): The provided link does not work.

We sincerely apologize for this issue. The anonymous website encountered a bug displaying "The requested file is not found" when trying to browse the code online. To resolve this, you may directly download the source code from the anonymous website or access it through the Supplementary Material on OpenReview. We hope this works for you.

Q2: What is the impact of giving an incorrect W to the model?

In Figure 6, we have systematically explored the impact of incorrect W settings, except for the specific case you mentioned where W is set to a multiple of the full periodicity. Our additional experiments show that when W = 2 × 168, TQNet's performance declines but remains close to that when W = 168. This is primarily because setting W to an integer multiple only reduces the number of training samples allocated to TQ parameters proportionally, without significantly affecting the effectiveness of the TQ mechanism itself. We will add this result to Figure 6.

Q3: Whether interactions between time steps t and t' and those between t and t' + W are identical.

They are not identical. This is because the TQ mechanism only affects the correlation of Q in the attention mechanism, while K and V remain dependent on local samples. Therefore, the fundamental purpose of the TQ mechanism is to provide a globally stable correlation supplement.

Apologies again for the concise text, and thank you again!

Thank you very much for your detailed review!

W1: The paper only provides results for multivariate-to-multivariate forecasting.

We have further supplemented the comparison between TQNet and baseline models in the multivariate-to-univariate forecasting scenario. The results show that TQNet still exhibits a significant advantage in this setting, demonstrating its superior capability in multivariate modeling.

W2 (also mentioned in Suggestions): There are some inconsistencies between Figure 2, Algorithm 1, and their descriptions in the main text.

We appreciate your careful examination and suggestions! We will correct these inconsistencies in the final version.

W3: The paper does not discuss potential limitations, such as cases where there is no periodicity or no significant multivariate dependencies.

Thank you for highlighting this. We will add a discussion of this potential limitations of TQNet in the revised version to provide readers with a clearer understanding of TQNet and identify areas that require further investigation. Specifically, in scenarios where there is no significant periodicity, TQNet may not perform optimally, as its TQ mechanism relies on periodic shift operations. However, long-horizon forecasting in non-periodic settings is inherently challenging. In such cases, incorporating additional prior features may help compensate for the lack of periodic information. We will include these clarifications in the revised paper.

Q1: What happens when a suitable $W$ cannot be found?

Many recent studies have demonstrated that periodicity is a crucial factor for achieving long-horizon forecasting [1]. Moreover, real-world time series data are often influenced by social or natural factors, meaning they typically exhibit at least a daily periodic fluctuation. Therefore, identifying a suitable $W$ is feasible and straightforward. Additionally, the results in Figure 6 indicate that even without leveraging the dataset's periodicity (e.g., setting $W$ to 1), the TQ mechanism can still improve forecasting accuracy. This is because, in such cases, TQ can act as an implicit channel identifier [2], enhancing the model's ability to distinguish between multivariate channels.

[1] Lin S, Lin W, Hu X, et al. Cyclenet: Enhancing time series forecasting through modeling periodic patterns. NeurIPS, 2024. [2] Shao Z, Zhang Z, Wang F, et al. Spatial-temporal identity: A simple yet effective baseline for multivariate time series forecasting. CIKM, 2022.

Q2: Why are the learnable parameters in TQ initialized to zero?

TQ can adaptively learn optimal representations of inter-variable relationships through backpropagation. Therefore, its initialization does not significantly impact the final learned representations. Specifically, we verified this by experimenting with different initialization strategies on the Electricity dataset and found that the performance remained consistent across different initialization methods.

Q3: Why can TQNet achieve nearly the same computational efficiency as DLinear (as shown in Figure 7)?

This is primarily due to two factors: (i) The lightweight design of TQNet, which includes only a single attention layer and a two-layer MLP, maximizing efficiency. (ii) The powerful parallel computing capabilities of modern GPUs. Specifically, the attention computations and MLP structures in TQNet can be highly parallelized, allowing it to achieve computational efficiency comparable to linear models.

Q4: Are the results reported in Table 2 averaged over multiple runs?

Table 5 presents results from multiple runs of TQNet with different random seeds and learning rates, demonstrating its robustness. The results in Table 2 for TQNet are from a single run with a random seed of 2024, following the experimental setup described in Appendix A.2. Notably, these results are consistent with those in Table 5. Furthermore, the baseline results in Table 2 are sourced from their official reports (or reproduced from the iTransformer paper) to ensure reliability. We have clarified this in the caption of Table 4, where the full results are provided.

Thanks again!

Thank you for your insightful review!

Claims And Evidence and Essential References Not Discussed.

Thanks for pointing this out. Indeed, changes in time scales can cause variations in inter-variable correlations, and in fact, Figure 1 of TQNet also demonstrates this. The key differences are:

1. MSGNet's multi-scale approach focuses on different time scales within the look-back window (e.g., variations within 24 and 96-time steps in a window of length 96).
2. TQNet's multi-scale nature manifests in two ways: the local sample look-back window represents short-term scales, while the global sequence (the entire training set) represents long-term scales.

Thus, MSGNet’s findings actually support our claim that considering correlations at different scales is necessary. We will supplement the revised paper with discussions and comparisons with MSGNet. The complete comparison results can be found at this link: https://anonymous.4open.science/r/TQNet-MSGNet-7EBE, which demonstrates the significant advantage of TQNet.

Questions For Authors: How does temporal query technique benefit representation learning of inter-variable relationships? Why can TQ-enhanced MHA mechanism learns a globally consistent and adaptive representation of inter-variable correlations within individual samples? How does it deal with non-stationary disturbances in real-world data? Maybe some theoretical explanation or case discussion will help.

The benefits of the Temporal Query (TQ) technique arise from its learnable nature combined with a periodic shifting mechanism. In the TQ-enhanced MHA, the query $Q$ is generated from globally shared, periodically shifted learnable vectors, while the keys $K$ and values $V$ are directly derived from the raw time series and thus capture more localized correlations. During training, the model is encouraged to maximize the attention output:

$$O = \operatorname{Softmax}\left( \frac{Q_i K_i^\top}{\sqrt{L}} \right) V_i,$$

which implies that for each sample $i$, the learned query $Q_i$ tends to align with the key $K_i$, meaning that it actively incorporates more relevant inter-variable information for accurate forecasting. This alignment can be formulated as:

$$\operatorname{Corr}(Q_i) \approx \operatorname{Corr}(K_i) = \frac{Q_i K_i^\top}{\|Q_i\|\,\|K_i\|},$$

where $\operatorname{Corr}(\cdot)$ represents a normalized measure of correlation between the vectors.

Moreover, since the query $Q_i$ is periodically extracted from the shared learnable parameter $\theta_{\text{TQ}}$ (i.e., multiple samples share the same $Q$ due to the periodic shift), its effective correlation remains consistent across periods and is averaged over multiple local samples.

$$\operatorname{Corr}(Q_i) = \operatorname{Corr}(Q_{i+nW}), \quad n=0,1,\dots,N-1,$$

and

$$\operatorname{Corr}(Q_i) \approx \frac{1}{N} \sum_{n=0}^{N-1} \operatorname{Corr}(K_{i+nW}),$$

where $W$ is the periodic length, $N$ is the number of sampled periods, and $K_{i+nW}$ represents the keys obtained from the raw time series, capturing localized correlations.

Thus, in practice, $Q_i$ serves as an averaged representation of correlations across all samples within the dataset over multiple periods, mitigating the impact of non-stationary disturbances in local samples.)

This also explains why the TQ technique enables each sample's $Q$​ to learn a globally consistent and adaptive representation of inter-variable correlations, addressing the limitation of conventional attention mechanisms, which can only capture localized sample correlations. Since the learned correlations incorporate information from all periodic samples in the training set, they approximate the overall dataset correlation, thereby enhancing the representation learning of inter-variable relationships.

Theoretical Claims.

We hope the above theoretical analysis addresses your concerns. In summary, the TQ technique, through its periodic cyclic sharing mechanism, effectively neutralizes non-stationary disturbances across multiple periods, thereby improving the robustness of the correlation learned in the attention mechanism.

Additionally, Figure 1 serves as a case study demonstrating the effectiveness of the TQ technique. The data is collected from the first sample of the real-world Traffic dataset (where Figure 1d highlights several channels with strong disturbances). It can be observed that even in the presence of noisy disturbances, the correlation of $Q$ generated by the TQ technique is more stable and aligns more closely with the dataset's global correlation. This characteristic enables TQ-MHA to comprehensively consider correlations at different time scales—where $Q$ models the global correlation via cross-period contributions, while $K$ and $V$ model localized correlation with noisy perturbations.

Finally, thank you again for your informative review. We hope our response addresses your concerns.

Thank you for your valuable comments!

C1: TQ is fixed across different samples, whereas per-example correlations differ from global correlations.

This concern is valid in general, but it is effectively addressed within the TQ-MHA mechanism. In TQ-MHA, the learnable shifted TQ serves only as the $Q$, while the raw series acts as the $K$ and $V$. This design means that the correlations among $Q$ emphasize more stable global dependencies, while those among $K$ and $V$ focus on per-example correlations. Through the attention mechanism, the model integrates both global and local dependencies, rather than relying solely on per-example correlations. We will properly revise the original claim to clarify this point.

C2: How does TQ capture inter-variable correlations if there are no explicit constraints?

In fact, TQ introduces two implicit biases that facilitate the learning of inter-variable correlations:

1. Adaptive learning through backpropagation – TQ is inherently designed to learn correlations via backpropagation optimization. As shown in Figure 4 of the paper, TQ effectively models the relative relationships among variables.
2. Periodic shifting mechanism – This ensures that TQ-based learned representations are averaged per period, mitigating the effects of random noise perturbations. Figure 1 demonstrates that the correlations derived from TQ-generated $Q$ are more stable and globally consistent.

C3: TQ seems to violate time-invariance because it assigns fixed ( C × L ) vectors to each timestamp.

In fact, only sample on timestamps $t$ and $t+W$ share the same $Q$, since $Q$ ($C×L$) is extracted from $\theta_{TQ}$ ($C×W$) via a periodic shifting mechanism (with a period of $W$ timesteps). However, the remaining samples within the interval $[t, t+W-1]$, the $Q$ are different across timestamps (manifesting as a shift along the time axis). Therefore, this setting does not violate time-invariance. On the contrary, by explicitly considering periodic fluctuations, it better aligns with the inherent biases of real-world time series.

W1: There is no clearly way to easily and automatically determine determine W given a specific dataset..

There are many simple methods to clearly determine the hyperparameter $W$. On one hand, real-world time series data are often influenced by social or natural factors, meaning they typically exhibit at least a daily or weekly periodic fluctuation. Therefore, considering the sampling interval of the data (e.g., 15 min), it is easy to infer the potential periodic length. On the other hand, the autocorrelation function (ACF) serves as an effective mathematical tool to verify the periodic length, where peaks in the ACF correspond to potential periodic lengths. Therefore, adjusting and selecting $W$ is even easier than tuning the learning rate in the practical training scenarios. We will provide a more detailed explanation of how to determine $W$ in the revised paper, along with a script tool (utilizing ACF) in the open-source code to facilitate direct usage by users.

W2: It might overlook the cases where there are multiple periodic lengths.

Thanks for pointing this out. This issue can be discussed in two cases:

1. When multiple periodicities overlap, it can be easily handled. For example, the Traffic dataset exhibits both daily (24-hour) and weekly (7×24-hour) periodic patterns. In this case, considering only the longest periodicity (weekly) is sufficient, as it inherently encompasses the shorter daily cycle. In fact, most real-world scenarios fall into this category, so this does not pose a significant challenge for the practical application of TQNet.
2. When multiple periodicities are irregularly interwoven, such as weekly (7×24-hour) and monthly (30×24-hour) cycles, it introduces some challenges for TQNet. Simply considering the longest periodicity cannot precisely capture the overlapping weekly patterns. In such cases, a practical compromise is to focus only on the daily periodicity (24-hour). Additionally, an alternative approach is to integrate multiple TQNet models, each configured with different $W$ values, to enhance adaptability in such scenarios. Overall, fully addressing this issue remains an open research question, even for existing models. We will include a more detailed discussion in the revised paper to provide readers with a clearer understanding of TQNet and suggest directions for further investigation.

Suggestion to clearly put the message in the Figure 1 caption.

Thanks for your suggestion! We will add a clearer message to the caption of Figure 1.

Finally, sorry about the concise response due to length limits. For any unclear points, we are happy to provide further clarification in the next stage. Thanks again!