Capturing The Channel Dependency Completely Via Knowledge-Episodic Memory For Time Series Forecasting

Thank you Reviewer a9EC for taking the time to read our paper and giving detailed feedback and questions. Please see below for our response to your specific questions, we hope they fully address any concerns and queries you have, and remain committed to address any further issues.

Q1: Lack of the performance comparison of PatchTST and TimeNet, which are two SOTA baselines for LT-MTS forecasting from ICLR2023.

A1: We would like to thank you for your comment and suggestion to compare with these two models. We provide the following table to compare the performance of the two models on multivariate setting, which is the same as 'response to all reviewers'. It can be seen that our model outperforms these models in most settings. The full efficiency comparison is provided in Table 7 of Appendix B.7.

Q2: No clear statements of the default values of $\gamma_1$ and $\gamma_2$. Although it has the effect of hard example weight in ablation study, we have two hyperparameters, which causes confusion.

A2: We have made the modifications in the article according to your suggestions and highlighted them in blue. $\gamma$ is the hard example weight and $\gamma_1, \gamma_2$ are replaced with $\alpha_1, \alpha_2$ which indicate the balance parameter of two constraints in Loss function.

Q3: It is not convincing that channel dependencies are captured from the visualization of the model in 4.4.

A3: We have made the modifications in the article and highlighted them in blue. The visualization is replaced with a correlation matrix. It is obvious that the similar channels recall the similar patterns which make similar channels have similar prediction.

Q4: In the Recall strategy section, we use m to denote the aggregated knowledge patterns but never used again in other paper’s equations. Does it correspond to the output of Recall(M) ? If yes, better to clear it in the equation 2.

A4: Thank you for your advice. It corresponds to the output of Recall(M), so we have corrected it as you pointed.

Q5: Could you provide more detailed implementation details such as normalization protocols, splitting ratio, optimizer/scheduler etc.

A5: We use Z-score Normalization to normalize the dataset. The splitting ration is set 7:1:2 for train, val and test set on Illness, weather, exchange and Electricity dataset. The splitting ration is set 3:1:2 for train, val and test set on ETTh1, ETTh2, ETTm1 and ETTm2 dataset. We use Adam as the optimizer. The scheduler is used the same as Linear model in their official code. All of these settings are the same as the Linear model and previous methods.

Thank you Reviewer cd83 for taking the time to read our paper and giving detailed feedback and questions. Please see below for our response to your specific questions, we hope they fully address any concerns and queries you have, and remain committed to address any further issues.

Q1: While the proposed approach is interesting I don't think the added complexity justifies the performance improvement over the linear model. From Table 1, the results for SPM-Net are nearly identical to Linear except for the long range prediction, and I suspect that most of them will not pass statistical significance so I don't think this method is ready for publication.

A1: In the paper of Linear model, they use a single Linear layer to predict the future value. However, in our method we use a Linear backbone which consists of a Linear encoder and a Linear decoder. Thus, the results for SPM-Net are nearly identical to Linear except for the long range prediction. We suggest using Table 2 to assess the effectiveness of our approach. We have done robustness checks in Appendix B.6 for your concerns. We also combine our method with PatchTST and conduct tests on various datasets to prove the effectiveness of our method in Appendix B.8 for your concerns. The results prove that our method is effective.

Q2: In the ablation Table 2, why is performance worse than Linear when both memory modules are removed ("w/o both")? I thought that in this case the model would essentially be the same as Linear?

A2: For fair comparison, the results used in Table 1 are replicated from the paper of the Linear model. However, in ablation study the 'without both' means the Linear backbone we used, which is equipped with a Linear encoder and a Linear decoder. In the paper of Linear model, they use a single Linear layer to predict the future value. Thus, the results in the Table 2 may be a little better or worse than the results in Table 1.

Thank you Reviewer D3M5 for taking the time to read our paper and giving detailed feedback and questions. Please see below for our response to your specific questions, we hope they fully address any concerns and queries you have, and remain committed to address any further issues.

About Writing:

Q1: The terminology used in the paper appears to be inappropriate, e.g., 'student-like pattern memory,' 'knowledge pattern memory,' and 'episodic pattern memory.'

A1: We have made the modifications in the article according to your suggestions and highlighted them in blue.

Q2: The word 'completely' in the title is inappropriate as there is a lack of evidence to demonstrate that the proposed model can perfectly capture the complex dependencies. The proposed SPM-Net just introduces two memory modules to aid prediction. Symbols in all equations are not clearly introduced. For example, what are the sizes of W and A in (1)?

A2: About capturing the complex dependencies, we discuss it in 'About model' part Q2 and A2. About other questions, we have made the modifications in the article according to your suggestions and highlighted them in blue.

Q3: All references are cited incorrectly. Most of them should be cited using \citep{}.

A3: We have made the modifications in the article according to your suggestions.

Q4: There are numerous typos and grammar mistakes in the paper.

A4: We have try our best to correct the mistakes in the article according to your suggestions.

About Model

Q1: Details of the combination part before outputting the final prediction results are missing.

A1: We have revised the combination and prediction in equation 2 mentioned in section 3.3. The 'Concat' means the concatenation between different tensors, which is usually implemented by torch.cat() in pytorch.

Q2: It would be beneficial to explain why memory can capture the dependencies and what advantages it has over graph structural learning methods."

A2:We believe that channel dependency arises from the fact that different channels can benefit from relevant information from other channels when predicting their own future results. Traditional CNN and GNN methods can achieve this operation, but our memory network captures channel dependency from different perspectives.

Regarding knowledge memory: typically, different channels share knowledge memory, so the loss generated during prediction considers the influence of information from other channels, thus facilitating information transfer. Because the message passing between different channels is not direct, we name it latent channel dependency, which distinguishes it from graph learning models. The detail of message passing of knowledge memory can be found in Appendix B.1.2.

Regarding episodic memory: episodic memory involves remembering representative channels, and each channel recalls several representative channels most similar to itself during prediction, enabling the intersection of information between the channel and representative channels. The key difference from graph learning models lies in the fact that our episodic memory aggregates information from the K nearest representative channels instead of aggregating all the information like GNN. This approach, to some extent, reduces unnecessary redundancy in the information.

Our memory network achieves the capture of channel dependency by employing two different memory structures to aggregate effective information from different perspectives across channels. Through this approach, each channel considers the information from other channels when predicting its own results, thus capturing channel dependency.

More details can be found in section 3.2 and the 'Effect of Memory update strategy' part of section 4.3. In ablation study, we introduce another data based memory based network to compare with our pattern based memory network, which proves our model can indeed capture channel dependency from distinct perspectives. Because in the data based memory network each channel has its own memory, which means different channels can not share the same pattern.

About Experiments:

Q1: In your released source code, I noticed that in the test set dataloader, you set 'drop_last' to True (batch size=8). However, the Linear (Zeng et al. 2023) paper uses 'drop_last=False' and batch size=32. As you directly use their reported results of Linear for comparisons, there may be some inconsistencies in the experimental setups. The training objective (5) of the memory module (knowledge memory) is basically from the paper by Jiang et al. (2023). Thus, it is recommended to include this spatio-temporal baseline in the experiments. Additionally, it would be beneficial to include more commonly used spatio-temporal datasets in the experiments, such as METR-LA and PEMS-BAY, as suggested by Jiang et al. (2023).

A1: We set batch size=32 in the scripts when we train our model. The default setting in the code is not the final setting for our model. About 'drop_last problem': The results in the paper of Linear model are obtained from 'drop_last=True'. This issue can be found in their github issue(https://github.com/cure-lab/LTSF-Linear/issues/76). For fair comoparison, the previous methods and our model both use the 'drop_last=True' setting. Due to Jiang et al. (2023) focus on traffic forecasting, their model is based on RNN module which takes too much time to forecast long time series. Another reason is that the RNN based model is not good at long-term series forecasting. Considering that, we do not compare it with our model. About spatio time series: We have added a traffic dataset which records the road occupancy rates from different sensors on San Francisco freeways. The result is shown in 'response to all Reviewers'.

Q2: Why choose Linear for comparison, not NLinear or DLinear (Zeng et al. 2023)?

A2: Because we do not use 'revin normalization' or 'trend-seasonal decompose' operation which are proved to be effective to improve the performance of forecasting model, we think use Linear model is a more fair baseline. However, considering the issue you mentioned, we add DLinear model as a baseline in 'response to all reviewers'.

Q3: "Figure 2 is somewhat challenging to read. It would be better to display the correlation matrix found by the memories to demonstrate the channel dependencies.

A3: We have made the modifications in the article according to your suggestions and highlighted them in blue.

About Discussions on channel-independence

Q1: One recent paper, PatchTST, utilizes channel-independence for multivariate time series. Could you provide some insights on the comparisons between the channel-independence and channel-dependency modeling methods?

A1: I think that this article makes sense to some extent. However, they have not provided compelling evidence to demonstrate that reintroducing channel dependency after channel independence through effective means will lead to performance degradation. By combining our method with PatchTST and conducting tests on various datasets, we observed significant improvements. Therefore, we believe that the assumption of channel independence may not be entirely reasonable. Our future work will be dedicated to address this issue completely.

Q2: For some of the datasets in the paper where the units of variables differ, it is worth considering whether dependency modeling is necessary because PatchTST's performance seems to be good on them by channel independence.

A2: We visualize different channels on different datasets and find that most of channels have similar trends as shown in section 4.4. Thus, we consider that the effective way to capture the channel dependency is necessary. Our viewpoint is to use channel independence to eliminate redundant correlation between channels and then effectively blend information between relevant channels using suitable methods, which may be more effective. By combining our method with PatchTST and conducting tests on various datasets, we observed significant improvements. Therefore, we believe that the assumption of channel independence may not be entirely reasonable. Our future work will be dedicated to address this issue completely.

Thank you Reviewer sqDM for taking the time to read our paper and giving detailed feedback and questions. Please see below for our response to your specific questions, we hope they fully address any concerns and queries you have, and remain committed to address any further issues.

Q1: While the paper does a good job of introducing the model architecture and memory modules, more detailed explanations of certain components, such as the initialization of knowledge patterns and the selection process for hard patterns, could further enhance the reader's understanding.

A1: We introduce the initialization of knowledge patterns in the 'storage strategy' of section 3.4. We typically select the top K channels with the highest prediction loss in each batch as hard patterns. For specific details on the update operations, please refer to the 'Update strategy' part in section 3.5. Due to the limited length of the main text, the selection process and update procsee for hard patterns are introduced in Appendix B.1.

Q2: The paper could benefit from more thorough discussions about the generalizability of the proposed SPM-Net across different types of MTS data and its limitations in handling noise or outliers.

A2: The generalizability of the proposed SPM-Net across different types of MTS data are shown in Table 4,5 in Appendix B.3,4. Due to the fact that our model does not specifically address noise and outliers, it does indeed have certain limitations. However, it's worth noting that this issue may also exist in other models, as most current models do not extensively analyze or handle noise and outliers. Our future work will be dedicated to designing models that are robust to noise and outliers, across different types of models.

Q3: It would be valuable to provide a comparative analysis of the computational complexity of the proposed approach compared to existing methods, as it could impact the practicality of the model.

A3: Thank you for your advice. We have added computational complexity of the proposed approach compared to existing methods as shown below: The original Transformer has $O(L^2)$ complexity on both time and space, where L is the number of input length. PatchTST use patch to reduce the L and achieve $O(N^2) (N<L)$ complexity on both time and space, where N is the number of input tokens. Dlinear is a Linear based model, so the complexity on both time and space is $O(L)$. The complexity of our model is caused by the calculation of attention score, which is $O(DM)$(D is the number of channels, M is the memory size). In most cases, D, M is much smaller than L, so the complexity of our model is smaller than the transformer based model.