ENHANCING MULTIVARIATE TIME SERIES FORECAST- ING WITH MUTUAL INFORMATION-DRIVEN CROSS- VARIABLE AND TEMPORAL MODELING

Q4: The term "single-step forecaster" can be confusing. I think it in fact outcomes the "multi-step" prediction. And it seems $\lambda=1$ is favored in most cases (even though experiments in Section 4.3 are not extensive). How to support the motivation of Equation 12?

A4: Thanks for the valuable question. We will answer this question in two parts:

1. The term "single-step forecaster" can be confusing We appreciate your comment regarding the term "single-step forecaster." To clarify, a single-step forecaster refers to a non-autoregressive model that generates multi-step predictions in a single step. In contrast, autoregressive forecasters predict one time step at a time, which can lead to error accumulation issues. While single-step forecasters alleviate this problem, they lack the ability to model the correlation between future time steps effectively.

2. And it seems $\lambda=1$ is favored in most cases (even though experiments in Section 4.3 are not extensive). How to support the motivation of Equation 12? We acknowledge the reviewer's observation that $\lambda=1$ appears to be favored in most cases, even though the experiments in Section 4.3 were not extensive. In response to this concern, we have conducted additional experiments and revised Figure 4 (modify a mistake in the orginal version) accordingly to provide a more comprehensive analysis of the impact of different $\lambda$.

The updated results presented in Figure 4 clearly demonstrate that the models' performance shows minimal improvement when $\lambda \geq 0.8$. In fact, we observed that when $\lambda$ was set to 0.8, the model's performance was slightly better than with $\lambda=1.0$. Similar trends were observed in other datasets as well, further reinforcing the notion that deviating from $\lambda=1$ does not significantly affect the model's performance.

We appreciate the valuable feedback provided by the reviewer and have incorporated it into our revised submission. Thank you for your question.

We appreciate the reviewer's comments. We update our paper and address your concerns here:

Q1: The experiments are not extensive. It seems there are some cherry-pickings in particular configurations. How do the performance of ETTh2 and PEMS in Table 1? Some experiments (Table 2,3,4 and Figure 4) only use the results on partial datasets and baseline models. The prediction visualization of Figure 1 (comparing Informer and PatchTST) seems quite different from the reported results in the PatchTST paper. I hope the authors can elaborate more on it.

A1: Thanks you for the detailed comments, we will answer the question in three parts:

1. How do the performance of ETTh2 and PEMS in Table 1?

Great suggestions. We have made revisions to the manuscript accordingly. First, we conducted extensive experiments to evaluate the effectiveness of InfoTime on the ETTh2 dataset. The results, shown in the updated Table below (also Table 2 and Table 6 of our manuscript), demonstrate that InfoTime can clearly improve the performance of base models on the ETTh2 dataset.

Moreover, we have made further improvements to Table 3 in order to better illustrate the performance improvement of InfoTime on datasets with clear correlations between channels. Following the advice from the reviewer, we have included the experimental results of Stationary and Crossformer in the revised manuscript. With these additions, the revised Table 3 allows for a more comprehensive evaluation of the performance of the models on the PEMS datasets, which are known to have clear correlations between channels.

Furthermore, we have added the ablation results of the aforementioned base models in Table 9. This provides insights into the contribution of InfoTime in improving the performance of both Channel-Independence models and Channel-mixing models.

Based on these experimental results, it can be observed that in datasets such as the PEMS datasets, where there is a clear correlation between variables, Channel-Independence models generally perform worse than Channel-mixing models. However, InfoTime is able to enhance the performance of both types of models.

2. Some experiments only use the results on partial datasets and baseline models We add the full experimental results of the Channel-Independence models in Table 6, and we also add the ablation results for all base models used in this paper in Table 7 and 8. We believe that these extensive experimental results are more convincing. We hope that the added experiments can help you to better understand InfoTime.

3. The prediction visualization of Figure 1 (comparing Informer and PatchTST) seems quite different from the reported results in the PatchTST paper.

We believe there may be some misunderstanding here. The dataset used in Figure 1 for prediction visualization is the PEMS08 dataset, which is adopted from SCINet. However, we do not find an evidence that the PatchTST paper has reported results on the PEMS08 dataset. In fact, the PatchTST paper reported results on the traffic dataset which is linked to http://pems.dot.ca.gov. Although PEMS08 and the traffic dataset may share some correlation, they are indeed different datasets. We hope the reviewer provide more details on this issue.

Q4: Page 6: How is generated with a single step forecaster? Does that require an initial training of the neural network, or is this a different neural network that is trained before the actual network? Without an initial neural network, how can we obtain $\hat{Y}$ and train using Eq. (13)?

A4: Thank you for your insightful question. In our work, the single-step forecaster is a neural network, typically a fully-connected network, that utilizes the historical data of the i-th channel $X^{i}$ and the cross-variable feature $Z^{i}$ to generate $\hat{Y^{i}}$.

In contrast to separate pre-training, we directly optimize the single-step forecaster along with the other neural networks in our model. This approach allows us to jointly train all the networks and optimize their parameters simultaneously.

Thanks for your suggestion once again. We have taken your suggestions into account and made significant improvements to our description and analysis of the TAM in the revised version.

I hope this revised answer captures the essence of your intended response.

We would like to sincerely thank Reviewer b8jU for providing a detailed and insightful suggestions.

Q1:What is the purpose of Eq. (2)? Since the unconstrained form is used, I think that this discussion could start from Eq. (3).

A1: Thank you for your insightful suggestions and feedback on our work. We appreciate your suggestions and would like to address your question regarding the inclusion of Equation (2) in our paper.

Equation (2) in our work is derived from the paper ["Deep Variational Information Bottleneck"] (https://arxiv.org/pdf/1612.00410.pdf). This equation represents the unconstrained form of the optimization problem, where the objective is to maximize the mutual information between the inputs $Y^{i},X^{i}$ and the latent variables $Z^{i}$. The aim is to capture as much relevant information as possible from the inputs of other channels.

However, we acknowledge your suggestion to start the discussion from Equation (3) instead. Equation (3) introduces the information bottleneck regularizer, which provides a more detailed understanding of the trade-off between extracting relevant information and preserving unnecessary information.

In light of your feedback, we agree that it would be beneficial to remove Equation (2) and start the discussion directly from Equation (3) in future revisions of the paper.

Thank you once again for your valuable suggestions. We appreciate your time and effort in reviewing our work.

Q2: The objective function approximation in Eq. (8) seems to be a bit ad hoc. It is not an upper bound on the entire objective function, but rather a combination of individual approximations. Discussion on this choice were not given in the paper. Is there some reason that makes approximation of the entire function prohibitively difficult? My main concern is that if these bounds are not tight the approximation could be way off.

A2: We would like to thank you for the valuable feedback on our work. Regarding the objective function approximation, we acknowledge that the method we proposed may appear ad hoc at first glance. However, we would like to emphasize that the approach we employed for the objective function was carefully designed based on comprehensive analysis and extensive experimentation. In our work, the objective is to maximize $R_{IB}=\frac{1}{C}\sum_i R^{i}_{IB}=\frac{1}{C}\sum_i [ I(Y^{i},X^{i};Z^{i})-\beta I(X^{o};Z^{i})] , i \in 1,2,...,C$

By utilizing Equation (4), we can modify the objective function as follows: $\sum_{i} R_{IB}^{i} =\sum_{i}[ E_{p(z^{i},y^{i},x^{i})} \left[ \log p(y^{i}|x^{i},z^{i})\right]+E_{p(z^{i},x^{i})} \left [ \log p(x^{i}|z^{i}) \right] + H(Y^{i},X^{i}) -\beta I(X^{o};Z^{i})], i \in 1,2,...C$
Since $H(Y^{i},X^{i})\geq 0$, we have $\sum_{i} R_{IB}^{i} \geq \sum_{i=1}[ E_{p(z^{i},y^{i},x^{i})} \left[ \log p(y^{i}|x^{i},z^{i})\right]+E_{p(z^{i},x^{i})} \left [ \log p(x^{i}|z^{i}) \right] -\beta I(X^{o};Z^{i})], i \in 1,2,...C$. By introducing the variational lower bound $I_{v}(X^{i},Y^{i};Z^{i})$ and upper bound $I_{vCLUB-S}(X^{o};Z^{i})$, we obtain $R_{IB}=\sum_{i} R_{IB}^{i} \geq \sum_{i}[ I_{v}(X^{i},Y^{i};Z^{i})-\beta I_{vCLUB-S}(X^{o};Z^{i})], i \in 1,2,...,C$. Therefore, $\sum_{i=1}^{C}[ I_{v}(X^{i},Y^{i};Z^{i})-\beta I_{vCLUB-S}(X^{o};Z^{i})]$ serves as the lower bound for $R_{IB}$. Consequently, we can maximize $R_{IB}$ by minimizing $L_{IB}=-\sum_{i=1}^{C}[ I_{v}(X^{i},Y^{i};Z^{i})-\beta I_{vCLUB-S}(X^{o};Z^{i})]$.

We hope that this clarification provides a better understanding of the rationale behind our objective function approximation. Through extensive experimentation and evaluation, we have observed that minimizing $L_{IB}$ effectively maximizes $R_{IB}$, leading to competitive results.

Q3: Page 6, first paragraph: Is it CDAM or TAM?

A3: Thanks for your carefully checking, it's TAM, and we have modified it.

Q3: How does the time complexity of CDAM, TAM, and InfoTime compare with the original base models - particularly as the time series get longer

A3: Thank you for the insightful question. In our experiments on the Informer model and ETTh1 dataset, we tested the running time (in seconds) of each epoch for both the base models and InfoTime. By default, we set N=4. Upon analyzing the results, we found that TAM is relatively time-consuming, while CDAM is less consuming. Additionally, we observed that InfoTime does not introduce additional time consumption as the length increases. Although InfoTime requires more time to train, we firmly believe that it is valuable in enhancing the performance of the base models. It is worth noting that future research on how to make InfoTime more lightweight is a meaningful and interesting area of exploration. By optimizing its time complexity, InfoTime has the potential to become even more efficient and practical for forecasting longer time series. Thank you again for bringing up this important point.

Q4: The paper introduces a trade-off parameter B in CDAM to control the balance between retaining important information and eliminating irrelevant information from the latent representation. How could this parameter be determined or optimized in practice? Are there any insights into choosing an appropriate value for B based on experimental results or theoretical considerations?

A4: Thanks for your question. The determination and optimization of the trade-off parameter B in CDAM is an important aspect of leveraging the balance between retaining important information and eliminating irrelevant information from the latent representation. In fact, we consider $\beta$ as a hyper-parameter that needs to be adjusted manually and remains constant during training. While it would be exciting to have the model automatically adjust $\beta$, which is not explored in our work. However, there have been other research works that propose dynamically adjusting $\beta$ during training, such as Generating Sentences from a Continuous Space, Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing, ControlVAE: Controllable Variational Autoencoder.

In our experiments, we find that a large value of $\beta$ ($\beta \geq 1e3$ usually) is needed to prevent overfitting and improve performance, as shown in Figure 4 of our paper. This observation suggests that deep-learning-based models have sufficient capacity to capture historical features, and it is crucial to focus on preventing the model from learning irrelevant features.

Q5: The paper introduces a variational lower bound ($I_{v}(X^{i}, Y^{i}; Z^{i})$) for the mutual information term in CDAM. Could you explain the practical implications of maximizing this lower bound during training? How does optimizing this lower bound improve the model's ability to capture cross-variable dependencies, and what trade-offs or challenges may arise in the optimization process?

A5: Thanks for your question, we will answer your question in three parts.

1. The variational lower bound $I_{v}(X^{i}, Y^{i}; Z^{i})$ is equivalent to $E_{p(z^{i},y^{i},x^{i})}[p_{\theta}(y^{i}|x^{i},z^{i})]$+$E_{p(z^{i},x^{i})}[p_{\theta}(x^{i}|z^{i})]$, the first term $E_{p(z^{i},y^{i},x^{i})}[p_{\theta}(y^{i}|x^{i},z^{i})]$ is the negative log-likelihood of the prediction of $Y^{i}$ given $Z^{i}$ and $X^{i}$, and the second term aims to reconstruction of $X^{i}$ given $Z^{i}$. Therefore, maximizing the variational lower bound can help the model to extract sufficient information of $X^{i}$ while ensuring prediction performance.

2. To improve the model's ability to capture cross-variable dependencies, it is crucial to simultaneously maximize the variational lower bound $I_{v}(X^{i}, Y^{i}; Z^{i})$ and minimize the upper bound $I_{vCLUB-S}(X^{o};Z^{i})$. Minimizing the variational upper bound reduces the mutual information between $Z^{i}$ and $X^{o}$, while maximizing the variational lower bound requires $Z^{i}$ to contain sufficient information for reconstructing $X^{i}$ and predicting $Y^{i}$. By striking a balance between these two objectives, CDAM can extract the most relevant information from $X^{o}$ while eliminating irrelevant information, thereby enhancing its ability to capture cross-variable dependencies.

3. However, optimizing CDAM presents a challenge in selecting the appropriate value for the trade-off parameter $\beta$. Although experiments have shown that a larger $\beta$ can improve model performance, it also leads to a reduction in the extraction of cross-variable features. Therefore, determining the optimal value for $\beta$ requires further exploration to strike the right balance between prediction performance and cross-variable feature extraction.

Q6: Additionally on the upper bound, the authors state that they choose to adopt the sampled vCLUB and minimize $I_{vCLUB-S}(X^{o}; Z^{i})$. Could they expand on why this sampling strategy was chosen and what the actual derived upper bound might be?

A6: Thank you for your question. In comparison to the VUB and L1OUT methods, vCLUB demonstrates a superior ability to approximate the upper bound. Consequently, we have opted to utilize vCLUB as the variational upper bound in our paper. Additionally, we have found that vCLUBSample offers greater efficiency than vCLUB. Hence, we have chosen to employ vCLUBSample in our study.

Q1:The experimentation work is not detailed. There is a lot of variation in results of benchmarks (ETTh1, ETTm1, ETTm2, Weather, Traffic, Electricity) across various papers. Have the authors considered running 5-fold cross validation on these benchmarks to better understand the error bounds around performance claims and then running tests to evaluate the statistical significance of the improvements? There are a number of factors that are unclear: the reported informer MSE and MAE on ETTh1, ETTm1, ETTm2 for example are quite a bit higher than in the original Informer, Non-stationary Transformer, and Crossformer papers (>1 for O=720 Informer by authors vs <0.3 in original paper). Perhaps more discussion of the training setup and how it might differ from past benchmarks and why it shows a much higher error could help separate out the actual performance gains given by InfoTime.

A1: Thank you for your question. We would like to provide further clarification on our experimental work. Firstly, we ensured the reliability and consistency of our results by repeating each experiment three times and reporting the average as the final result. Additionally, we have included the experimental settings of the base models in Appendix A.1, which provides transparency and allows for a better understanding of our methodology. To maintain a fair comparison, we used the same parameter settings as the Informer, Non-stationary Transformer, and Crossformer papers. The experimental scripts we used were obtained from the respective repositories DLinear, Non-stationary Transformer, and Crossformer,. However, we did make a modification by setting the input length $I$ to 96, which may differ from the experimental settings in the original papers. Additionally, for PatchTST and RMLP, we set the input length to 336, as longer input lengths tend to yield better performance for Channel-Independence models. For PEMS03, PEMS04, and PEMS08 datasets, we set $I=336$ for all of these models since all of them perform better in a longer input length.

Regarding your concern about the Informer results, we would appreciate it if you could specify the dataset you are referring to. We have thoroughly checked the results of Informer and have not identified any issues. It is worth noting that our experiments focus on multivariate forecasting rather than univariate forecasting, which may impact the comparison with the original Informer results.

I hope this revised response addresses your concerns more clearly. If you have any further questions or require additional clarification, please feel free to ask.

Q2: Given the large tables of evaluations, it would be helpful to introduce some visual aides to help with understanding the comparisons, perhaps showing the % improvement over other methods for an aggregation of all benchmarks.

A2: Thanks for your meaningful suggestion. We appreciate your feedback, and we have taken it into consideration. In response, we have made updates to the evaluation by including the average Mean Squared Error (MSE) and Mean Absolute Error (MAE) for four different prediction lengths, as well as the relative reduction of MSE and MAE. You can find this information in Table 1 and Table 6. We hope that these additions will provide a clearer understanding of the effectiveness of InfoTime. If you have any further questions or suggestions, please feel free to let us know. We value your input and strive to continuously improve our work.

Q7:More comparison with statistical forecasting methods or non-transformer methods, either empirically or theoretically, could help introduce the benefits of InfoTime in a more thorough manner. While Zeng et. al (2022) and other methods are briefly discussed for their direct forecasting strategy; it could be helpful to compare InfoTime on the DLinear model to see how CAM varies from the DMS approach. Perhaps this is why RMLP was introduced, but it’s not clear what motivated the construction of RMLP and if differs at all from previous approaches which could be more suitable baseline models. While the benchmarks used are standard for transformer-based evaluation, there are other baselines that could shed more light on the characteristics of the model and whether it may help resolve some of the difficulties with transformer approaches to forecasting. Could the authors shed any light on if they considered benchmarks like M3, M4 and what might be missing from the current evaluations?

A7: Thank you for your valuable feedback and suggestions. The DMS method introduced by Zeng et al. is consistant with the 'single-step forecaster' in our paper which means that predict multi-steps in a single step. AS disscussed in our work, the single-step forecaster or DMS assumes that the future time steps are independent of each other, therefore, they have the following formulation: $p(y^{i}|z^{i},x^{i})=\prod_{j=1}^{P}p(y^{i}_{j}|z^{i},x^{i})$. However, it is important to note that future time steps are not truly independent, and although the single-step forecaster can effectively alleviate the error accumulation problem, it cannot model the correlation between future sequences, which is why we propose the TAM module to address this limitation.

In our paper, we propose the linear model RMLP, which outperforms DLinear. As a result, we chose to use RMLP as the base linear model for comparison. Additionally, we also evaluated the effectiveness of InfoTime on the DLinear (linear model), TSMixer (linear model), and TimesNet (linear model) and included the Solar-energy dataset in our analysis. The results, as shown in the table below, demonstrate that InfoTime consistently improves the performance of DLinear, TSMixer, and TimesNet across different datasets. This highlights the efficacy of InfoTime in enhancing the forecasting capabilities of the Linear models and CNN-based models. Moreover, we appreciate your suggestion to include more benchmarks such as M3 and M4. We found that the M3 and M4 datasets are very different from the datasets used in our work. It is very meaningful to verify the effect of our model on these datasets in our future revision.