Channel-wise Influence: Estimating Data Influence for Multivariate Time Series

We appreciate your time to provide valuable comments and suggestions to improve our paper substantially. Before we begin addressing your questions, we would like to first clarify the primary contributions of our paper. Influence functions have demonstrated significant performance and value across various fields[4]-[10], with notable applications in outlier detection[7][8][9][10] and data pruning[4][5]. However, there is no preceding research on the influence functions of multivariate time series to shed light on the effects of modifying the channel of multivariate time series. To fill this gap, we propose a channel-wise influence function, which is the first data-centric method that can estimate the influence of different channels in multivariate time series. We conducted extensive experiments and found that the original influence function performed poorly in anomaly detection and could not facilitate channel pruning, underscoring the superiority and necessity of our approach. Additionally, we can further analyze the information learned by the model through the channel-wise influence matrix. For example, as mentioned in Section 5.2.1 of our paper, comparing the number of core channel subsets across different models reveals each model's capacity for capturing channel dependencies. A larger core subset indicates that the model has captured more effective information, which also highlights the interpretability of our approach.

Regarding the selection of tasks to verify our method: Anomaly detection has long been a critical issue in multivariate time series analysis, with relevant studies including[1][2]. Data pruning is equally important as it raises the question of whether we can train a high-performing model with less data than predicted by scaling laws[3], with related work found in studies such as [3][4][5]. However, data pruning has not yet been extensively studied within the context of time series. By analyzing the characteristics of multivariate time series, we identified redundancy between channels and developed a channel pruning method based on our proposed channel-wise influence function, which outperforms traditional data pruning approaches. In addition, this method also supports the interpretability analysis of the model's ability to model multivariate time series.

[1]Position paper: Quo vadis, unsupervised time series anomaly detection? ICML 2024

[2]Anomaly transformer: Time series anomaly detection with association discrepancy. ICLR 2022

[3]Beyond neural scaling laws: beating power law scaling via data pruning Neurips 2022

[4]Data Pruning via Moving-one-Sample-out Neurips 2023

[5]Dataset Pruning: Reducing Training Data by Examining Generalization Influence ICLR 2023

[6]Self-influence guided data reweighting for language model pre-training. ACL 2023

[7]Detecting adversarial samples using influence functions and nearest neighbors. CVPR 2020

[8]Estimating training data influence by tracing gradient descent. Neurips 2020

[9]Understanding Black-box Predictions via Influence Functions. ICML 2017

[10]Resolving Training Biases via Influence-based Data Relabeling. ICLR 2022

Q1: The paper primarily focuses on anomaly detection and forecasting, leaving the application to other relevant MTS tasks, such as classification, clustering, or imputation, unexplored. This limited scope restricts the paper's ability to fully demonstrate the potential of the proposed method in diverse MTS applications.

A1： Considering that the contribution of our paper lies in introducing a data-centric approach to evaluate channel-wise influence for multivariate time series, it is not feasible to validate our method across all time series methods. Extending our approach to other application tasks is a direction for future research, which we have acknowledged in the limitations section of the paper. Respectly, we do not view this as a weakness, as we have thoroughly demonstrated the effectiveness of our data-centric method in two representative time series tasks.

Q2: The method relies on gradient computations, which can become computationally demanding for complex models, particularly when applied to large-scale MTS datasets. To address this, the paper proposes using gradients from a subset of model parameters to improve efficiency. However, a more detailed analysis of the trade-off between computational cost and performance when employing a reduced set of gradients is warranted.

A2： In Section 5.1.3 of our paper, Parameter Analysis, we thoroughly discussed the impact of the number of gradient parameters used in the computation across three different datasets and found no significant differences. Additionally, other studies on influence functions have also addressed this trade-off, as referenced in [1], [2], and [3]. These works have highlighted that using only a subset of the model’s parameters can still achieve excellent results. Therefore, we believe that accelerating the computation of influence by calculating gradients for only a portion of the parameters is a reasonable and effective approach.

Q3: The paper employs an equidistant sampling strategy to select channels based on their ranked self-influence scores. This approach may introduce biases, particularly when the distribution of influence scores is uneven. For instance, if a large number of channels have similar influence scores, the equidistant sampling might lead to the exclusion of potentially informative channels simply because they are clustered together in the ranking.

A3： Thank you very much for your suggestion. We have also considered a similar issue, and a detailed response can be found in Response to Questions, Q5.

[1]Dataset Pruning: Reducing Training Data by Examining Generalization Influence ICLR 2023

[2]Self-influence guided data reweighting for language model pre-training. ACL 2023

[3]Estimating training data influence by tracing gradient descent. Neurips 2020

We appreciate your time to provide valuable comments and suggestions to improve our paper substantially. Before we begin addressing your questions, we would like to first clarify the primary contributions of our paper. Influence functions have demonstrated significant performance and value across various fields[4]-[10], with notable applications in outlier detection[7][8][9][10] and data pruning[4][5]. However, there is no preceding research on the influence functions of multivariate time series to shed light on the effects of modifying the channel of multivariate time series. To fill this gap, we propose a channel-wise influence function, which is the first data-centric method that can estimate the influence of different channels in multivariate time series. We conducted extensive experiments and found that the original influence function performed poorly in anomaly detection and could not facilitate channel pruning, underscoring the superiority and necessity of our approach. Additionally, we can further analyze the information learned by the model through the channel-wise influence matrix. For example, as mentioned in Section 5.2.1 of our paper, comparing the number of core channel subsets across different models reveals each model's capacity for capturing channel dependencies. A larger core subset indicates that the model has captured more effective information, which also highlights the interpretability of our approach.

Regarding the selection of tasks to verify our method: Anomaly detection has long been a critical issue in multivariate time series analysis, with relevant studies including[1][2]. Data pruning is equally important as it raises the question of whether we can train a high-performing model with less data than predicted by scaling laws[3], with related work found in studies such as [3][4][5]. However, data pruning has not yet been extensively studied within the context of time series. By analyzing the characteristics of multivariate time series, we identified redundancy between channels and developed a channel pruning method based on our proposed channel-wise influence function, which outperforms traditional data pruning approaches. In addition, this method also supports the interpretability analysis of the model's ability to model multivariate time series.

[1]Position paper: Quo vadis, unsupervised time series anomaly detection? ICML 2024

[2]Anomaly transformer: Time series anomaly detection with association discrepancy. ICLR 2022

[3]Beyond neural scaling laws: beating power law scaling via data pruning Neurips 2022

[4]Data Pruning via Moving-one-Sample-out Neurips 2023

[5]Dataset Pruning: Reducing Training Data by Examining Generalization Influence ICLR 2023

[6]Self-influence guided data reweighting for language model pre-training. ACL 2023

[7]Detecting adversarial samples using influence functions and nearest neighbors. CVPR 2020

[8]Estimating training data influence by tracing gradient descent. Neurips 2020

[9]Understanding Black-box Predictions via Influence Functions. ICML 2017

[10]Resolving Training Biases via Influence-based Data Relabeling. ICLR 2022

To better highlight the effectiveness of our method, we compared it with the approach proposed in the mentioned paper[1], referred to as NFS. The specific results are as follows:

From the results shown in the table, it is evident that our method is more effective. According to the method described in the paper[1], this approach introduces additional network parameters to evaluate the importance of different channels. Furthermore, the number of additional parameters required by this method scales with the number of channels, significantly increasing its computational time. Specifically, while the original iTransformer takes only 17 seconds to train one epoch on the ECL dataset, this method increases the time to 32 seconds per epoch.

I appreciate the author's efforts to address my previous concerns. However, a few issues remain:

1. The revised version does not appear to include the additional experiments. For example those utilizing the ETTh1, ETTm1 datasets, and the DLinear model, etc.

2. Regarding baselines comparison, I think there might be some relevant methods to compare to such those using either channel independence (CI) or channel dependence (CD) for time series analysis. Following are some examples that might be relevant:

[1] Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators. ICLR 2024

[2] Feature Selection for Multivariate Time Series via Network Pruning. https://arxiv.org/pdf/2102.06024

3. My previous request regarding computational complexity remains open. I require an analysis of the proposed method's computational complexity, particularly its scalability with high-dimensional data and the trade-off between this complexity and the achieved performance gains and not the complexity for a single channel.

4. Regarding your answer to Q6: The observation that a simple MLP model with only one layer outperforms a Transformer model raises concerns about the chosen datasets. It is possible that the datasets used might lack the complexity typically observed in real-world multivariate time series (MTS) data, where intricate temporal dynamics are prevalent.

However, I believe a rating of 5 is more appropriate. I would like to adjust my previous score of 3 accordingly.

Q2: The experimental results are not very impressive. In Table 2, we can observe that by using of proposed influence the improvement is not very significant. And also the datasets used in this paper are not enough.

A2: Our method achieved a 7% improvement on the SMD dataset and a 15% improvement on the SMAP dataset, along with varying degrees of improvement on other anomaly detection datasets, demonstrating its effectiveness. For the time-series anomaly detection task, we utilized five real-world datasets, while for the time-series forecasting task, we employed three real-world datasets. Additionally, we added experiments on the ETTh1 and ETTm1 datasets, as shown in the table below. Overall, we believe our use of a relatively large number of datasets in time-series research highlights the generalizability and effectiveness of our approach.

Since the original number of channels in ETTh1 and ETTm1 is only 7, the horizontal axis in the table directly represents the number of retained channels.

The results in the table demonstrate the effectiveness of channel pruning based on the channel-wise influence function, highlighting that PatchTST and iTransformer exhibit comparable utilization of channel information on the ETTh1 and ETTm1 datasets.

Q3: The time complexity of the proposed method is not analyzed in this paper.

A3: We have added an experiment measuring the time required for detection at each time point to demonstrate the complexity of our approach, as shown in the table below:

The results in the table indicate that our detection speed is at the millisecond level, which is acceptable for real-world scenarios.

Q4: The code of this paper is not provided.

A4: Since ICLR does not mandate code submission, we have not uploaded it at this time. However, if the paper is accepted, we commit to making our code publicly available.

We appreciate your time to provide valuable comments and suggestions to improve our paper substantially.

Q1: The technical contribution of this paper is not very high. Only the influence function is proposed in MTS, which has been well-studied in other domains. A1: Before we begin addressing your questions, we would like to first clarify the primary contributions of our paper. Influence functions have demonstrated significant performance and value across various fields[4]-[10], with notable applications in outlier detection[7][8][9][10] and data pruning[4][5]. However, there is no preceding research on the influence functions of multivariate time series to shed light on the effects of modifying the channel of multivariate time series. To fill this gap, we propose a channel-wise influence function, which is the first data-centric method that can estimate the influence of different channels in multivariate time series. We conducted extensive experiments and found that the original influence function performed poorly in anomaly detection and could not facilitate channel pruning, underscoring the superiority and necessity of our approach. Additionally, we can further analyze the information learned by the model through the channel-wise influence matrix. For example, as mentioned in Section 5.2.1 of our paper, comparing the number of core channel subsets across different models reveals each model's capacity for capturing channel dependencies. A larger core subset indicates that the model has captured more effective information, which also highlights the interpretability of our approach.

Regarding the selection of tasks to verify our method: Anomaly detection has long been a critical issue in multivariate time series analysis, with relevant studies including[1][2]. Data pruning is equally important as it raises the question of whether we can train a high-performing model with less data than predicted by scaling laws[3], with related work found in studies such as [3][4][5]. However, data pruning has not yet been extensively studied within the context of time series. By analyzing the characteristics of multivariate time series, we identified redundancy between channels and developed a channel pruning method based on our proposed channel-wise influence function, which outperforms traditional data pruning approaches. In addition, this method also supports the interpretability analysis of the model's ability to model multivariate time series.

[1]Position paper: Quo vadis, unsupervised time series anomaly detection? ICML 2024

[2]Anomaly transformer: Time series anomaly detection with association discrepancy. ICLR 2022

[3]Beyond neural scaling laws: beating power law scaling via data pruning Neurips 2022

[4]Data Pruning via Moving-one-Sample-out Neurips 2023

[5]Dataset Pruning: Reducing Training Data by Examining Generalization Influence ICLR 2023

[6]Self-influence guided data reweighting for language model pre-training. ACL 2023

[7]Detecting adversarial samples using influence functions and nearest neighbors. CVPR 2020

[8]Estimating training data influence by tracing gradient descent. Neurips 2020

[9]Understanding Black-box Predictions via Influence Functions. ICML 2017

[10]Resolving Training Biases via Influence-based Data Relabeling. ICLR 2022

Thanks for the author to answer my question. My previous concerns are partially solved, and I would like to increase my score to positive.

Thank you for your response. If you have any new questions, we would like to have a further discussion with you.

Q6: Insufficient Comparison with Baseline Models: The paper lacks comparison with enough baseline models, especially current mainstream state-of-the-art (SOTA) methods. This limitation makes it difficult to fully assess the proposed method's effectiveness relative to existing approaches, thus limiting the demonstration of its advantages. For instance, in time series forecasting, only PatchTST and iTransformer were used for comparison, while other competitive models like GPHT, SimMTM, TSMixer, TimesNet, and DLinear were omitted.

A6： Thank you for your suggestion. Considering that the primary purpose of our method is to accelerate training and better interpret the model's behavior, we did not include comparisons with a wide range of time series forecasting methods. However, given that DLinear is a highly representative approach, we have added it as a supplementary method and tested it under our original setting. The results are as follows:

The experimental results in the table show that the core channel subset of DLinear is less than 5%, which highlights the limited ability of simple linear models to utilize information from different channels effectively.

Q7: Additionally, while the authors designed a CHANNEL PRUNING experiment, it would also be valuable to see how the Channel-wise Influence method performs in a standard time series forecasting setup for a more comprehensive evaluation.

A7： We are not entirely sure we understand your question on this point. As described in Section 4.2 of our paper, the motivation behind channel pruning is to accelerate training and enable interpretability analysis, rather than to propose a method for improving time series forecasting performance. Therefore, we did not conduct experiments under the standard time series settings. If we have misunderstood your concerns, we would greatly appreciate your clarification.

Q5: Additionally, the forecast length was fixed at 96, which may restrict the credibility of the results and the method's broader applicability.

A5： Thank you for your suggestion. We have added experimental results for the prediction length of 192. The detailed results are as follows:

From the results shown in the table, it can be observed that channel-pruning based on channel-wise influence is more effective. Additionally, iTransformer still exhibits a larger core subset, demonstrating its superior ability to model channel dependency.

Q1: Lack of Clarity in Presentation: Figure 1, intended as an overview of the framework, does not clearly correspond with the main text, leaving several critical points unexplained. For instance, the calculation of the "Score" in the figure is not detailed, nor is its derivation clearly defined in the "CHANNEL-WISE INFLUENCE FUNCTION" section.

A1： We have revised the paper according to your suggestions and re-uploaded it. The modified sections are highlighted in blue for your reference.

Q2: Additionally, the term "well-trained model" lacks a concrete description of what type of model is being referred to.

A2： We have revised the paper according to your suggestions and re-uploaded it. The modified sections are highlighted in blue for your reference.

Q3: The paper also mentions that the "Channel-wise Influence" can serve as an explainable method to assess the channel-modeling capabilities of different approaches; however, it lacks detailed explanations and specific case studies to illustrate this claim.

A3: In our paper, we conducted interpretability analysis in Remark 3.2 of Section 3, as well as in the Result Analysis and Outlook sections of 5.2.1. Specifically, the size of the core subset (the red-marked proportions in Table 5 indicate the size of the core subset) reflects the model's ability to leverage information across different channels. A larger core subset indicates that the model can effectively distinguish and capture dependency between channels. This implies that the model is able to fully utilize information from various channels to enhance its predictive performance. The more information utilized, the stronger the model's capability to capture channel dependency. Therefore, the size of the core subset provides an explanation of the model's ability to model channel dependency.

Q4: Limited Datasets, Leading to Less Convincing Results: The experiments are conducted on a limited number of datasets, which reduces the generalizability and representativeness of the results. For example, in the time series forecasting task, only the electricity, solar-energy, and traffic datasets were used, without evaluating common benchmarks like ETTh, ETTm, and Exchange.

A4： We proposed a novel influence function and validated its effectiveness and superiority through diverse experiments and datasets. In the context of time series forecasting, one of the objectives of our channel pruning approach is to accelerate training. Therefore, we selected datasets with relatively high computational resource demands for our experiments. To better address your suggestions, we have added experiments on the ETTh1 and ETTm1 datasets, as shown in the table below. Overall, we believe our use of a relatively large number of datasets in time-series research highlights the generalizability and effectiveness of our approach.

Since the original number of channels in ETTh1 and ETTm1 is only 7, the horizontal axis in the table directly represents the number of retained channels.

The results in the table demonstrate the effectiveness of channel pruning based on the channel-wise influence function, highlighting that PatchTST and iTransformer exhibit comparable utilization of channel information on the ETTh1 and ETTm1 datasets.

We appreciate your time to provide valuable comments and suggestions to improve our paper substantially. Before we begin addressing your questions, we would like to first clarify the primary contributions of our paper. Influence functions have demonstrated significant performance and value across various fields[4]-[10], with notable applications in outlier detection[7][8][9][10] and data pruning[4][5]. However, there is no preceding research on the influence functions of multivariate time series to shed light on the effects of modifying the channel of multivariate time series. To fill this gap, we propose a channel-wise influence function, which is the first data-centric method that can estimate the influence of different channels in multivariate time series. We conducted extensive experiments and found that the original influence function performed poorly in anomaly detection and could not facilitate channel pruning, underscoring the superiority and necessity of our approach. Additionally, we can further analyze the information learned by the model through the channel-wise influence matrix. For example, as mentioned in Section 5.2.1 of our paper, comparing the number of core channel subsets across different models reveals each model's capacity for capturing channel dependencies. A larger core subset indicates that the model has captured more effective information, which also highlights the interpretability of our approach.

Regarding the selection of tasks to verify our method: Anomaly detection has long been a critical issue in multivariate time series analysis, with relevant studies including[1][2]. Data pruning is equally important as it raises the question of whether we can train a high-performing model with less data than predicted by scaling laws[3], with related work found in studies such as [3][4][5]. However, data pruning has not yet been extensively studied within the context of time series. By analyzing the characteristics of multivariate time series, we identified redundancy between channels and developed a channel pruning method based on our proposed channel-wise influence function, which outperforms traditional data pruning approaches. In addition, this method also supports the interpretability analysis of the model's ability to model multivariate time series.

[1]Position paper: Quo vadis, unsupervised time series anomaly detection? ICML 2024

[2]Anomaly transformer: Time series anomaly detection with association discrepancy. ICLR 2022

[3]Beyond neural scaling laws: beating power law scaling via data pruning Neurips 2022

[4]Data Pruning via Moving-one-Sample-out Neurips 2023

[5]Dataset Pruning: Reducing Training Data by Examining Generalization Influence ICLR 2023

[6]Self-influence guided data reweighting for language model pre-training. ACL 2023

[7]Detecting adversarial samples using influence functions and nearest neighbors. CVPR 2020

[8]Estimating training data influence by tracing gradient descent. Neurips 2020

[9]Understanding Black-box Predictions via Influence Functions. ICML 2017

[10]Resolving Training Biases via Influence-based Data Relabeling. ICLR 2022

Thanks for the responses. We sincerely and respectfully hope that you might consider reevaluating our work in light of the responses we have provided, which aim to address your main concerns. Your insights and contributions will be a nice addition to our community.

We greatly appreciate your time and effort in advance. If you have other concerns, we are willing to further address them.