From Similarity to Superiority: Channel Clustering for Time Series Forecasting

Thanks for your valuable feedback and positive comments. We address the potential concerns as follows.

The computational efficiency of CCM, especially in large-scale applications, is not extensively discussed.

As discussed in Sec.4.3, the computational complexity of CCM scales linearly with the number of channels $C$, which is computationally efficient for real-world deployment. Our extensive experiments, including large-scale datasets such as traffic (15,122,928 observations) and stock (13,900,000 observations), provide strong evidence for CCM's promising scalability on extremely large-scale datasets: 1) CCM consistently reduces model complexity for CI models (e.g. DLinear and PatchTST), regardless of dataset scale or size (Fig. 5); 2) The linear scaling w.r.t channel count enables efficient handling of high-dimensional data, crucial for many practical applications; and 3) our experiments reveal that the performance gains achieved by CCM are maintained across different dataset sizes, suggesting that the method's benefits are robust and not limited to specific data scales. This demonstrates the efficiency and scalability of CCM, making it promising for extremely large-scale real-world deployments.

Furthermore, we've identified several strategies to optimize the scalability of CCM, including leveraging parallel and distributed frameworks for cluster assigner training and applying sampling on time series similarity computation to optimize computational resources. Algorithmic optimizations, such as efficient and fast attention with linear time complexity further support CCM's scalability. However, we would like to emphasize that these techniques are compatible and orthogonal to the contribution of this manuscript and require substantial additional research that would expand beyond our current scope. We appreciate your suggestion and will expand our discussion on computational efficiency in the revised version.

Where is the official code?

As we mentioned in Line677 (Appendix C.3), the code is available at the following anonymous link: https://anonymous.4open.science/r/TimeSeriesCCM-4E83. We will have an official GitHub repository once the paper is accepted.

The principal component analysis method is actually similar to the proposed method. Please provide a detailed explanation.

PCA and the proposed CCM serve fundamentally different purposes and operate on distinct principles.

PCA is primarily a dimensionality reduction technique that transforms high-dimensional data into a lower-dimensional space, preserving maximum variance. It does not inherently cluster data but rather restructures it into orthogonal components. In contrast, CCM is specifically designed for time series forecasting, focusing on grouping channels based on their intrinsic similarities to enhance prediction accuracy and interpretability. CCM dynamically clusters channels into cohesive groups, leveraging these similarities to capture complex inter-channel dependencies. It enhances forecasting by allowing models to treat clusters as coherent entities, thus improving both individual channel fit and cross-channel interactions.

Essentially, while PCA emphasizes variance capture and dimensionality reduction, CCM prioritizes similarity-based clustering to optimize time series analysis and forecasting.

Detailed discussions on the selection of similarity metrics and the impact of hyperparameters on performance would benefit the paper.

The selection of similarity metrics and other alternatives are discussed thoroughly in Appendix A.1. We also conducted ablation studies on cluster ratio and look-back window length to investigate their impact on model performance. Please refer to Appendix D for detailed results and analysis.

We hope that the above clarification improves your confidence in our work. Let us know if you have any further questions/concerns.

Dear Reviewer SRyH,

As the discussion period is close to end, we would like to follow up and ensure that our responses have adequately addressed your concerns.

We sincerely appreciate your comments and suggestions, which have significantly contributed to improving our paper. In response to your valuable feedback, we have added more discussion on the model efficiency and clarified the difference between PCA and our proposed CCM. We would be more than happy to further discuss if there are any remaining questions. Thanks again for your time and consideration.

Regards,
Authors

Thanks for your valuable feedback and positive comments. We address the potential concerns as follows.

Some LLM-based models were not selected as baselines.

After careful consideration, we've decided not to include LLM-based time series models in this manuscript for several reasons. Firstly, many LLM-based models treat multivariate time series as independent univariate series, which doesn't align with our core assumption about the importance of cross-channel interactions [1,2]. Secondly, some LLM-based methods that concatenate variates into a single long series [3] are incompatible with CCM's clustering approach, as the clustering will destroy the attention calculation along the single univariate time series. While we acknowledge the growing importance of LLM-based methods in time series forecasting, investigating clustering effects on these models requires substantial additional research that would expand beyond our current scope. We believe this warrants a separate, dedicated study. Our focus on traditional deep learning models allows for a thorough analysis of CCM's effectiveness within an established framework. We are more than happy to add discussions in our revised version.

[1] One fits all: Power general time series analysis by pretrained lm
[2] Time-llm: Time series forecasting by reprogramming large language models
[3] Unified Training of Universal Time Series Forecasting Transformers

CCM introduces additional space and time complexity. CCM’s scalability remains to be tested.

As demonstrated in Sec 4.3, CCM's computational complexity scales linearly with the number of channels $C$, ensuring efficient real-world applications. Our extensive experiments provide strong evidence for CCM's promising scalability on extremely large-scale datasets:

• CCM consistently reduces model complexity for CI models (e.g. DLinear and PatchTST), regardless of dataset scale or size (Fig. 5).
• The linear scaling w.r.t channel count enables efficient handling of high-dimensional data, crucial for many practical applications.
• CCM's performance gains persist across various dataset sizes, indicating robust benefits independent of data scale.

These findings collectively underscore CCM's efficiency and scalability, making it a highly promising approach for large-scale real-world deployments.

The improvements brought by CCM may sometimes be caused by randomness. Can the author provide a significance test?

We conducted a significance test using p-values for the ETTh1 dataset in Table 4, which represents the scenario where CCM shows a minimal performance boost among our experiments. The p-value test results are as follows (values lower than 0.05 are in bold):

CCM's improvement effect is significant in general. Even in this "worst-case" scenario for CCM (i.e., ETTh1 dataset), we observe significant improvements across multiple forecast horizons and base models. Specifically, the majority of the p-values (11 out of 14) are below the conventional significance threshold of 0.05, indicating statistical significance. Despite a few cases where the p-values are above 0.05, the overall trend strongly supports the effectiveness of CCM. We will include a more comprehensive significance test in our revised version.

The percentage increase in runtime and memory usage on the Traffic dataset?

We provide the percentage increase in model size and runtime on the Traffic dataset based on TimesNet and TSMixer (two CD models) as follows.

$\Delta$ Param (%):

$\Delta$ iter. time (%):

We observe that CCM represents a minimal increase in memory usage and runtime on TimesNet, demonstrating its parameter efficiency when applied to TimesNet. For TSMixer, there is a greater increase compared to TimesNet. However, it's important to note that TSMixer is inherently a more compact model. The absolute increase in parameters remains relatively small and reasonable. It's worth noting that these increases should be considered in the context of the performance improvements that CCM brings, which justifies this modest increase in model size. Moreover, when applied to CI models, (eg, DLinear and PatchTST), CCM consistently reduces the model complexity and runtime, further highlighting its versatility and efficiency.

A hyperparameter analysis for the parameter $\beta$?

We conducted a hyperparameter analysis for $\beta$, ranging from 0.1 to 2.0 on ETTh2 dataset (H=720) as follows.

We observe that the introduction of CCM (with appropriate $\beta$) consistently improves performance compared to the baseline without CCM. The optimal $\beta$ value varies slightly across different architectures, but generally falls in the range of 0.3 to 0.5. Moreover, we observe that TSMixer, PatchTST, and TimesNet show more stable performance than DLinear across different $\beta$ values. We will include the sensitivity analysis on $\beta$ in our revised version.

We hope that the clarification improves your confidence in our work. Let us know if you have any further concerns.

Dear Reviewer rxdY,

As the discussion period is close to end, we would like to follow up and ensure that our responses have adequately addressed your concerns. We sincerely appreciate your comments and suggestions, which have significantly contributed to improving our paper. In response to your valuable feedback, we conducted additional experiments on significance test, complexity evaluation and hyperparameter ablation study. We would be more than happy to further discuss if there are any remaining questions. Thanks again for your time and consideration.

Regards,
Authors

Thanks for your valuable feedback, which significantly improves the quality of this work. We also address below the potential concerns.

The introduction of CCM increases the model's complexity. The scalability of CCM to extremely large datasets remains untested. What strategies to ensure the scalability of CCM? What techniques to mitigate the increased computational overhead of CCM?

As discussed in Sec.4.3, the computational complexity of CCM scales linearly with the number of channels $C$, which is computationally efficient for real-world deployment. Our extensive experiments, including large-scale datasets such as traffic (15,122,928 observations) and stock (13,900,000 observations), provide strong evidence for CCM's promising scalability on extremely large-scale datasets:

• CCM consistently reduces model complexity for CI models (e.g. DLinear and PatchTST), regardless of dataset scale or size (Fig. 5).
• The linear scaling w.r.t channel count enables efficient handling of high-dimensional data, crucial for many practical applications.
• Our experiments reveal that the performance gains achieved by CCM are maintained across different dataset sizes, suggesting that the method's benefits are robust and not limited to specific data scales.

This demonstrates the efficiency and scalability of CCM, making it promising for extremely large-scale real-world deployments.

Furthermore, we've identified several strategies to optimize the scalability of CCM, including leveraging parallel and distributed frameworks for cluster assigner training and applying sampling on time series similarity computation to optimize computational resources. Algorithmic optimizations, such as efficient and fast attention with linear time complexity further support CCM's scalability. However, we would like to emphasize that these techniques are compatible and orthogonal to the contribution of this manuscript and deserve substantial additional research that would expand beyond our current scope. We appreciate your suggestion and will expand our discussion on computational efficiency in the revised version.

How can the similarity metric be further refined to enhance the clustering quality and address performance variations across different domains?

The similarity metrics and alternatives are discussed thoroughly in Appendix A.1. We select the Euclidean-based similarity metric due to its efficiency and generalization. Our extensive experiments also justify the effectiveness and robustness of this metric in evaluating the cross-channel similarity (see evidence in Appendix E) and consistently enhancing model performance.

The similarity metric can be refined by incorporating domain-specific feature selection and engineering or dynamically adjusting parameters based on data distribution characteristics. These techniques are compatible and orthogonal to the proposed method, so we consider them as future work. Again, we would like to emphasize that the current metric already demonstrates the efficiency across different domains and serves as a generalizable tool to evaluate channel similarity.

How does CCM perform in real-world applications with diverse and dynamic datasets, and what adjustments are necessary to optimize its performance in such scenarios?

Firstly, we clarify that our experiments already showcase CCM’s robust performance in real-world applications with diverse datasets (such as electricity, traffic, etc.). As mentioned in Line 246-247, CCM improves long-term forecasting performance in 90.27% of cases in MSE and 84.03% of 247 cases in MAE across 144 different experiment settings. Moreover, CCM's versatility extends beyond scenarios where cross-channel relations remain static over time. It supports dynamic clustering across different minibatches and time steps, which is crucial for practical applications. Due to the space limit, we discussed possible improvements such as incorporating dynamic similarity metrics or using domain-specific similarity metrics in Appendix F.

Explanations of a few cases that CCM does not outperform CI/CD strategies.

As we clearly discussed in Appendix C.5, CCM method is more useful in scenarios where channel interactions are complex and significant, which is usually the case in real-world data. Therefore, in datasets where channels are nearly independent or where the inter-channel relationships are not as pronounced, the benefits of CCM's clustering mechanism may be less significant.

We would like to emphasize that the average improvement rate across all cases (with different base models / datasets / forecasting lengths) is 2.443% in the long-term and 6.456% in short-term benchmarks. Specifically, CCM improves the performance in 90.27% of cases in MSE and 84.03% of cases in MAE in long-term benchmarks, given the context that the base models are already optimized for high performance. Therefore, any additional gains through CCM are noteworthy and indicative of its effectiveness in refining forecasting performance.

We hope that the above clarification improves your confidence in our work. Let us know if you have any further questions/concerns.

Dear Reviewer yXcm,

We sincerely appreciate your comments and suggestions. As the discussion period is close to end, we would like to follow up and ensure that our responses have adequately addressed your concerns.

We would like to emphasize that most of the issues raised were minor and had already been addressed or discussed in the original submission and appendix. Nevertheless, we have taken this opportunity to clarify these points and further improve the revision. Specifically, we have added more discussion on the model efficiency and potentially improved similarity metric (please refer to the author rebuttal and global comments).

We would be more than happy to further discuss if there are any remaining questions. Thanks again for your time and consideration.

Regards,
Authors

Thanks for your valuable feedback and positive comments. We address the potential concerns as follows.

Adding evaluation on synthetic dataset will help solidify the claims.

While synthetic datasets serve their purpose in controlled experiments, our study's emphasis on real-world datasets is crucial and more reasonable for validating the practical applicability and robustness of the proposed CCM. Unlike synthetic datasets generated in previous studies, which may not encompass the full spectrum of real-world complexities, we leverage diverse real-world benchmarks. These datasets inherently capture intricacies such as varied data distributions, noise levels, and nuanced interdependencies between channels in real-time forecasting scenarios. This focus allows us to demonstrate CCM's effectiveness in addressing practical challenges across different domains, highlighting its superiority over synthetic benchmarks that often fail to replicate such complex interactions.

Why shuffle channels in each batch? What happens when not shuffling the channels?

Shuffling channels aim to remove channel identity information. In the toy experiment in Sec 4.1, we train the model in two patterns: A) the model is trained on the original dataset and B) The model is trained on the randomly shuffled dataset. Let $W_i$ represent the weights for the $i$-th channel. In Pattern A, $W_i$ is optimized upon the $i$-th channel only, while in Pattern B, the optimization of $W_i$ will lose channel identity information, which causes performance degradation on both CI and CD models (from Table 1). Therefore, this toy experiment justifies that channel identity information benefits model performance in general.

How does the prediction error change when the prediction time horizon is changed between training and inference? Is it possible to modify the time horizon after training?

Yes, it is possible to adjust the forecasting horizon after training, although it's not typically done in practice. When shortening the forecasting horizon post-training, the prediction error remains relatively stable because the training loss averages errors over each future step. However, if extending the horizon (e.g., from H1 to H2 where H2 > H1), the conventional approach involves forecasting H1 steps first, then using that prediction as input to forecast the subsequent H2 - H1 steps. This sequential process tends to increase prediction error due to accumulated inaccuracies over the extended horizon.

How do you use random and k-means for cluster assignment?

In the Random method, each channel is assigned to clusters with uniform probability. We use sklearn.cluster.KMeans to implement the k-means algorithm with a default maximum number of iterations (max_iter=300). The input channel embeddings remain the same as that of our proposed Cluster Assigner. We keep the number of clusters consistent for three clustering methods in the ablation study. We will add details in our revised version.

How do you expect the performance of the CCM to change when the channel count is on the order of millions?

Our clustering technique, which scales linearly with respect to channel counts, has been tested to efficiently handle large datasets. Numerous experiments across diverse datasets, including large-scale datasets such as traffic (15,122,928 observations) and stock (13,900,000 observations), provide strong evidence for CCM's promising scalability on large-scale datasets: 1) CCM consistently reduces model complexity for CI models (e.g. DLinear and PatchTST), regardless of dataset scale or size (Fig. 5); 2) The linear scaling w.r.t channel count enables efficient handling of high-dimensional data, crucial for many practical applications; and 3) our experiments reveal that the performance gains achieved by CCM are maintained across different dataset sizes, suggesting that the method's benefits are robust and not limited to specific data scales. Therefore, CCM is supposed to effectively group channels based on similarities while maintaining optimal computational efficiency.

What does a cluster ratio of 1.0 mean?

A cluster ratio of 1.0 indicates that the number of clusters equals the number of channels, but it does not imply channel independence. In our experiments, even with a cluster ratio of 1.0, we observed clustering phenomena in channel embeddings, albeit with some empty clusters. This ratio is maintained for experimental rigor and to validate the robustness of the model.

Dear Reviewer 9puR,

As the discussion period is close to end, we would like to follow up and ensure that our responses have adequately addressed your concerns. We sincerely appreciate your comments and suggestions, which have significantly contributed to improving our paper. We would be more than happy to further discuss if there are any remaining questions. Thanks again for your time and consideration.

Regards, Authors