W1. Principles underlying channel dependency (CD)

To better clarify (1) the concept of implicit/explicit heterogeneity and (2) the motivation for addressing CDs varying across TS datasets, we have revised the 1. Introduction section in the revised manuscript. We kindly invite you to reread the revised 1. Introduction section to facilitate the below discussions.

W1-1. Necessity of accounting for varying CD between datasets

In a conventional setting where a single model is trained on a single dataset (i.e., non-TSFM), varying CDs across datasets may not pose a significant issue; we can choose either CI or CD model to perform effectively on that particular dataset.

However, the challenge arises when training a TSFM, which is a single model designed to generalize across multiple datasets with varying CDs. Because each dataset may benefit from a different approach (CI or CD), it is crucial for a TSFM to account for varying CD across datasets, as illustrated in Figure 1 and noted in L43--46.

Despite this, previous TSFMs have mainly focused on accommodating explicit heterogeneity, such as difference in dataset shape and size, while overlooking the implicit heterogeneity stemming from varying CD, as noted in L47--50. This oversight motivated us to propose a PCD framework that adjusts the CDs of various datasets (captured by a single model) based on the unique characteristics of each dataset to handle the implicit heterogeneity. To achieve PCD, we propose a channel mask (CM) constructed for each dataset, which is then element-wise multiplied with the CD captured by a model (attention matrix) for adjustment, with the visualization of CM of ETTh1 dataset shown in the second panel of Figure 5.

W1-2. Global CM under distribution shift (at test time)

W1-2-a) Intention of CM

By "global", we refer to features that are ”fixed across time points” , capturing stable characteristics rather than ”adapting to local changes”. Adjusting the (global) CM based on test-time local changes would thus conflict with the intended purpose, reducing their ability to encapsulate stable characteristics. To address local variations, we rely on the ”attention matrix”, which dynamically adapts to (local) input TS. This approach ensures that while the (global) CM provides a stable foundation, the (local) attention matrix directly addresses temporal and local shifts. Additionally, we confirmed that (global) CM and the (local) attention matrix effectively complement each other on 13 real-world datasets, which are known to have significant distribution shifts [A], as shown in Table 12. The results indicate that the proposed method effectively captures both stability and adaptability under varying conditions.

W1-2-b) Stability of (global) CM

To address the reviewer’s concern regarding the stability of the (global) CM when local fluctuations occur, we conducted an additional experiment using the Weather dataset with iTransformer: Specifically, we considered two intervals ((0%,80%) and (0%,100%)), and trained the model separately on each interval. The L2 distance between the two trained CMs was approximately 0.0002, indicating minimal difference, given that two random matrices with the same range (0,1) and shape yield an average distance of approximately 0.02. Regarding the stability of “performance” under distribution shifts, we provide a more detailed discussion in W2-1 below.

[A] Han, Lu, Han-Jia Ye, and De-Chuan Zhan. "The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting." TKDE (2024)

Thank you for your clarification. While your response addresses certain concerns (e.g., differing reproduction results, though I still believe further clarification is warranted—though I will not contest it given that UniTS, a peer-reviewed paper, reports a similar experimental outcome for iTransformer), it does not fully resolve my earlier concern regarding global and local CD. You have explained that CM is designed for global CD, while local CD is addressed through channel-wise attention. If this is indeed the case, I require further elaboration on how attention mechanisms alone are sufficient to effectively capture local, time-varying CD without the incorporation of additional data. If your assertion holds, iTransformer should demonstrate the most significant performance gains in ablation studies conducted on datasets exhibiting the largest degree of time-varying CD (such as Exchange and Solar-Energy); however, this does not appear to be the case. Furthermore, I find it unclear how the proposed attention mechanism for local CD can be consistently applied across different domains.

Given that some of my major concerns remain unaddressed despite your clarification, I maintain my current score as it is.

Thank you for engaging in the discussion!

Below we provide further clarification, hopefully it addresses misunderstanding on our work:

Concern regarding global and local CD

First of all, regarding your question "how the proposed attention mechanism for local CD ~", we do not specifically propose a method to capture local CD; it is naturally captured by the conventional attention mechanism applied across channels in Transformers, and this contrasts with global CDs captured by our proposed channel masks (CMs). In other words, our contribution lies in introducing CMs capturing global CD, which complements local CDs captured by the attention mechanism.

To further clarify the type of CDs captured by the attention mechanism and CMs, we provide an example below:

Given a TS with 7 channels, a length of 5000, and a lookback window of 96:

• CM is constructed with the entire TS of shape $(7, 5000)$ to capture the (global) dependency of channels over the entire time period.
• An attention matrix is constructed with the input TS segment of shape $(7, 96)$ to capture the (local) dependency of channels within the lookback window.

Given this example, we merely name (1) the CD captured by the CM as global CD, as the CM is constructed from the (global) entire TS $(7, 5000)$, and (2) the CD captured by the attention matrix as local CD, as the attention matrix is constructed from the (local) input TS $(7, 96)$, as noted in L178--180.

Based on the above clarification, we answer your questions below:

"Q1. how attention mechanisms alone are sufficient to effectively capture local, time-varying CD"

The conventional attention mechanism used in Transformers captures dependencies between tokens, and inspired by prior works such as iTransformer [A], each channel is treated as a token in our setting. Hence, the model captures the local CD within the input TS segment, where its effectiveness has already been demonstrated in previous studies [A,B,C,D,E].

Q2."iTransformer should demonstrate the most significant performance gains ~ on datasets exhibiting the largest degree of time-varying CD (such as Exchange and Solar-Energy); however, this does not appear to be the case."

First of all, we kindly ask you to share a supporting reference on your claim "Exchange and Solar-Energy are datasets exhibiting the largest degree of time-varying CD", as we believe this will be essential to strengthen our analysis on the effectiveness of our method.

In contrast to your expectation, we argue that if datasets exhibit the largest degree of time-varying (or local) CD, then the performance gain by capturing global CD would be minimal, as iTransformer already captures the essential (local) CDs for such datasets.

In general, it is hard to anticipate a relatively larger gain on a specific dataset than others, as numerous factors contribute to the final performance. For example, the fixed size of the lookback window we used in an experiment could be large enough to capture global CDs in some datasets, such that the gain by our CMs is relatively small regardless of the degree of local CDs.

We emphasize that Table 12 shows an ablation study highlighting the effectiveness of our method. Specifically, global CD only outperforms local CD only on both Exchange and Solar-Energy, demonstrating the importance of capturing global CD through our CM. Of course, they have complementary information, such that using both results in the best result across all datasets.

Q3."how the proposed attention mechanism for local CD can be consistently applied across different domains.

Again, the attention mechanism for capturing local CD is essentially the same as those in the conventional Transformers, and its effectiveness has already been demonstrated in previous studies [A,B,C,D,E]. Furthermore, as the attention mechanism has no domain-specific parameters, and we believe there is no specific issue on its applicability across different domains [B]. Introducing domain-specific attention mechanisms would be an interesting idea, but it involves lots of additional learnable parameters; as such, we leave this exploration for future works.

[A] Liu et al. "iTransformer: Inverted transformers are effective for time series forecasting." ICLR (2024)

[B] Gao et al. "Units: Building a unified time series model." NeurIPS (2024)

[C] Ilbert et al. “SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention.” NeurIPS (2024)

[D] Wang et al. “CARD: Channel aligned robust blend transformer for time series forecasting.” ICLR (2024)

[E] Yu et al. “Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling.” ICML (2024)

W1. PCD vs. CCM

We believe the proposed PCD framework is fundamentally different from the channel clustering method in CCM [A], which is concurrent to our work (Note: CCM will appear in NeurIPS 2024, which has not yet been held!).

While CCM and our method share a superficial similarity in balancing CI and CD, the goals and approaches differ significantly. CCM enhances performance by incorporating channel cluster information to (CI or CD) models, whereas our method adjusts the varying CD across multiple datasets for TSFMs using dataset-specific information (no clustering involved in our method). Prior works with similar objectives [B, C] also combine CI and CD; however, this does not imply that they represent the same algorithm.

[A] Chen, Jialin, et al. "From Similarity to Superiority: Channel Clustering for Time Series Forecasting." NeurIPS (2024)

[B] Nie, Tong, et al. "Channel-aware low-rank adaptation in time series forecasting." CIKM (2024)

[C] Chi, Ce, et al. "InjectTST: A Transformer Method of Injecting Global Information into Independent Channels for Long Time Series Forecasting." arXiv preprint (2024).

W2, W5-2. Lack of generality: The proposed method is only applicable to models based on attention-mechanism that captures CD.

PCD aims to enhance TSFMs capturing CDs by leveraging dataset-specific characteristics, rather than proposing a plug-in-play method applicable to any type of TSFMs existing in the world, as mentioned in L60--61 and L69--70. Specifically, CM is designed to adjust the CD estimated by the model, making it applicable to CD models but infeasible for CI models.

Among the five TSFMs the reviewer mentioned ([2]--[6]), the reasons we did or could not experiment with them--either due to model architecture limitations ([2,3,4,6]), dataset size constraints ([3,4,5,6]), or code availability ([4]). Most importantly, these works are concurrent with ours (NeurIPS 2024: [2], ICML 2024: [3--6]), which made it difficult for us to run experiments due to the large dataset size and the timing of the code release. To address these challenges, we applied our method to two other SOTA models, iTransformer [A] (ICLR 2024) and TimeSiam [F] (ICML 2024), both of which are single-task models with released code.

Additionally, while we acknowledge that not all TSFMs rely on attention mechanisms and that Transformers are widely employed to capture temporal dependencies (TD), recent methods--particularly with the emergence of iTransformer [A]--have increasingly emphasized leveraging attention mechanisms to capture CD [B, C, D, E, F]. Our method aligns with this trend and builds upon the strengths of these approaches.

Furthermore, the limitation of being applicable only to the attention mechanism for capturing CD has been already discussed in L482--484, where we identified the development of a novel approach to achieve PCD without relying on Transformer-based methods as a promising direction for future work.

[A] Liu, Yong, et al. "iTransformer: Inverted transformers are effective for time series forecasting." ICLR (2024)

[B] Gao, Shanghua, et al. "Units: Building a unified time series model." NeurIPS (2024)

[C] Ilbert, Romain, et al. “SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention.” NeurIPS (2024)

[D] Wang, Xue, et al. “CARD: Channel aligned robust blend transformer for time series forecasting.” ICLR (2024)

[E] Yu, Guoqi, et al. “Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling.” ICML (2024)

[F] Dong, et al. "TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling." ICML (2024)

Thanks for the response. I'm still concerned about the applicability of CM. As the author emphasizes in the title of the paper and lines 22-23, CM serves the time series foundation models. However, this paper only uses a foundation model for experiments, which I think is not enough. Specifically, the ICML 2024 papers were made available online in February, accepted in May, and the latest ones were open-sourced in May as well. The UniTS architecture used in this paper was accepted by NeurIPS 2024 (the acceptance time of this conference is later than ICML 2024). Therefore, the excuse of working during the same period may not be reasonable. Furthermore, the issues with the architecture and dataset further demonstrate the limitations of the proposed method. Your approach is designed for foundation models, but it does not easily apply to most existing foundation models, which leads to significant limitations in applicability. Therefore, I will maintain my score.

Thank you for your response. After careful consideration, I maintain my decision based on the following points:

1.Utilizing attention mechanisms to capture channel correlations is not a universally accepted approach; alternatives such as GNN[1] also exist. While PCD offers potential value, its applicability appears limited to a narrow range of scenarios.

2.Verifying the method only with UniTS does not sufficiently demonstrate the generalizability of PCD.

3.A comprehensive comparison with mainstream time series foundation models is essential. If the model with PCD cannot achieve competitive performance against state-of-the-art models, its contribution to the field may be perceived as limited.

Given these considerations, I suggest allocating more time to strengthen the evidence for your work. This conference may not be the most appropriate venue for its publication at this stage. Additionally, it seems the authors are more enthusiastic about engaging in discussions than optimizing the manuscript itself, which is worth reflecting upon.

[1] Wu, Z., Pan, S., Long, G., Jiang, J., Chang, X., & Zhang, C. (2020, August). Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 753-763).

W1-1. Inconsistent results with the original paper

We acknowledge the inconsistency between the results reported in the original paper and our paper. However, we reproduced the results using the official code repositories of iTransformer [A] and UniTS [B] for our experiments, running the provided scripts without any modifications.

Unfortunately, this reproduction yielded inconsistent results for both models compared to the original paper, and we found that this inconsistency is shared with others, according to GitHub issues in both the iTransformer and UniTS repositories.

• Specifically, for iTransformer, we found a similar issue reported in a Chinese GitHub discussion, which we translated to confirm its relevance.
• For UniTS under the prompt tuning setting, we encountered an issue where the model failed to converge using the provided script and the problem is also reported by others in a GitHub issue. This issue was resolved by setting the hidden dimension to $D=32$, which yielded degraded performance compared to the original paper. We applied this setting uniformly across both UniTS and its application to our method for fair comparison, as detailed in Appendix B.2.

Nonetheless, following the reviewer’s feedback, we have added further reproduction details in Appendix B to avoid any confusion.

[A] Liu, Yong, et al. "iTransformer: Inverted transformers are effective for time series forecasting." ICLR (2024)

[B] Gao, Shanghua, et al. "Units: Building a unified time series model." NeurIPS (2024)

W1-2. Full results of ETT datasets (across all prediction lengths)

We agree with the reviewer’s observation that the ETT datasets are widely used and important for TS forecasting tasks. However, due to space constraints, we have provided the comprehensive results for these datasets, including comparisons across all prediction lengths, in Appendix C.1, with a reference included in L251.

W2. Comparison with other baseline methods.

Our primary focus is on improving TSFMs' ability to better capture CD, rather than achieving better performance compared to non-TSFMs, including recent methods (e.g., RLinear, ModernTCN, CrossGNN) mentioned by the reviewer. Nevertheless, we compared some non-TSFMs (task-specific models) in Table 3, following the baseline established by the backbone model we used (UniTS [A]), as the proposed CM is a plug-in method. Nonetheless, recognizing the significance of the methods mentioned by the reviewer, we have included them in Section 2. Related Works to highlight their relevance to this area of research.

[A] Gao, Shanghua, et al. "Units: Building a unified time series model." NeurIPS (2024)

W3. Varying size of lookback window ($L$)

Thank you for your insightful comment.

We set $L$ to be the same as that used in the backbone method we applied, ensuring a fair and consistent comparison. Nonetheless, we agree with the reviewer on the importance of the size of the lookback window ($L$) and we have conducted an additional ablation study presented in Appendix F, using three datasets (ECL, Traffic, PEMS03). In these experiments, we varied $L \in \\{48, 96, 192, 336, 720\\}$, with a forecast horizon of $H = 12$ for PEMS03 and $H = 96$ for the other datasets, in accordance with previous work [A]. The results demonstrate that the impact of CM remains consistent across different values of $L$, further validating its robustness.

[A] Liu, Yong, et al. "iTransformer: Inverted transformers are effective for time series forecasting." ICLR (2024)

W1-2. CM may be insufficient to capture temporal variation in CD

Regarding the phrase "complex and changing characteristics inherent in time series data" mentioned by the reviewer, it could be interpreted in two ways:

• (1) (Narrow meaning) Temporal variation (distribution shift)
• (2) (broad meaning) Inherent dynamic nature of the TS dataset

Below are the responses to each interpretation:  

(Narrow meaning) Temporal variation

It is important to emphasize that the primary role of CM is to capture the varying CD between datasets, rather than within a single dataset over multiple time steps. Therefore, adjusting the (global) CM based on dynamic changes would conflict with its intended purpose, potentially diminishing its ability to capture the stable, dataset-specific characteristics. To address dynamic changes within datasets the reviewer noted, we rely on the attention matrix, which is constructed using the local TS. This approach ensures that while the (global) CM provides a stable foundation, the (local) attention matrix directly addresses temporal and local shifts.

Furthermore, as shown in Table 12, our experiments on 13 real-world datasets, known to exhibit significant distribution shifts [A], demonstrate that the (global) CM and the (local) attention matrix effectively complement each other. The results indicate that the proposed method effectively captures both stability and adaptability under temporal variations.

(Broad meaning) Inherent dynamic nature of the TS dataset

In response to the question about the sufficiency of our approach in capturing the dynamic and complex characteristics of TS, our work provides a practical solution for addressing CD in the TS domain. While we recognize the value of a comprehensive theoretical analysis, exploring the full complexity of TS data goes beyond the scope of this study and is challenging to address within a conference paper. We leave further investigation in this direction to future works.

[A] Han, Lu, Han-Jia Ye, and De-Chuan Zhan. "The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting." TKDE (2024)

W2. Limited technical contribution

W2-1. Dataset-specific model for capturing dataset-specific CD

It is important to note that our work focuses on time series foundation models (TSFM), which involve training a single model on multiple datasets. By contrast, the models mentioned by the reviewer are dataset-specific models (i.e., single dataset-single model), which are beyond the scope of our current work . In such cases, varying CD across datasets is less of a concern, as each dataset is trained with its own individual model. However, this approach requires a separate model for each dataset, resulting in inefficiency.

W1. Lack of theoretical grounding or in-depth analysis

While we believe our paper has already addressed most of the concerns raised by the reviewer, below we provide explanations and references for each (W1-1, 1-2, W-3).  

Nonetheless, to better clarify (1) the concept of implicit/explicit heterogeneity and (2) the motivation for addressing CDs varying across TS datasets, we have revised the 1. Introduction section in the revised manuscript. We kindly invite you to reread the revised 1. Introduction section to facilitate the below discussions.

W1-1. Definition of implicit heterogeneity

W1-1-a) Concept of implicit heterogeneity and its necessity

• [Concept] As stated in L37--43, ”explicit” heterogeneity refers to observable differences among datasets, such as varying sequence lengths and the number of channels. In contrast, ”implicit” heterogeneity pertains to differences in unobservable factors across datasets, such as variations in CD, where some datasets exhibit strong inter-channel dependencies, while others require weak or minimal dependencies.
• [Necessity] As highlighted in L43--46, addressing implicit heterogeneity is essential for TSFMs; unlike a dataset-specific model tailored to a single dataset, a TSFM is a single model trained on multiple datasets with varying CD, each potentially requiring a distinct approach (CI or CD), as illustrated in Figure 1. We also show that addressing this heterogeneity brings a significant performance gain on various tasks, as shown in Table 7 ( second row (CD) vs. last row (PCD) ).
• [Motivation] This underscores the importance of considering TSFMs not only in terms of explicit heterogeneity (focusing on the model architecture), but also in terms of implicit heterogeneity (focusing on the dataset itself).

W1-1-b) Concept of CD & Comparison with previous works

The concept of CD we refer to aligns with that in prior works, but the key distinction lies in focus. While previous works emphasize capturing CD within a single dataset through model architecture, we address varying CD across multiple datasets by concentrating on dataset characteristics.

We believe the following comparison with previous works (focusing on CD) highlights the distinctiveness of our work and the differing focus on CD:

• (1) Non-TSFMs: A single model is trained on a single dataset, and works on the trained data distribution. Their research focus is on designing model architectures capturing CD within a single dataset.
• (2) Prior TSFMs: A single model is trained across multiple datasets, and works on any data distribution. Their research focus is on designing model architectures accommodating multiple datasets with varying input shapes (explicit heterogeneity)
• (3) Ours: A single model is trained across multiple datasets, and works on any data distribution. Our research focus is on designing model architectures better accommodating multiple datasets with varying CD (implicit heterogeneity) based on dataset characteristics.

W1-1-c) Implicit heterogeneity and its connection with CM

To address ”implicit heterogeneity” across TS datasets, we introduce the concept of partial channel dependence (PCD), which adjusts the CD captured by the model using dataset-specific information to align with the unique characteristics of each dataset. PCD is implemented through a ”channel mask (CM)”--a matrix constructed from dataset-specific information (e.g., correlation)--that is multiplied with the model's CD (e.g., the attention matrix) for adjustment.

I apologize for my delayed response, which was due to an accident early in the last Thanksgiving break. I appreciate the authors' efforts and detailed responses to my comments as well as those from the other reviewers. I now understand this work better than before. I acknowledge the motivation of this work, but I believe the authors' argument that their proposed method can be used for various TSFMs (despite the fact that most TSFMs have emerged recently) or at least existing CD-based models is not supported by either concrete theoretical analysis or a rich set of experimental results. I think that the authors' conceptual elaboration on the theoretical grounding and the limited empirical results focused on experiments with UniTS and iTransformer are still insufficient to support their argument. In conclusion, I believe the authors' claim of broad applicability for the proposed method is not well supported. Personally, I think the authors do not need to claim that their method can be applied to multiple TSFMs but should instead focus on rigorously demonstrating the methodological contribution of PCD to capture varying CD across different datasets and models.

W3, Q1. Missing critical baseline comparison (PCD vs. Full CD/CI)

PCD ($\mathbf{A}=\mathbf{M}$) vs. Full CD ($\mathbf{A}=\mathbf{1}$)

It is important to clarify that the "full CD" model mentioned by the reviewer corresponds to the "baseline method" (without CM), as it employs CD captured by the model without adjustment, which also corresponds to the fixed dependency structure in W2-2. This (baseline (= Full CD) vs. baseline with CM (= PCD)) comparison has been extensively conducted in our experiments, consistently demonstrating the superiority of our PCD over full CD.

PCD ($\mathbf{A}=\mathbf{M}$) vs. Full CI ($\mathbf{A}=\mathbf{I}$)

As discussed in L60---61, CM is designed to adjust the CD estimated by the model, making it applicable to CD models but infeasible for CI models. However, to facilitate a comparison, we can enforce the CD model to behave as CI by replacing $\mathbf{A}$ in Equation 1 with the identity matrix, allowing us to compare CD and CI under the same architecture. We have conducted this comparison in Figure 7, which shows the performance gain achieved by applying (full) CD over (full) CI, demonstrating that full CD > full CI, and consequently PCD > full CI.

Initially, we omitted the full CI case from Table 7 as our focus was on adjusting the CD model (starting with the CD model as the baseline). However, we have now included the results in the first row of Table 7, with the first and second rows corresponding to full CI and full CD, respectively.

Q2. PCD in the extreme (full CI/CD) cases

In the extreme cases mentioned by the reviewer, the correlation matrix aligns naturally with either CI or CD for the following reasons:

• Full CI: If the two channels are completely independent, the correlation is “0”, resulting in the correlation matrix becoming the identity matrix.
• Full CD: If the two channels are completely correlated (either positively or negatively), the correlation is either +1 or -1, resulting in an absolute correlation of one and producing a matrix of ones (or negative ones).

The CM extends the flexibility of the correlation matrix by applying monotonic transformation via affine adjustments and mapping values to the range (0,1) using a sigmoid function. As a result, the CM aligns with either CI or CD in these extreme cases, while accommodating intermediate levels of dependence.

To further address concerns regarding PCD’s performance under extreme cases, we performed an additional toy experiment in Appendix G. Specifically, we designed a scenario where the channels in TS exhibit no correlation. We generated a synthetic TS dataset with two channels using sine waves oscillating at frequencies of 0.5 and 2.0 over a length of 18,000 (similar to ETTh1). The results show that the CD ratio of CM is approximately 0.018 and the forecasting MSE is around 0.0014, confirming strong channel independence and demonstrating the effectiveness of our method even under extreme CI conditions.

Q4. Efficiency analysis (vs. CD vs. CI)

As stated in W3(Q1), CM adjusts CD captured from the CD model and applies only to it, so we compare exclusively with the CD model. The results of this comparison are already presented in Table 14 and L465--470. These results show that incorporating CMs does not significantly impact computational time, even for datasets with a large number of channels. Additionally, as noted in L463--464, incorporating CM requires only two additional parameters per dataset, resulting in minimal memory overhead and maintaining computational efficiency.

W1, Q3. Dataset Identification Requirement

The assumption noted by the reviewer is a fundamental premise for TSFMs in general [A, B, C, D]. While TSFMs are capable of training on multiple heterogeneous datasets simultaneously, they typically require knowledge of each sample’s data source during training. Assuming the dataset source is unknown during training--as opposed to inference--would be uncommon in practice. Instead, a more realistic scenario involves conducting inference on datasets that were unseen during training, which we have already addressed in Table 6 and Table 9.

That said, developing a TSFM that does not rely on dataset source information during training could be an intriguing direction for future research, even though it falls beyond the scope of our current work.

If the reviewer is aware of any settings or models where this assumption is unnecessary, we would be grateful for the reviewer’s insight.

[A] Zhou, et al. “One fits all: Power general time series analysis by pretrained lm.” NeurIPS (2023)

[B] Goswami, et al. “Moment: A family of open time-series foundation models.” ICML (2024)

[C] Liu, et al. “Timer: Generative pre-trained transformers are large time series models.” ICML (2024)

[D] Woo, et al. “Unified training of universal time series forecasting transformers.” ICML (2024)

W2. Lack of Fixed vs. Variable Dependency Analysis

W2-1. Why fixed dependency structure for TSFM wouldn't work equally well?

In a conventional setting where a single model is trained on a single dataset (i.e., non-TSFM), varying CDs across datasets may not pose a significant issue; we can choose either CI or CD model to perform effectively on that particular dataset.

However, the challenge arises when training a TSFM, which is a single model designed to generalize across multiple datasets with varying CDs. Because each dataset may benefit from a different approach (CI or CD), it is crucial for a TSFM to account for varying CD across datasets, as illustrated in Figure 1 and noted in L43--46.

Despite this, previous TSFMs have mainly focused on accommodating explicit heterogeneity, such as difference in dataset shape and size, while overlooking the implicit heterogeneity stemming from varying CD, as noted in L47--50. This oversight motivated us to propose a PCD framework that adjusts the CDs of various datasets (captured by a single model) based on the unique characteristics of each dataset to handle the implicit heterogeneity. To achieve PCD, we propose a channel mask (CM) constructed for each dataset, which is then element-wise multiplied with the CD captured by a model (attention matrix) for adjustment, with the visualization of CM of ETTh1 dataset shown in the second panel of Figure 5.

W2-2. Comparison of fixed vs. variable dependency analysis

The “fixed vs. variable dependency analysis” raised by the reviewer aligns with our “baseline vs. baseline with CM” comparisons, and is extensively addressed throughout the manuscript .

The “baseline” assumes fixed CD across datasets by employing a single model, while the “baseline with CM” adjust the CDs captured by a single model by leveraging dataset-specific information to handle the varying CD across datasets .

Follow-up discussion on 3. Generality of our method (21yw, bW9C, saeS)

We emphasize again that our method is designed to work exclusively with models that capture CD using attention mechanisms, as stated in L55--56 and L69--70. We admit that the emphasis on the scope of applicability in the initial submission might be insufficient. To address this, we have revised the manuscript to clarify the applicability of the proposed PCD during the rebuttal; we emphasized the changes in green.

Although it might not meet some reviewers' expectations regarding the scope, we believe our contribution is still significant in other aspects:

• CD models' relevance to TSFMs: We highlight the importance of CD, as CI can be achieved with CD by disregarding dependencies between irrelevant channels when well captured [A]. We further argue that the CD architecture is especially crucial for TSFMs which are trained on large-scale datasets, due to the capacity-robustness trade-off, where CD models benefit more from larger datasets than CI models [B].

• Emphasis on dataset when building TSFMs: Our work highlights the importance of considering both model architecture and datasets when constructing TSFMs. As large-scale datasets and TSFMs become more prevalent, capturing varying CD across datasets becomes increasingly critical, as each dataset may require a different approach (CI or CD). Our proposed PCD addresses this need by explicitly accounting for dataset-specific characteristics.

We recognize that the current title might suggest broader applicability than intended, though we initially thought that "Partial CD (with CM) for TSFMs" already limits the application of our PCD to "TSFMs modeling CDs." We welcome suggestions for a more precise title. For example, "Partial Channel Dependence with Channel Masks for Channel-Dependent Time Series Foundation Models" would be an alternative, while it has redundancy because of repeating "CD". Please feel free to share your opinion with us, we are willing to revise the title accordingly.

[A] Nie, Yuqi, et al. "A time series is worth 64 words: Long-term forecasting with transformers." ICLR (2023)

[B] Han, Lu, Han-Jia Ye, and De-Chuan Zhan. "The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting." TKDE (2024)

It is worth emphasizing that, first of all, the author may have misunderstood my summary. I did not think that the method proposed in this paper is novel, but only used the words "a concise approach". Secondly, just as most reviewers have concerns about the generalization of the proposed method, the author emphasizes in the title and lines 22-23 that this paper is suitable for time series foundation models, but the authors refuse to employ other foundation models (except UniTS ) conducted more experiments to demonstrate the generalization of the proposed method. Therefore, I will keep my original rating.

Thank you for further continuing to engage in the discussion.

While we appreciate the reviewer's efforts, we believe that some criticisms may stem from misunderstandings of our work and research on FMs. Below we further clarify them and provide detailed responses below.

I did not think that the method proposed in this paper is novel, but only used the words "a concise approach".

We apologize for our miscategorization of reviews when summarizing them. We updated our general comment accordingly.

the author emphasizes in the title and lines 22-23 that this paper is suitable for time series foundation models

Thank you for specifically raising this concern. Our title already addresses "channel dependence (CD)": namely, Partial CD (with CM) for TSFMs. Additionally, the statements in L69--70 ("The proposed CM is a plug-and-play method applicable to any model that captures CD using an attention mechanism") and L55--56 ("which adjusts the CD estimated by the Transformer-based model") both demonstrate that the scope is not "all" TSFMs but specifically models that capture CD using attention mechanisms. L22-23 the reviewer mentioned refers to the applicability to UniTS, iTransformer, and TimeSiam that we experimented with. Hence, we believe this title already limits the application of our PCD to TSFMs modeling CDs, so we thought that it reflects its applicability while concise enough. To account for your concern, "Partial Channel Dependence with Channel Masks for Channel-Dependent Time Series Foundation Models" would be an alternative, while it has redundancy because of repeating "CD". We think this requires further discussion with other reviewers, and if they agree, we are willing to revise the title accordingly.

but the authors refuse to employ other foundation models (except UniTS ) conducted more experiments to demonstrate the generalization of the proposed method.

We emphasize that we did NOT refuse to experiment other FMs; rather, it is infeasible due to fundamental limitations on them. As previously stated, our method is designed for "models that capture CDs using attention mechanisms." The models mentioned by saeS do not fall into this category, and hence, they cannot be used. Instead, we experimented with iTransformer [ICLR 2024] and TimeSiam [ICML 2024] that capture CDs using attention mechanisms to show the applicability of our proposed method.

We found that the reasons why saeS gives rejection is summarized as

• 1. the title is somewhat broad, and
• 2. we did not experiment with FMs published this year.

However, regarding 1), we are willing to revise upon consensus with reviewers, so this should not be the reason for rejection.

Regarding 2), we argue that this is not reasonable for recent research in deep learning, especially for FMs; for example, the time between the ICLR 2025 submission and ICML 2024 presentation was only around 2 months, and saeS is asking us to address many papers including 4 ICML 2024 papers [2--5] with experiments, which is not feasible. The reason why we could make quick progress on top of UniTS [NeurIPS 2024] is that they released their code before acceptance, and the model/dataset size of UniTS was relatively small while applicable to multiple tasks; however, all works mentioned by saeS do not meet this condition, i.e., they did not make their work quickly reproducible. We believe research in FMs is quickly progressing in this way, and asking a research on FMs to experiment ALL concurrent works would be a bottleneck of the progress in this field. There might be multiple ways of building TSFMs, and we believe our proposed PCD is a meaningful step forward progressing TSFMs capturing CDs.

In other words, for our understanding, your criticism is "your work is on top of a NeurIPS 2024 paper, and I am going to reject your paper because you did not take account of several ICML 2024 papers," which we believe is not appropriate for research on deep learning and/or FMs. Generally speaking, reviews with this kind of criticism will simply result in rejecting most research on FMs.

Furthermore, all concurrent works saeS mentioned are out of scope for our work due to the inherent difference in the model architecture.

Thank you for providing a summary of your concerns. While we appreciate Reviewer saeS is aware of recent progress in TSFMs and trying to provide helpful comments, we believe your concerns come from misunderstanding of our work and/or progress in this area of research. Below we response to each of your concern:

1. "PCD can only be applied to models that use attention mechanisms to capture correlations, which significantly limits its scope of application."

We believe "limited scope of application" does not deteriorate the contribution of our work, as the scope of general TSFMs is too broad. In particular, "foundation model" is a broad term that does not specify its methodology. For example, we believe no one would criticize the popular Wasserstein GAN paper because it was not applied to other generative models, such as VAE or diffusion models.

Indeed, our method effectively improves the state-of-the-art TSFM (and 2 other models) in this line and we compared it with 2 TSFMs (UniTS [1] and GPT4TS [14]), validated through extensive experiments.

2. "Most time series foundation models adopt CI strategies"

We respectfully disagree with this statement. Based on our survey, 5 out of 9 TSFMs adopt CD strategies, while 4 of them adopt CI strategies. Below we summarize recent TSFM works, which were mostly uploaded this year (2024), including the works cited by the reviewer:

CD Models

• [1] UniTS (NeurIPS 2024, final code update: 2024.02)
• [2] Moirai (ICML 2024, final code update: 2024.11)
• [3] GTT (CIKM 2024, final code update: 2024.02)
• [4] UniST (KDD 2024, final code update: 2024.07)
• [5] Tiny Time Mixers (TTMs) (arXiv 2024.01, no GitHub / available only on Hugging Face): Pretrained with CI, fine-tuned with CD

CI Models

• [6] Timer (ICML 2024): CI
• [7] TimesFM (ICML 2024): CI
• [8] MOMENT (ICML 2024): CI
• [9] ROSE (arxiv 2024.05): CI

Reviewer saeS's misunderstanding seems to come from the fact that TSFMs published in ICML 2024 mostly adopt CI strategies.

3. "for the few models like Moirai [1] that consider correlation through positional encoding, PCD is still unsuitable."

We also respectfully disagree with this assertion. Moirai [2] applies attention along the time axis after channel flattening. As these methods utilize the attention mechanism to capture both TD and CD simultaneously, the proposed CM can still be applied by extending its shape from $C \times C$ to $C \cdot P \times C \cdot P$, where $P$ is the number of patches in each channel, by duplicating the values within the same channels. Furthermore, as the reviewer mentioned, Moirai considers correlation through "positional encoding"; however, this is intended to distinguish between channels in a flattened single sequence, which is orthogonal to our approach.

4. Why not apply our method to other CD models, e.g., Moirai?

Although five CD models are listed above, the following explains why only UniTS [1] was used:

• [2] Moirai (ICML 2024): Their GitHub repository does not provide enough information for replicating pretraining, such that implementing "Moirai + CM" requires more effort than adding CM on top of it. Additionally, the reproduced fine-tuning results perform significantly worse than the zero-shot performance reported in the original paper (e.g., MSE of 0.424 (reproduced) vs. 0.375 (original paper) on ETTh1 with $H=96$), which is not only our issue but also shared by others in the GitHub issue. Furthermore, experiments are only feasible with the ETTh1 dataset. Additionally, the dataset size (231B) and the base model size (91M) make experiments not feasible to moderate experimental setup (they specified A100 for hardware). In fact, the official Moirai GitHub focuses mainly on enabling users to utilize the model as a package, e.g., performing zero-shot inference on their own datasets via Jupyter Notebooks, rather than replicating their model from scratch.

• [3] GTT (CIKM 2024): Only provides inference and fine-tuning code, with NO pretraining code available.

• [4] UniST (KDD 2024): Designed specifically for urban spatio-temporal prediction, using “4D-grid” spatial input $(N,T,H,W)$, where $H$ and $W$ represent spatial dimensions.

• [5] Tiny Time Mixers (TTMs) (arXiv 2024): It employs a special non-Transformer architecture without attention mechanism.

5. to demonstrate the effectiveness of the method, I believe it is necessary to compare it with state-of-the-art time series foundation models

Why not compare with other (CI or CD) TSFMs?

The reviewer further mentioned that we could still compare our method with the concurrent TSFMs. However, most TSFMs have emerged recently, and the field is still in its early stages, with no standardization and significant variability in pretraining datasets and experimental setups. This makes direct comparisons among TSFMs difficult, and only recently has research begun to address the necessity of benchmarking in this area [10, 11, 12], with both papers submitted as part of the ICLR 2025 submissions..

Notably, even Moirai [2], cited by Reviewer saeS, does not compare its results with other TSFMs. Similarly, TimesFM compares only with GPT4TS [10], an earlier LLM-based TSFM.

We emphasize that we already incorporated the comparison with two state-of-the-art TSFMs (UniTS and GPT4TS). Since other concurrent TSFM works also compare a similar number (0--2) of TSFMs, we believe this is sufficient. Nonetheless, we are actively exploring other TSFM works, and will try to incorporate relevant findings until the camera-ready submission.

Indeed, several works including studies on benchmarking TSFMs [11--15] report substantial variations in performance for the same TSFM due to differences in experimental setups.

For instance, on the Weather dataset, the zero-shot performance of Moirai measured by averaged MSE across four prediction lengths $\\{96,192,336,720\\}$ varies significantly:

• Moirai [2]: 0.238 (Table 6)
• FlexTSF [14]: 0.246 (Table 3)
• Time-MoE [13, ICLR 2025 submission]: 0.287 (Table 3)
• FoundTS [12]: 0.312 (Table 11)
• Gift-Eval [11]: used different metrics (MAPE, CRPS, rank)
• WaveToken [15, ICLR 2025 submission] : used different metrics (WQL, MASE, VRSE)

From Table 3 in our paper, the average MSE of ours is 0.245. Based on the replication of Moirai by concurrent works, our work is competitive to Moirai with much less parameters (1.57M vs. 91M), though we cannot assure the comparison is fair.

As setting a fair comparison of TSFMs is beyond our scope, we leave it to future works.

[1] Gao, Shanghua, et al. "Units: A Unified Multi-task Time Series Model." NeurIPS (2024)

[2] Woo et al. “Unified training of universal time series forecasting transformers.” ICML (2024)

[3] Feng et al. “Only the Curve Shape Matters: Training Foundation Models for Zero-Shot Multivariate Time Series Forecasting through Next Curve Shape Prediction.” arxiv (2024)

[4] Yuan, Yuan, et al. "Unist: a prompt-empowered universal model for urban spatio-temporal prediction." KDD (2024)

[5] Ekambaram, V. et al. “TTMs: Fast Multi-level Tiny Time Mixers for Improved Zero-shot and Few-shot Forecasting of Multivariate Time Series”, arxiv (2024)

[6] Liu, Y., Zhang, H., Li, C., Huang, X., Wang, J., & Long, M. (2024). Timer: Transformers for time series analysis at scale. arXiv preprint arXiv (2024)

[7] Das, Abhimanyu, et al. "A decoder-only foundation model for time-series forecasting." (ICML 2024)

[8] Goswami, et al. “Moment: A family of open time-series foundation models.” ICML (2024)

[9] Wang, Yihang, et al. "ROSE: Register Assisted General Time Series Forecasting with Decomposed Frequency Learning." arxiv (2024)

[10] Zhou, et al. “One fits all: Power general time series analysis by pretrained lm.” NeurIPS (2023)

[11] Aksu et al., “GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation” NeurIPS Workshop (2024)

[12] Z Li et al., “FoundTS: Comprehensive and Unified Benchmarking of Foundation Models for Time Series Forecasting” arxiv (2024)

[13] https://openreview.net/forum?id=e1wDDFmlVu

[14] Xiao, Jingge, et al. "FlexTSF: A Universal Forecasting Model for Time Series with Variable Regularities." arxiv (2024)

[15] https://openreview.net/forum?id=D9liZ0D8z8