Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shift

Q6. Constraints $R_1$ and $R_2$.

1. Definition for q and d in $\mathbf{D}$.

• In this paper, $\mathbf{D} = [\mathbf{D}^{(1)}, \mathbf{D}^{(2)}, ..., \mathbf{D}^{(K)}]$ represents the bases of $K$ subspaces, each $\mathbf{D}^{(j)} \in \mathbf{R}^{q \times d}$.
• $q = C \times D$ represents the dimension of the input features $z_i$, where $C$ is the number of input channels and $D$ is the hidden dimension for embedding patches.
• On the other hand, $d = q / K$ is the dimension for each subspace, where $K$ is the number of subspaces. This ensures that the total dimension is evenly distributed across the $K$ subspaces.

For clarity, we have detailed this in $\underline{\text{Appendix G Algorithm 1 of revised paper}}$.

2. Why control column size in Equation (3).

Equation (3) and (4) work synergistically to improve the compactness and separability of the representation learned by the model.

The regularization term $R_1$ directly penalizes the squared difference between the column norms of $\mathbf{D}$ and 1. If the columns of $\mathbf{D}$ are normalized (i.e., they have unit norm), then the matrix $\mathbf{D}^T \mathbf{D} \odot \mathbf{I}$ will be the identity matrix $\mathbf{I}$. In this case, the subtraction $\mathbf{D}^T \mathbf{D} \odot \mathbf{I} - \mathbf{I}$ will be zero, leading to a minimal value of $R_1$.

By minimizing $R_1$, the model is effectively encouraged to adjust $\mathbf{D}$ such that the columns have unit norms, which can be interpreted as controlling the "size" or "magnitude" of the columns of $\mathbf{D}$.

This regularization helps to:

• Prevent scale differences between the columns of $\mathbf{D}$, ensuring that no single subspace dominates the learning process due to its large scale.
• Avoid degenerate solutions, such as the collapse of column vectors towards zero, which would negatively impact the quality of the learned representations.

Thus, by minimizing $R_1$, we enhance the stability and robustness of the subspace representations, making the model less sensitive to scale variations across different experts.

3. The purpose of Equation (4).

The goal of the regularization term $R_2$ is to encourage orthogonality between different subspaces represented by $\mathbf{D}$. This constraint ensures that each subspace focuses on different patterns of the data, thereby improving the diversity of the learned experts.

By minimizing $R_2$, the model is discouraged from allowing subspaces to become too similar or overlap in terms of the features they represent. This promotes the separation of different experts, leading to a more effective allocation of responsibility across the subspaces.

This regularization term $R_2$ directly influences:

• Separability: It helps ensure that each expert represents distinct data patterns, making the model more adaptable to various types of distribution shifts.
• Improved performance: By maintaining diversity among the experts, the model can handle complex data distribution shifts better, improving overall forecasting accuracy.

In summary, by minimizing both $R_1$ and $R_2$, we achieve a compact and well-separated set of subspaces, which allows the model to effectively utilize different experts for different data patterns, leading to improved representation learning.

4. Ablation study about these constraints.

We conducted ablation experiments to further verify the important roles of $R_1$ and $R_2$.

These analyses have also been included in the $\underline {\text{Appendix Q.3 of revised paper}}$.

5. Visualization of clustering.

In addition, we present the t-SNE visualization of the learned embedded representation on ETTh1 in $\underline{\text{Appendix L of revised paper}}$. Figure 9 (b) shows the visualization of the representation learned by the proposed method, which effectively captures discriminative features and reveals significantly clearer clustering patterns.

We thank the reviewer for offering the valuable feedback. Below we give a point-by-point response to your concerns and suggestions.

W1. The setting of the patch length.

In this paper, we adopted the same patch length setting method as PatchTST, which has been proven effective in time series forecasting tasks. This setting allows us to focus on validating the effectiveness of our pattern-specific modeling strategy, where distinct experts handle specific pattern segments of the time series.

By using this method, we aim to highlight the advantages of our pattern-specific expert modeling over conventional uniform distribution modeling approaches, without introducing additional complexity from patch length selection at this stage. We acknowledge this as a potential limitation, as detailed in the $\underline {\text{Appendix I.Limitations of original submission}}$.

W2. The ability to adapt to entirely new patterns.

In our current framework, the model learns patterns based on historical data distributions and adapts to different data segments by selecting appropriate experts. However, for entirely new patterns, the model may require additional mechanisms for achieve dynamic adaptation.

As stated in $\underline {\text{Appendix I.Limitations of original submission}}$, addressing this challenge lies beyond the scope of the current work. Future research could explore techniques such as online learning, which would enable the model to better generalize to rare or previously unseen events.

W3. Computational complexity.

To make this clearer, we present the results of ETTh1 for a prediction length of 192 from $\underline{\text{Table 2 of the original submission}}$ and include additional results on runtime and computational complexity. Due to the sparsity of MoPE, TFPS achieves a remarkable balance between performance and efficiency:

1.Performance Superiority: TFPS achieves an MSE of 0.423, outperforming TSLANet (0.448), FITS (0.445), PatchTST (0.460), and FEDformer (0.441). This represents a 5.6% improvement over TSLANet and a 8.0% improvement over PatchTST, highlighting its significant accuracy gains. While DLinear achieves an MSE of 0.434, TFPS still demonstrates a 2.5% relative improvement, making it the most accurate model among all baselines.

2.Efficiency Gains: TFPS maintains competitive runtime and memory efficiency.

• Runtime: TFPS runs in 6.457 ms, making it 2.8× faster than PatchTST (17.851 ms) and 11.2× faster than TimesNet (72.196 ms).
• Memory Usage: TFPS uses 9.643 MB of GPU memory, significantly less than FEDformer (62.191 MB) and comparable to iTransformer (3.304 MB). This makes TFPS suitable for resource-constrained applications while maintaining superior performance.

3.Balancing Trade-offs: While lightweight models like DLinear (0.434 MSE, 0.789 ms runtime) are slightly more efficient, TFPS delivers a substantial performance improvement of 2.5%, providing a well-rounded solution that balances accuracy and efficiency effectively.

These results and analyses are also included in $\underline {\text{Appendix O of revised paper}}$.

Q1. The clarification of Figure 1.

Regarding the heatmap in Figure 1, we highlight the differences across patches.

• Specifically, in the time domain of Figure 1(a), patch #9 shows the onset of a sudden drift, which completes in patch #10. This lag results in a distinct pattern for patch #10. This abrupt change leads to a significant difference between patch #10 and the other patches, producing the highest MMD value. Additionally, patch #9, as part of the mutation process, also shows a large MMD value due to the emerging drift. In contrast, patches #3-#4 exhibit notable MMD differences when compared to the other patches in the frequency domain.

• In Figure 1(b), which depicts a gradual drift process, the pattern differences between patches #5–#10 and patches #0–#4 are significant, as indicated by the time domain heatmap. After converting to the frequency domain, the distinction between patch #6 and the other patches become more pronounced, leading to a large MMD value.

In summary, Figure 1 demonstrates that the time domain and frequency domain emphasize different aspects of distribution drift, and their combined perspective offers a more comprehensive understanding of these shifts.

Q2. Robustness for different types of datasets

Our proposed TFPS specially considers the combination of time and frequency domains to adapt to a wider range of data characteristics. By constructing subspaces, it effectively divides time series segments into distinct patterns, with specialized experts assigned to model each pattern. Through the results in $\underline{\text{Table 2 of original submission}}$, this approach has demonstrated robustness across various datasets, including those with clear periodicity, such as electricity data, as well as those with less apparent periodicity, like weather dataset.

Q3. Comparison with MoE-based methods and Distribution Shift methods.

To make this clearer, we list some results from $\underline{\text{Table 6 of original submission}}$ as follows and add the comparisons with MoE-based methods and Distribution Shift methods. Since IN-Flow and MixMamba do not provide source code, we instead compare two other recent MOE-based methods: MoU and KAN4TSF and method for distribution shift: Koopa. In addition, we cited IN-Flow and MixMamba in the Line 144 and 881, respectively.

1. Comparison with MoE-based methods. As shown in the table, unlike MoE-based methods that rely on the Softmax function as a gating mechanism and adopt the Uniform Distribution Modeling strategy (as presented in the $\underline{\text{Introduction of original submission}}$), our approach constructs a pattern recognizer to assign specific experts to handle distinct patterns. This tailored design results in TFPS achieving relative improvements of 2.3%, 9.0%, 10.6%, and 9.1% across the four datasets, respectively.

2. Comparison with Distribution Shift methods. Furthermore, as highlighted in the $\underline{\text{Related Work of the original submission}}$, existing distribution shift methods over-rely on normalization, leading to over-stationarization and diminishing the series' intrinsic non-stationarity. In contrast, our approach reintroduces the inherent non-stationarity into the latent representation, enabling TFPS to better handle distribution shifts by tailoring experts to the evolving patterns and densities within the data. Consequently, TFPS achieves notable relative improvements of 4.2%, 28.9%, 4.2%, and 28.9% across the four datasets, respectively.

These analyses have also been included in the $\underline{\text{Appendix M and N of revised paper}}$.

Q4. The contribution of PI.

• The PI module plays a crucial role in identifying and characterizing distinct patterns within the time series data, while the gating network dynamically selects the most relevant experts for each segment. This collaborative mechanism allows the model to specialize in handling different patterns and adapt effectively to distribution shifts, thus mitigating the overfitting risks that arise from treating all data equally.

• To validate the importance of PI empirically, we have conducted the ablation experiments comparing the model’s performance by replacing the PI module with a linear layer in the $\underline{\text{Table 3 of original submission}}$.

• In addition, we supplement some ablation experiments to further verify the effectiveness of PI:

These analyses have also been included in the $\underline {\text{Appendix Q.2 of revised paper}}$.

Q5. Computational complexity.

Refer to the answer to W3.

Q7. The difference between TFPS and conventional MoE.

The routing strategies in conventional MoE architectures are automatically learned, which can result in load imbalance and lead to inefficiencies.

1. In contrast, the Pattern Identifier (PI) with two constraints $R_1$ and $R_2$ in our TFPS model serves as the routing strategy, where it distinguishes the pattern differences of various time series segments through subspace clustering.

2. As mentioned in Q6, we increase the compactness and separability of the subspace by the constraints. We ensure that each segment is assigned to relevant experts, reducing the risk of load imbalance and improving the diversity of expert utilization across different data segments.

3. Through $\underline{\text{Figure 5 of original submission}}$ and $\underline{\text{Figure 9 of revised paper}}$, we demonstrate that our proposed TFPS effectively addresses the load imbalance problem.

Q8. Experiments on Traffic dataset.

We conducted addition experiments on high-dimensional Traffic dataset to further evaluate the performance and generalizability of TFPS.

These analyses have also been included in the $\underline{\text{Appendix J of revised paper}}$.

Q9. How RevIN affect PI.

1. ReVIN is proposed to help mitigate global distribution shifts by normalizing data distributions. However, the PI module operates on a complementary level by explicitly identifying and modeling different patterns at the patch-level.

2. In addition, by comparing the results of PatchTST with TFPS-Time Branch, we can demonstrate the effectiveness of our proposed strategy for identifying latent patterns of segments and modeling them separately.

3. The two components work together: ReVIN helps stabilize the learning process by reducing distributional variance, while the PI module enhances the model’s ability to specialize in different patterns, improving performance under more complex distribution shifts.

Q4. The fairness of the paper.

To ensure fair comparisons, we reran all experiments using the official implementations provided by the baseline methods, as detailed in $\underline{\text{Section 4.1 of the original submission}}$.

1. In this unified seq_len=96 setting, TFPS surpasses other baselines significantly. As shown in $\underline {\text{Table 2 and 4 of the original submission}}$, TFPS clearly outperforms the previous state-of-the-art methods, including TSLANet, FITS, iTransformer, PatchTST, DLinear, etc. This setting is particularly valuable in real-world scenarios where time and resources for extensive hyperparameter tuning are limited.

2. Under hyperparameter-search seq_len setting, TFPS still surpasses other baselines. To further ensure fairness, we also conducted experiments using hyperparameter tuning for the input sequence length (seq_len) across 96, 192, 336, and 720, consistent with the practices of some baseline methods. These results confirm that TFPS continues to outperform other baselines. All baselines were reproduced using their official code.

Below, we summarize the results under hyperparameter-search seq_len setting:

These results underscore the robustness of TFPS and its consistent ability to achieve state-of-the-art performance across diverse datasets.

3. In addition, we would like to highlight some points, that might be helpful to you in understanding our experiment settings:

• In this paper, we have provided two types of experiments: unifying input length as 96 and searching input length. Actually, all the previous papers only evaluate their models under one of the above two settings, such as iTransformer (ICLR 2024), TimesNet (ICLR 2023), and PatchTST (ICLR 2023). Especially, the MoLE and MixMamba models that you mentioned also only report results for the input-336 and input-96 settings, respectively.

• To maintain clarity in the paper’s structure, we report the unified seq_le=96 results in the main text, while the results with hyperparameter-search seq_le are included in the $\underline {\text{Appendix K of revised paper}}$. We have made every effort to ensure a fair comparison and clear presentation.

• Given many previous papers only provide the input-96 setting experiments (e.g. iTransformer, TimesNet, FEDformer), we believe that our previous rebuttal under the input-96 setting (more than 170 new results) is meaningful and convincing. Note that these ablations are all under an aligned hyperparameter setting to ensure rigor.

Many thanks for your prompt response and further clarifying your concerns, which is quite instructive for us to answer your questions in every detail.

Q1. The clarify about the efficiency.

We sincerely apologize for the inaccurate statement in our original response and have revised it in $\underline {\text{Appendix O of revised paper}}$.

While TFPS introduces additional computational cost, it achieves notable performance improvements, and we believe the increased resource requirements remain within an acceptable range.

We fully acknowledge the importance of efficiency and plan to explore further optimizations in future work, such as reducing memory and GPU usage through techniques like sparsity, pruning, or lightweight architectures, while maintaining TFPS's superior performance.

Furthermore, TFPS addresses the critical issue of distribution drift in time series data by identifying distinct patterns and leveraging pattern-specific experts for separate modeling. This strategy not only improves prediction performance but also provides a degree of interpretability compared to traditional uniform distribution modeling. Therefore, we believe TFPS is a practical model in real-world applications and is valuable to deep time series forecasting community.

Q2. The clarify about the Traffic results.

iTransformer was evaluated using an NVIDIA P100 GPU in the original experiments, while we conducted our experiments using an NVIDIA RTX 3090 GPU. We think that slight variations in results can occur due to differences in hardware and environmental factors.

Q3. The clarify about the PatchTST results.

Following iTransformer, we fixed the lookback length to 96 to obtain the reported results for PatchTST (0.413, 0.460, 0.497, 0.501). In contrast, the results in the original paper (0.370, 0.413, 0.422, 0.447) were obtained with a lookback length of 336, which significantly influences forecasting performance. This difference in lookback length accounts for the discrepancies observed in the MSE values.

Thank you for your responses. While some concerns have been addressed, a few issues persist:

1. The claim of a 'remarkable balance between performance and efficiency' for TFPS in your response to my Q5, particularly in comparison to DLinear, seems inaccurate. I have seen your response to similar question from another reviewer, but even with a lighter backbone model, the gap in memory and GPU footprint remains significant.
2. There are notable differences between the reported results for the traffic dataset and those in the original papers. For instance, iTransformer's reported MSE and MAE values differ significantly from those in the original paper.
3. The reported results for Patch-TST show significant discrepancies compared to the original paper. For example for MSE, the original paper's results are: 0.370, 0.413, 0.422, 0.447 while the reported results are: 0.413, 0.460, 0.497, 0.501? There's a significant difference?
4. This raises a series concerns about the experimental setup and fairness of the comparisons throughout the paper.

Dear Reviewer,

This is a gentle reminder that it has been 4 days since we submitted our rebuttal. We would appreciate it if you could let us know whether our response has effectively addressed your concerns.

As per your valuable feedback, we have made revisions in the following areas:

• Appendix P is added to analyze the hyperparameter selection experiments.
• Included appendix O to discuss the trade-off between computational complexity and performance.
• We have added 6 new baselines (MoLE, MoU, KAN4TSF, Koopa, SOLID, OneNet) to demonstrate the advancement of TFPS in appendix M and N.
• We can complete 6 types of ablations on PI and $R_1$ / $R_2$, which verifies the effectiveness of our design in TFPS. The implementation details of each ablation have also been included.
• Elaborate the difference between TFPS and conventional MoE in using dynamical gating.
• Explore the interaction between Revin and PI.

In total, we have added more than 170 new experiments. All of these results have been included in the $\underline{\text{revised paper}}$.

We sincerely appreciate your insightful review and look forward to your feedback. Please feel free to reach out if you have any further questions or concerns.

Dear Reviewer,

We appreciate your efforts in reviewing our paper. We have:

1. Modified the inaccurate description.

2. Explained the differences between the comparison methods and the original text.

3. Clarified the fairness of this paper.

As the Author-Reviewer discussion period is nearing its conclusion, we would like to kindly inquire whether our most recent response has addressed your remaining concerns. If so, we kindly hope that you can reconsider your rating of our work.

Please feel free to reach out if there are any remaining points that need clarification.

Best regards,

The Authors

We thank the reviewer for offering the valuable feedback. Below we give a point-by-point response to your concerns and suggestions.

Q1. Rationality of frequency encoder.

As presented in the $\underline{\text{Section 3.4 of original submission}}$, we follow FNet [1] to replace the self-attention sublayer of the Transformer with a Fourier sublayer, which has been proven to be effective for extracting frequency-domain information.

• The decision to discard the imaginary part of the Fourier transform was made to simplify the design and focus on the real-valued components.
• Additionally, FreTS [2] has observed that the real part of the frequency-domain representation plays a more significant role than the imaginary part.
• Despite this simplification, our empirical results demonstrate that TFPS achieves competitive performance, demonstrating its capacity to model the underlying distribution structure of time series data effectively.
• To further validate the robustness of our approach, we adopted similar operations in FreTS to conduct experiments incorporating both the real and imaginary parts. The results in the table show that the performance of TFPS with the real part only is very similar to that when both parts are included, while requiring fewer parameters. This further reinforces the conclusion that TFPS remains highly effective even when focusing solely on the real part of the Fourier transform.

These analyses have also been included in the $\underline {\text{Appendix Q.1 of revised paper}}$.

[1] Lee-Thorp J, Ainslie J, Eckstein I, et al. FNet: Mixing Tokens with Fourier Transforms. NAACL, 2022.

[2] Yi K, Zhang Q, Fan W, et al. Frequency-domain MLPs are more effective learners in time series forecasting. NeurIPS, 2024.

Q2. Ablation study on $\beta$.

1. In our work, we performed a grid search to optimize the hyperparameter $\alpha_t$ = {0.0001, 0.001, 0.01} and $\alpha_f$ = {0.0001, 0.001, 0.01} as shown in $\underline{\text{Figure 10 in Appendix P of the revised paper}}$, which were ultimately fixed at $10^{-3}$ in our experiments.

2. Additionally, we conducted a grid search to optimize the balance factors $\beta_t$ = {0.01, 0.05, 0.1, 0.5, 1} and $\beta_f$ = {0.01, 0.05, 0.1, 0.5, 1}. The forecasting performance under different parameter values is displayed in $\underline{\text{Figure 10 in Appendix P of the revised paper}}$. Based on these results, we draw the following observations:

• First, the performance is affected when the value of $\beta_t$ and $\beta_f$ is too low, indicating that the proposed clustering objective is crucial for distinguishing patterns effectively.
• Second, excessively large values of $\beta_t$ and $\beta_f$ also degrade performance. One plausible explanation is that the excessive value influences the learning of the inherent structure of original data, which leads to disturbances of the embedding space.
• Overall, we recommend setting $\beta_t$ and $\beta_f$ to around 0.1 for optimal performance.

These analyses have also been included in $\underline{\text{Appendix P of revised paper}}$.

Q3. The varying contributions of time and frequency domains across datasets.

With the above hyperparameter tuning, the individual loss terms are already balanced, making it unnecessary to introduce additional balancing for the three terms in the final loss.

We agree with your opinion regarding the varying degrees of distribution shifts across different datasets and domains. To address this, we adapt the number of time-domain and frequency-domain experts, $K_t$ and $K_f$, for each dataset, as described in $\underline {\text{Section 4.6 of the original submission}}$.

Our findings suggest that a larger number of experts is beneficial in scenarios with more pronounced distribution shifts, such as ETTh1 in the time domain, whereas fewer experts are sufficient when the drift is less severe.

Q4. The results of treating seq_len as a hyperparameter.

(1) In this unified hyperparameter setting, TFPS surpasses other baselines significantly. As shown in $\underline {\text{Table 2 and 4 of the original submission}}$, TFPS clearly outperforms the previous state-of-the-art methods, including TSLANet, FITS, iTransformer, PatchTST, DLinear, etc. This advantage is particularly meaningful, as in real-world applications, there are often limitations in time and resources for hyperparameter searching.

(2) In the hyperparameter-search setting, TFPS still surpasses other baselines. As suggested by the reviewer, we conducted experiments by searching for input lengths among 96, 192, 336, and 720 to compare with the baselines. All baselines are reproduced by their official code.

To make the paper structure clear, we place the input-96 setting in the main text, while the results with hyperparameter searching are included in the $\underline {\text{Appendix K of revised paper}}$.

Q5. Runtime.

To make this clearer, we present the results of ETTh1 for a prediction length of 192 from $\underline{\text{Table 2 of the original submission}}$ and include additional results on runtime and computational complexity. Due to the sparsity of MoPE, TFPS achieves a remarkable balance between performance and efficiency:

1.Performance Superiority: TFPS achieves an MSE of 0.423, outperforming TSLANet (0.448), FITS (0.445), PatchTST (0.460), and FEDformer (0.441). This represents a 5.6% improvement over TSLANet and a 8.0% improvement over PatchTST, highlighting its significant accuracy gains. While DLinear achieves an MSE of 0.434, TFPS still demonstrates a 2.5% relative improvement, making it the most accurate model among all baselines.

2.Efficiency Gains: TFPS maintains competitive runtime and memory efficiency.

• Runtime: TFPS runs in 6.457 ms, making it 2.8× faster than PatchTST (17.851 ms) and 11.2× faster than TimesNet (72.196 ms).
• Memory Usage: TFPS uses 9.643 MB of GPU memory, significantly less than FEDformer (62.191 MB) and comparable to iTransformer (3.304 MB). This makes TFPS suitable for resource-constrained applications while maintaining superior performance.

3.Balancing Trade-offs: While lightweight models like DLinear (0.434 MSE, 0.789 ms runtime) are slightly more efficient, TFPS delivers a substantial performance improvement of 2.5%, providing a well-rounded solution that balances accuracy and efficiency effectively.

These results and analyses are also included in $\underline {\text{Appendix O of revised paper}}$.

Many thanks for your prompt response and further clarifying your concerns, which is quite instructive for us to answer your questions in every detail.

Q1. Results on the hyperparameter search setting.

1. Supplementary Results for Multi-Channel Scenarios: We have supplemented experimental results for scenarios with many channels, including the traffic and electricity datasets, to further validate the effectiveness and robustness of TFPS. These results are provided in $\underline {\text{Table 8 of Appendix K of original submission}}$.

2. Fairness in Baseline Reproduction: To ensure fairness, we reran all baseline experiments on a single NVIDIA RTX 3090 24GB GPU with codes provided by their official implementation, as shown in $\underline{\text{Section 4.1 of the original submission}}$. Slight differences in the results compared to the original papers may occur due to hardware and environmental variations.

3. Compared with the original paper: We statistically compare the results with those reported in the original paper and demonstrate that TFPS achieves highly competitive performance. The original iTransformer paper provides only the overall average result for ETT (0.383), whereas TFPS achieves a superior result of 0.332.

Q2. Computational Cost and Scalability

1. Lightweight Backbone: We conducted experiments replacing the encoder in TFPS with a lightweight model, Linear, to reduce model complexity. Compared to TFPS, TFPS-Linear halves the GPU memory and runs significantly faster, while still maintaining strong performance relative to DLinear due to the effectiveness of the proposed pattern-specific modeling.

2. Justification of Current Results: In addition, TFPS obtains large performance gains than DLinear on the ETTh2 (6.2%) and ETTm2 (16.7%) datasets, which demonstrate its robustness to effectively handle distribution drift. Moreover, TFPS offers interpretability by enabling analysis of pattern-specific modeling, which provides valuable insights into the underlying dynamics of time series data.

We thank the reviewer for offering the valuable feedback. Below we give a point-by-point response to your concerns and suggestions.

Q1. The ablation study on the hyper parameters.

1. In our work, we performed a grid search to optimize the hyperparameter $\alpha_t$ = {0.0001, 0.001, 0.01} and $\alpha_f$ = {0.0001, 0.001, 0.01} as shown in $\underline{\text{Figure 10 in Appendix P of the revised paper}}$, which were ultimately fixed at $10^{-3}$ in our experiments.

2. $\eta$ is set to the same value as $d$ as the dimension of $\mathbf{D}^{(j)} \in \mathbf{R}^{q \times d}$ to control the smoothness.

3. Additionally, we conducted a grid search to optimize the balance factors $\beta_t$ = {0.01, 0.05, 0.1, 0.5, 1} and $\beta_f$ = {0.01, 0.05, 0.1, 0.5, 1}. The forecasting performance under different parameter values is displayed in $\underline{\text{Figure 10 in Appendix P of the revised paper}}$. Based on these results, we draw the following observations:

• First, the performance is affected when the value of $\beta_t$ and $\beta_f$ is too low, indicating that the proposed clustering objective is crucial for distinguishing patterns effectively.
• Second, excessively large values of $\beta_t$ and $\beta_f$ also degrade performance. One plausible explanation is that the excessive value influences the learning of the inherent structure of original data, which leads to disturbances of the embedding space.
• Overall, we recommend setting $\beta_t$ and $\beta_f$ to around 0.1 for optimal performance.

These analyses have also been included in $\underline{\text{Appendix P of revised paper}}$.

4. Finally, with the above hyperparameter tuning, the individual loss terms are already balanced, making it unnecessary to introduce additional balancing for the three terms in the final loss.

5. We employed regularization techniques such as early stopping to mitigate overfitting. These strategies, combined with careful hyperparameter tuning, help us maintain a balance between model complexity and predictive performance. Furthermore, our extensive experiments, as presented in the paper, demonstrate stable and competitive performance across various datasets, highlighting the robustness of our model.

Through these efforts, we believe that our model is not prone to overfitting, and its generalization capabilities have been rigorously tested.

Q2. Comparison with the stacking attention layers.

• The key distinction between TFPS and the stacking attention layers lies in their modeling approaches:

The stacking attention layers adopts the Uniform Distribution Modeling strategy for all data (as presented in the $\underline{\text{Introduction of original submission}}$), whereas TFPS follows an MoE structure [1].

Specifically, TFPS leverages the Pattern Identifier as a gating network to dynamically select relevant experts based on the input patterns. This distribution-aware approach allows each expert to specialize in different data patterns, thereby improving overall performance.

[1] Shazeer N, Mirhoseini A, Maziarz K, et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR, 2016.

• In addition, we have added addition experiments where we replaced the MoPE module with stacked self-attention layers, keeping all other components and configurations identical. The results demonstrate that our proposed method significantly outperforms the stacked attention layers, which validates the importance of the Top-K selection and pattern-aware design in enhancing the model's representation capacity. In contrast, self-attention typically treats all data points uniformly, which may limit its ability to capture subtle distribution shifts or evolving patterns across patches.

These analyses have also been included in the $\underline {\text{Appendix R of revised paper}}$.

Dear Reviewer,

We would like to kindly remind you that it has been 4 days since we submitted our rebuttal. We hope our response has addressed your concerns and would appreciate any feedback you may have.

In line with your suggestions, we have made several improvements and provided detailed clarifications in the following aspects:

• Appendix P is added to analyze the hyperparameter selection experiments.
• The robustness of TFPS is analyzed.
• Appendix R elaborates on the differences between TFPS and stacking attention layers.

We have also conducted additional experiments, visualizations, and ablations to support our findings, all of which are now included in the $\underline{\text{Revised Paper}}$.

Thank you again for your thoughtful review. We look forward to your response and are happy to provide further clarification on any additional points.

Dear Reviewer,

We appreciate your efforts in reviewing our paper. We have:

1. Clarified the hyperparameter selection strategy.

2. Clarified the difference between TFPS and stacking attention layers.

As the Author-Reviewer discussion period is nearing its conclusion, we would like to kindly inquire whether our most recent response has addressed your concerns. If so, we kindly hope that you can reconsider your rating of our work.

Please feel free to reach out if there are any remaining points that need clarification.

Best regards,

The Authors

We thank the reviewer for offering the valuable feedback. Below we give a point-by-point response to your concerns and suggestions.

Q1. The number of subspace variables $k$.

The number of subspace variables $k$ is equivalent to the number of experts $K$. To improve clarity, we have unified the notation of $k$ in $\underline{\text{Section 3.5 line 288-293 of revised paper}}$.

Additionally, we have conducted experiments to analyze the impact of different values of $K$, as detailed in $\underline{\text{Section 4.6 of original submission}}$, and the corresponding results are thoroughly discussed in the paper.

Q2. The initialization of subspace.

In our method, the subspace variables $\mathbf{D} \in \mathbf{R}^{Kd \times Kd}$ are initialized with a random Gaussian distribution to ensure sufficient expressive power [1]. This initialization facilitates efficient learning of feature representations in the early stages of training, while iterative refinement during training enhances representation accuracy. For clarity, we have detailed this initialization process $\underline{\text{Appendix G Algorithm 1 of revised paper}}$.

[1] Jiang Z, Zheng Y, Tan H, et al. Variational deep embedding: A generative approach to clustering. CoRR, 2016.

Q3. The visualization of the clustering subspace.

We have included the t-SNE visualization of the learned embedded representation on the ETTh1 dataset in $\underline{\text{Appendix L of revised paper}}$. In Figure 9 (a), where the pattern identifier is replaced with a linear layer, the representation lacks clear clustering structures, resulting in scattered and indistinct groupings. In contrast, Figure 9 (b) demonstrates the representation learned by the proposed method, which effectively captures discriminative features and reveals significantly clearer clustering patterns.

Q4. The value of MoPE, $\alpha$, $\beta$.

1. In our work, we conducted a grid search to optimize the number of MoPEs in the time domain $K_t$ = {1, 2, 4, 8} and the number of experts in the frequency domain $K_f$ = {1, 2, 4, 8}, as detailed in $\underline{\text{Section 4.6 of original submission}}$.

2. Additionally, we performed a grid search to optimize the hyperparameter $\alpha_t$ = {0.0001, 0.001, 0.01} and $\alpha_f$ = {0.0001, 0.001, 0.01} as shown in $\underline{\text{Figure 10 in Appendix P of the revised paper}}$, which were ultimately fixed at $10^{-3}$ in our experiments.

3. Finally, we conducted a grid search to optimize the balance factors $\beta_t$ = {0.01, 0.05, 0.1, 0.5, 1} and $\beta_f$ = {0.01, 0.05, 0.1, 0.5, 1}. The forecasting performance under different parameter values is displayed in $\underline{\text{Figure 10 in Appendix P of the revised paper}}$. Based on these results, we draw the following observations:

• First, the performance is affected when the value of $\beta_t$ and $\beta_f$ is too low, indicating that the proposed clustering objective is crucial for distinguishing patterns effectively.
• Second, excessively large values of $\beta_t$ and $\beta_f$ also degrade performance. One plausible explanation is that the excessive value influences the learning of the inherent structure of original data, which leads to disturbances of the embedding space.
• Overall, we recommend setting $\beta_t$ and $\beta_f$ to around 0.1 for optimal performance.

These analyses have also been included in $\underline{\text{Appendix P of revised paper}}$.

Q5. The explanation of $K_t$ and $K_f$.

$K_t$ and $K_f$ refer to the number of experts in the time-domain and frequency-domain components, respectively. We have clarified this definition in $\underline{\text{Section 4.1 of revised paper}}$ for better readability.

Dear Reviewer,

We kindly remind you that it has been 4 days since we posted our rebuttal. Please let us know if our response has addressed your concerns.

Following your suggestion, we have answered your concerns and improved the paper in the following aspects:

• We clarify the notation of subspace variables $K$, $K_t$, and $K_f$, and explain the subspace initialization operations.
• We have provided the implementation details for the visualization subspace.
• Appendix P is added to analyze the hyperparameter selection experiments.

We have provided extensive experiments, visualization, and ablations to support our insight. All of these results have been included in the $\underline{\text{Revised Paper}}$.

Thanks again for your valuable review. We are looking forward to your reply and are happy to answer any future questions.

We thank the reviewer for offering the valuable feedback. Below we give a point-by-point response to your concerns and suggestions.

W1. The innovation of combining time and frequency domain.

Our contribution goes beyond simply decoupling time-domain and frequency-domain information (Dual-Domain Encoder). The novelty of our method lies in identifying multiple patterns within the data (Pattern Identifier) and assigning them to specialized experts for modeling (Mixture of Pattern Experts). This effectively addresses distribution shifts in time series data, which is a significant challenge in time series forecasting and leads to a significant improvement in forecasting performance.

W2. An explanation of MoE section.

• The key distinction between TFPS and the weighted multi-output predictor lies in their modeling approaches.

The weighted multi-output predictor adopts the Uniform Distribution Modeling strategy for all data (as presented in the $\underline{\text{Introduction of original submission}}$), whereas TFPS follows an MoE structure [1].

Specifically, TFPS leverages the Pattern Identifier as a gating network to dynamically select relevant experts based on the input patterns. This distribution-aware approach allows each expert to specialize in different data patterns, thereby improving overall performance.

[1] Shazeer N, Mirhoseini A, Maziarz K, et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR, 2016.

• In addition, we have added addition experiments where we replaced the MoPE module with weighted multi-output predictor, keeping all other components and configurations identical. The results demonstrate that our proposed method significantly outperforms the multi-output predictor, which validates the importance of the Top-K selection and pattern-aware design in enhancing the model's representation capacity. In contrast, multi-output predictor typically treats all data points uniformly, which may limit its ability to capture subtle distribution shifts or evolving patterns across patches.

These analyses have also been included in the $\underline {\text{Appendix R of revised paper}}$.

W3. Compared with RevIN.

We have conducted experiments comparing our approach to RevIN, as presented in $\underline{\text{Appendix E.2, Table 6 of original submission}}$. However, we sincerely apologize for the error in the average MSE statistics reported in Table 4. This issue has been corrected in $\underline{\text{Table 4 of the revised paper}}$.

The updated average MSE results for all normalization-based comparison methods are provided below. By leveraging distribution-specific modeling, TFPS achieves significant improvements, with an average MSE reduction of 18.9%.

Q1. Adaptively discovering time-series patterns.

We fully agree with your perspective on the selection of $K$. Directly linking $K$ to the number of inherent patterns in time series data is indeed a more practical and insightful approach. It would eliminate the need for manual tuning and allow for a more accurate representation of the data's intrinsic structure.

However, due to the unknown nature of patterns in time series data, such an approach requires a mechanism to automatically detect the number of patterns. In a separate study, we have explored this idea using a Dirichlet Process Gaussian Mixture Model for unsupervised pattern discovery, followed by forecasting. While initial results are promising, this research is still in progress and requires further refinement.

Your observation resonates strongly with our research direction, and we appreciate this insightful suggestion.

Q2. Typographical error.

We apologize for the typographical error. We have corrected this in the revised paper.

Dear Reviewer,

Thanks for your valuable and constructive review, which has inspired us to improve our paper further substantially. This is a kind reminder that it has been 4 days since we posted our rebuttal. Please let us know if our response has addressed your concerns.

Following your suggestions, we have provided the following revisions to our paper:

• Further clarification of our innovative points.
• Appendix R is added to elaborate on the differences between TFPS and weighted multi-output prediction in terms of modeling strategy and experimental demonstration.
• Further discussion on the prospect of "adaptively discovering time series patterns."

In this paper, we propose the TFPS as a simple but effective model and provide extensive experiments, visualization, and ablations to support our insight. All the revisions are included in the $\underline{\text{Revised Paper}}$.

Sincere thanks for your dedication! We are looking forward to your reply.