Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shift

We thank the reviewer for the constructive feedback and the recognition of our MoE-based pattern modeling, visualization insights, and presentation quality. Below we give a point-by-point response to your concerns and suggestions.

W1 & Q1. Clarification regarding similarity to DEC.

We thank the reviewer for pointing out the connection to Deep Embedded Clustering (DEC). We acknowledge its influence and will cite it appropriately in the revised manuscript. We also appreciate the opportunity to clarify the distinctions between our approach and DEC.

While our formulation bears some structural similarity to DEC, the underlying mathematical formulation and application objectives differ substantially:

• Clustering mechanism: DEC computes soft cluster assignments using pairwise similarities between embedded features and cluster centroids via a t-distribution kernel. In contrast, TFPS derives subspace affinities by projecting latent features onto learned subspace bases and measuring projection magnitudes, resulting in a structure-aware similarity that is better suited for patch-level modeling in time series.
• Downstream usage: While DEC uses sharpened targets for unsupervised representation learning, TFPS leverages them to guide distribution-specific expert routing for time series forecasting. This introduces a distinct application context and serves a different modeling purpose.

We will make these distinctions clearer in Section 3.5 of the revised manuscript.

W2 & Q2. The rationale for iFFT in Fig 2.

We confirm that the iFFT operation in the frequency branch is intentional and serves a functional role.

Following the extraction of spectral features via FFT and the identification of pattern-specific subspaces in the frequency domain, we apply iFFT to reconstruct temporally aligned signals. This design enables seamless fusion between frequency-aware and time-domain features in the final representation, as similarly adopted in recent works like FreTS and FITS.

We will revise the corresponding description in Section 3.6 to clarify this design and its role more explicitly.

[1] Yi K, Zhang Q, Fan W, et al. Frequency-domain MLPs are more effective learners in time series forecasting. NeurIPS, 2023.

[2] Xu Z, Zeng A, Xu Q. FITS: Modeling Time Series with $10 k$ Parameters. ICLR, 2024.

W3. Function of the frequency-domain encoder.

1. Although FNet employs the Fourier transform as a lightweight replacement for attention in NLP, our frequency encoder serves a distinct purpose: it captures global periodic patterns and frequency-aware representations that complement the local dependencies modeled by the time-domain encoder.
2. Rather than replacing attention, TFPS integrates diverse inductive biases through its dual-branch structure. We focus on the real-valued spectrum to avoid training instability and unnecessary complexity, as shown in Table 13.
3. Empirically, the frequency branch is essential for identifying distributional heterogeneity (Table 2), validating its contribution beyond mere architectural variation.

W4 & Q3. Comparison with recent baselines.

To address the reviewer’s suggestion, we include comparisons with several recent state-of-the-art methods, including FilterTS, TimeMixer, SOFTS, and, FreTS, using their official implementations. As shown below, TFPS achieves competitive or superior MSE results on 6 out of 7 datasets and performs particularly well on those with complex temporal dynamics, such as ECL and Weather.

Regarding the results on the Traffic dataset, we believe that SOFTS performs better primarily due to its explicit modeling of inter-variable dependencies. As shown in Table 5, the dataset exhibits relatively stable temporal dynamics with limited distribution shift, and its strong channel-wise correlations may favor methods specifically designed to capture such relationships.

These results demonstrate that TFPS maintains strong performance under competitive baselines, with consistent top-tier ranking across most datasets.

[1] Wang Y, Liu Y, Duan X, et al. Filterts: Comprehensive frequency filtering for multivariate time series forecasting. AAAI 2025.

[2] Wang S, Wu H, Shi X, et al. TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. ICLR, 2024.

[3] Han L, Chen X Y, Ye H J, et al. Softs: Efficient multivariate time series forecasting with series-core fusion. NeurIPS, 2024.

[4] Yi K, Zhang Q, Fan W, et al. Frequency-domain MLPs are more effective learners in time series forecasting. NeurIPS, 2023.

Many thanks for your prompt response and further clarifying your concerns.

W1 & Q1. Acknowledging and positioning related work.

We agree that these prior methods have inspired aspects of our formulation, particularly the subspace-based clustering perspective. To acknowledge this, we have updated Line 155 in Section 3.3 of the revised manuscript to explicitly state:

“The core of our approach lies in leveraging subspace clustering to detect concept shifts across multiple subspaces, inspired by prior subspace-based clustering methods [1,2].”

These references are now clearly cited, and we appreciate the opportunity to better position our work within the broader clustering literature.

W2 & W3. On the performance of the frequency branch and imaginary part modeling.

We thank the reviewer for the thoughtful comments and opportunity to clarify. Below, we address the concerns in detail:

1. Purpose of the frequency branch: The frequency branch is not intended to outperform the time branch individually, but rather to complement it by capturing global, position-invariant patterns. As shown in Table 2, the full TFPS model consistently outperforms either branch alone, demonstrating the complementary nature and necessity of both branches.

2. Effect of imaginary part modeling:

• To directly address the reviewer’s concern, we conducted an additional ablation study isolating the frequency branch only, comparing variants that use only the real part (Table 2) with those incorporating both real and imaginary components. As shown below, adding the imaginary part does not consistently improve performance, even when the frequency encoder is used alone:

• This observation aligns with prior findings: FNet [3] and E2USD [4] omit the imaginary part altogether while still achieving strong downstream performance. FreTS [5] empirically shows that the real part contributes more significantly than the imaginary component.

• Moreover, incorporating both real and imaginary parts increases memory usage from 9.643MB to 11.189MB on the ETTh1-192 setting (Table 12, 13)—a ~16% overhead with no consistent benefit.

Therefore, in our default TFPS architecture, we retain only the real part to preserve model simplicity, interpretability, and computational efficiency.

3. Design of the imaginary-part experiment in Table 13: To incorporate the imaginary part in our ablation, we followed a design similar to FreTS: we concatenate the real and imaginary parts of the FFT output, then apply a linear projection to reduce the combined representation back to the original d_model size. This transformed feature is then passed into the subsequent encoder layers. This design allows us to retain the full spectral information while maintaining compatibility with the overall architecture.

4. On future directions: We agree that frequency modeling can be further improved. As part of future work, we plan to investigate complex-valued encoders and learned spectral filters, which may offer more expressive modeling of the frequency domain.

W4.Clarification on baseline results and comparison strategy

We appreciate the reviewer’s observation. While baseline results may differ across implementations, we find that the TimeMixer results we report are consistent with those presented in the FilterTS paper, confirming the alignment of our reproduction with their findings.

To ensure fairness, we reused the best TFPS results from Table 6 and compared them against the original TimeMixer and FilterTS results reported in their papers, as well as the FreTS results reported in the FilterTS paper. These comparisons demonstrate that TFPS maintains strong performance under competitive baselines.

[1] Unsupervised Deep Embedding for Clustering Analysis. ICML, 2016.

[2] Improved Deep Embedded Clustering with Local Structure Preservation. IJCAI, 2017.

[3] FNet: Mixing Tokens with Fourier Transforms. NAACL, 2022.

[4] E2usd: Efficient-yet-effective unsupervised state detection for multivariate time series. WWW, 2024.

[5] Frequency-domain MLPs are More Effective Learners in Time Series Forecasting. NeurIPS, 2023.

We thank the reviewer for offering the valuable feedback and for recognizing the clarity of our presentation and the motivation design. Below we give a point-by-point response to your concerns and suggestions.

W1. Clarity of Figure 1 and Figure 7.

We thank the reviewer for this helpful suggestion, which we believe will significantly improve the presentation quality of the paper. In the revised version, we will enlarge the font size of heatmap annotations in Figure 1, ensure consistent boundary line styles, and reorganize Figure 7 to eliminate overlapping elements and improve overall visual clarity.

W2. Comparison with recent baselines.

To address the reviewer’s suggestion, we include comparisons with several recent state-of-the-art methods, including FilterTS, TimeMixer, SOFTS, and, FreTS, using their official implementations. As shown below, TFPS achieves competitive or superior MSE results on 6 out of 7 datasets and performs particularly well on those with complex temporal dynamics, such as ECL and Weather.

Regarding the results on the Traffic dataset, we believe that SOFTS performs better primarily due to its explicit modeling of inter-variable dependencies. As shown in Table 5, the dataset exhibits relatively stable temporal dynamics with limited distribution shift, and its strong channel-wise correlations may favor methods specifically designed to capture such relationships.

These results demonstrate that TFPS maintains strong performance under competitive baselines, with consistent top-tier ranking across most datasets.

[1] Wang Y, Liu Y, Duan X, et al. Filterts: Comprehensive frequency filtering for multivariate time series forecasting. AAAI 2025.

[2] Wang S, Wu H, Shi X, et al. TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. ICLR, 2024.

[3] Han L, Chen X Y, Ye H J, et al. Softs: Efficient multivariate time series forecasting with series-core fusion. NeurIPS, 2024.

[4] Yi K, Zhang Q, Fan W, et al. Frequency-domain MLPs are more effective learners in time series forecasting. NeurIPS, 2023.

W3. Performance on ECL and Traffic datasets in Table 9.

We thank the reviewer for this insightful comment and provide further analysis on the ECL and Traffic datasets. The TFPS results reported below are consistent with those in Table 1. As shown below, TFPS significantly outperforms several representative DLinear variants augmented with different normalization strategies, confirming its robustness and generalizability on these challenging benchmarks.

W4. Ablation study on the loss function.

We conduct an ablation study to assess the contribution of each loss component in TFPS. Specifically, we evaluate the effect of removing the core forecasting loss $L_{\text{MSE}}$ and the two subspace-guided pattern identification losses $L_{PI_t}$ and $L_{PI_f}$.

As shown below:

• Removing $L_{\text{MSE}}$ causes substantial degradation, confirming its fundamental role in forecasting,
• Removing either $L_{PI_t}$ or $L_{PI_f}$ also results in consistent performance drops, validating their importance in guiding pattern-aware expert specialization.

W5. Effectiveness of positional embeddings.

We thank the reviewer for raising this important point. While recent studies [1] have questioned the utility of positional embeddings in time-series models, such observations often refer to step-wise or global fixed embeddings used in token-level Transformers. In contrast, TFPS adopts learnable patch-wise positional embeddings, specifically tailored to its patchified input structure, inspired by the design in PatchTST. This design helps encode relative segment positions and enhances inter-patch distinction.

To assess their effectiveness, we conduct an ablation study by removing the positional embeddings entirely. As shown below, this leads to consistent performance degradation across all datasets, with up to a 6% increase in MSE on ETTh1. These results confirm that positional embeddings provide a meaningful inductive bias in TFPS’s patch-level representations.

[1] Zeng A, Chen M, Zhang L, et al. Are transformers effective for time series forecasting? AAAI, 2023.

Dear Reviewer k7h4,

I hope this message finds you well. As the discussion period is nearing its end with less than three day remaining, we would like to ensure that we have fully addressed your concerns.

Following your valuable suggestions, we have made several improvements to the paper:

• We have clarified the figures to enhance readability.
• We have added new experiments to further validate the effectiveness of TFPS.
• We have included additional ablation studies to support the design choices and justify the contributions of individual components.

If there are any remaining concerns or further feedback you would like us to consider, please do not hesitate to let us know. Your insights are greatly appreciated, and we are eager to make any necessary refinements to strengthen our work.

Thank you once again for your time and thoughtful review.

We sincerely thank the reviewer for the encouraging feedback and for highlighting our contributions in pattern-specific expert modeling, dual-domain clustering, and adaptive subspace affinity design. Below we give a point-by-point response to your concerns and suggestions.

W1. On the expressiveness of MLP-based experts.

We appreciate the reviewer’s comment. While we explored CNN-based and LSTM-based alternatives, we found that the original MLP-based experts in TFPS actually yielded better forecasting performance across most datasets.

This observation highlights that increased architectural complexity does not always lead to better results. Specifically:

• Task alignment: MLP-based experts better match the patch-level granularity and localized modeling requirements of TFPS.
• Training stability: Their simplicity promotes more stable training and better generalization, especially when combined with the subspace-guided routing mechanism.
• Modular extensibility: Although our current design uses MLPs, TFPS can flexibly accommodate more complex expert architectures when needed.

TFPS’s primary innovation lies in how it adaptively assigns patches to specialized experts via latent subspaces, rather than overparameterizing the experts themselves.

Q1. Generalization to emerging or unseen patterns.

We thank the reviewer for raising this important and exploratory question. TFPS addresses non-stationarity by routing input patches to specialized experts based on learned subspace affinities. This allows the model to flexibly capture evolving dynamics and distributional shifts in real-world time series.

However, for entirely novel patterns outside the training distribution, additional adaptation mechanisms may be needed.

We consider this an important direction for future work. Approaches such as test-time entropy minimization, meta-learning, or continual learning could be integrated to enhance generalization to unseen patterns.

We will outline this in the revised Limitations section.

Q2. On sensitivity to the number of subspaces K.

We thank the reviewer for raising this important question. The number of subspaces $K$ determines the granularity of expert specialization in TFPS. In our experiments, we observe that datasets with stronger distribution shifts benefit from larger $K$, enabling better disentanglement of heterogeneous patterns. On more stationary datasets, a smaller $K$ is typically sufficient.

While $K$ is currently selected via validation performance, TFPS demonstrates robustness across a range of values, and does not require precise tuning to perform well.

Additionally, we have explored the use of Dirichlet Process Gaussian Mixture Models to estimate the number of latent patterns in an unsupervised fashion. This ongoing work aims to further automate subspace discovery, and we plan to integrate it into future iterations of the framework.

We sincerely thank the reviewer for the thoughtful feedback and positive recognition of our design in modeling distribution shifts via dual-domain encoders and subspace-specific experts, as well as the thorough empirical validation. Below we give a point-by-point response to your concerns and suggestions.

W1 & Q1. On sliding window vs. hard segmentation.

We agree with the reviewer that hard segmentation may lead to degraded forecasting performance. We would like to clarify that TFPS operates under a sliding window setting rather than hard segments. As described in $\underline{\text{Section 3.3}}$, we follow the patching strategy of PatchTST, where the stride $S$ defines the overlap between adjacent patches.

To directly assess the reviewer's concern, we conducted additional experiments using non-overlapping (hard) segmentation, where each patch is strictly isolated. As expected, due to the loss of temporal continuity, performance deteriorated:

These results confirm that preserving partial overlap between patches improves the model's ability to capture longer-term dependencies and supports more stable expert assignment—both of which are critical for accurate forecasting in TFPS.

W2. Discussion on the time-series input.

We thank the reviewer for raising this important point. We agree that handling irregularly sampled or sparse time series is crucial for real-world applicability.

While TFPS currently assumes regularly sampled sequences, its modular architecture allows for future extension. Specifically:

• Time-stamp encoding and attention masking can be integrated into the encoders to directly handle missing or unevenly spaced data;
• Interpolation-based preprocessing can reconstruct uniformly sampled inputs without altering the patching mechanism.

These strategies would enable consistent patch segmentation while modeling uncertainty introduced by irregular sampling, making TFPS applicable to a broader range of real-world scenarios. We will outline this in the revised Limitations section.

W3 & Q3. On the choice of $\alpha$.

In practice, only $\alpha$ and $\beta$ require tuning, while other parameters are fixed across all experiments and do not need to be adjusted during deployment.

• As shown in $\underline{\text{Appendix J}}$, we performed a - sensitivity analysis and selected $\alpha, \beta \in [10^{-4}, 10^{-2}]$, a range chosen to maintain balanced magnitudes across loss terms and ensure stable optimization. Within this range, performance remains robust across datasets.
• We set $\alpha_t = \alpha_f = 10^{-3}$ to achieve a reliable trade-off between convergence and regularization, which also improves reproducibility by eliminating the need for extensive tuning.

We observed similar robustness for $\beta$, simplifying practical deployment.

W4. On IMP calculation and the innovation.

We thank the reviewer for this valuable comment. Based on comparisons with the second-best baseline, TFPS still achieves a relative MSE improvement of over 2.3% on the ETTh2 dataset, along with competitive results on other benchmarks.

Beyond the absolute MSE values, the key strength of TFPS lies in its architectural design. Unlike traditional models that adopt a Uniform Distribution Modeling approach, TFPS introduces a divide-and-conquer paradigm that adaptively decomposes the input distribution into latent sub-patterns and assigns them to specialized experts for targeted modeling. This enables the framework to learn more flexible projection functions that capture complex relationships between historical inputs and future predictions.

This design not only improves forecasting accuracy, but also enhances robustness to distribution shifts, improves interpretability through expert specialization, and ensures greater stability across heterogeneous scenarios. These benefits are often underappreciated or entirely overlooked by monolithic forecasting models.

Q2. On the rationale behind linear projection and positional encoding.

• First, following PatchTST, the linear projection is a crucial step that maps raw patches into a Transformer latent space of dimension $D$. This produces denoised and semantically enriched embeddings, which are more robust for downstream modeling.
• Second, positional encoding is essential, even in the frequency branch, as it introduces temporal order awareness prior to FFT. This allows the frequency encoder to distinguish between spectrally similar but temporally misaligned patterns. Empirically, we validate this design with an ablation study removing positional encoding:

Unlike traditional methods that directly operate on raw FFT values, TFPS integrates frequency modeling into a learnable latent space, enabling joint optimization of temporal and spectral semantics, which is an important innovation of our framework.

Q4. On temporal dependencies across patches and expert isolation.

TFPS preserves the ability to model temporal dependencies across patches via two mechanisms:

• Time and frequency encoders process the entire patch sequence using attention and feedforward layers, establishing early-stage temporal interactions before expert routing.
• After expert inference, outputs from all patches are concatenated and passed through a shared linear projection, which acts as a global decoder to fuse inter-patch information.

Q5. On dual-domain expert modeling and alignment.

We thank the reviewer for the insightful question. To clarify, TFPS does not assume or enforce alignment between time-domain and frequency-domain expert assignments.

Each domain operates independently:

• The time experts captures local, position-sensitive variations (e.g., trends, shifts);
• The frequency experts captures global, position-invariant structures (e.g., periodicity, harmonics).

Figure 5 visualizes representative inputs assigned to specific time-domain experts for interpretability, it does not imply cross-domain alignment.

This dual-branch design allows the model to leverage complementary inductive biases, and both representations are fused only at the final prediction stage.

Q6. On the purpose and necessity of KL divergence.

We thank the reviewer for the thoughtful question regarding the role of KL divergence in our framework. The KL divergence loss in TFPS is not meant to reconstruct $S$, but to refine and stabilize subspace assignment through regularization.

• $S$ is computed from raw latent features and may contain noisy, uncertain affinities—especially early in training.
• $\hat{S}$ is a sharpened version that emphasizes high-confidence entries via temperature-based sharpening.

By minimizing $\text{KL}(\hat{S} \parallel S)$, we:

• Encourage $S$ to concentrate on confident subspace relationships,
• Suppress noisy or ambiguous affinities,
• Stabilize expert routing and improve subspace consistency.

This loss acts as a structural regularizer rather than a reconstruction loss. Importantly, we empirically validate its necessity:

These results show consistent performance drops when removing the KL term, supporting its necessity.

Many thanks for your prompt response and further clarifying your concerns.

Hyperparameters in TFPS

TFPS introduces a minimal number of hyperparameters, each grounded in clear modeling goals and shown to be robust across diverse datasets. Its components are essential for handling non-stationarity and patch-level distribution shifts, discussed below:

1. Number of hyperparameters: While TFPS includes several components, the number of hyperparameters that need to be tuned remains small.
2. Justification for selection: Though deriving optimal values for $\alpha$ and $\beta$ is intractable, both serve well-defined modeling purposes:

• $\alpha$ promotes orthogonal, compact subspaces;
• $\beta$ stabilizes routing by aligning refined and raw affinities.

3. Generalization across datasets: TFPS targets evolving patterns and patch heterogeneity, not domain-specific overfitting. We use fixed $\alpha$ and $\beta$ across 9 datasets, showing strong generalization without per-dataset tuning.
4. Necessity of components: The claim that these structures contribute little is contradicted by Table 2, where removing the Pattern Identifier or MoPE consistently degrades performance. Figure 5 further demonstrates expert specialization, validating their utility.

Position embedding for frequency

1. We follow FNet [1] in including a shared, additive positional embedding before the frequency encoder. This does not distort the spectral content—instead, it gives the frequency branch a weak “which patch am I?” signal, ensuring that spectrally identical segments that occur at different times aren’t collapsed into one. An ablation study shows that removing these positional embeddings degrades MSE:

2. The time-domain branch captures local, position-sensitive dynamics. The frequency-domain branch models global, position-invariant patterns, but needs a hint of “where” to avoid conflating distinct patches. Take $\sin(x)$ vs. $\sin(x + \pi/2)$: they share the same spectrum, so the frequency branch groups them by frequency—but without a small positional cue, it treats them as identical, losing the fact that one patch lags the other by $\pi/2$. The position embedding prevents this collapse, complementing the time branch rather than contradicting it.

Expert assignment in time domain

TFPS clusters in a learned latent space, not on raw signals, relying on the well-established assumption that semantically similar embeddings lie in low-dimensional subspaces.

1. Subspace assumption applies to latent features. We don’t assume raw signals are linear combinations—instead, the encoder (via our loss) embeds similar patterns into compact latent regions where subspace structure naturally emerges.
2. Proven in deep clustering. DEC [2] also perform soft clustering over latent embeddings (via a t-distribution kernel) without any linearity assumption in input space.
3. Inner-product routing. Our routing via $\mathbf{z}_t^\top \mathbf{D}^{(j)}$ mirrors modules such as CCM [3], which assign embeddings to learned prototypes by normalized inner products—a standard practice in neural clustering.

The purpose and necessity of KL divergence

While the raw subspace affinity matrix $S$ may be uncertain early in training, the KL loss between its sharpened version $\hat{S}$ and itself is not designed to blindly reinforce noise, but serves as an entropy-reducing self-regularizer, inspired by DEC [2] and TENT [4].

1. In DEC [2], a similar KL loss refines clustering by comparing current soft assignments with a sharpened target derived via normalized squaring. This informs our use of $\hat{S}$, which amplifies confident affinities and suppresses ambiguous ones, guiding structured expert assignment
2. Our method also draws from TENT [4], which minimizes prediction entropy during test-time to encourage confident decisions. Crucially, both TENT and our KL operate at the batch level, a key design that prevents overfitting to noisy samples. Batch-level regularization promotes consistent expert routing by leveraging shared structure across samples.
3. Additionally, we include a supervised MSE loss to align predictions with ground truth and correct early-stage misassignments. This dual-objective design enables KL to enhance clustering consistency while maintaining forecasting accuracy.

[1] FNet: Mixing Tokens with Fourier Transforms. NAACL, 2022.

[2] Unsupervised Deep Embedding for Clustering Analysis. ICML, 2016.

[3] From similarity to superiority: Channel clustering for time series forecasting. NeurIPS, 2024.

[4] Tent: Fully Test-Time Adaptation by Entropy Minimization. ICLR, 2021.

Many thanks for your prompt response and further clarifying your concerns.

Discussion of TFPS hyperparameters.

We thank the reviewer for emphasizing practical usability. In TFPS, each hyperparameter directly supports a core mechanism, they generalize without per-dataset tuning, and our ablations confirm their necessity:

1. Clear purpose. Every parameter is introduced to fulfill a specific modeling goal.
2. Robust defaults. Results on nine diverse datasets demonstrate TFPS’s strong generalizability.
3. Empirical necessity. Removing or varying these parameters in our ablation study consistently degrades performance, demonstrating that none are redundant.

By (a) grounding each hyperparameter in a modeling objective, (b) showing strong “no-tune” performance across datasets, and (c) providing ready-to-use defaults in our public code, TFPS remains both expressive and easy to adopt. We hope this clarifies the necessity and practical configurability of our hyperparameter set.

Pisition embedding.

Both TFPS and FNet (as implemented in fnet-pytorch) employ a small, learned positional embedding vectors:

# FNet:
self.position_embeddings = nn.Embedding(config['max_position_embeddings'], config['embedding_size'])

# TFPS:
W_pos = torch.empty((q_len, d_model))
nn.init.uniform_(W_pos, -0.02, 0.02)

These learned vectors, when added to the input and passed through the FFT, do not collapse into a single frequency. Instead, each embedding “fingerprint” disperses across multiple spectral coefficients, providing just enough “which patch am I” signal to break symmetry and prevent collapse—without substantially distorting the underlying spectral content.

Justifying temporal pattern subspaces.

Thank you for raising this important point. Unlike images, showing the same for temporal data is indeed challenging. Nonetheless, we observe that time-domain encoders in deep models learn to map series into structured latent spaces, where similar patterns cluster tightly, effectively forming separable and meaningful "subspaces."

Empirical and architectural evidence from recent work supports this view:

• In MoLE [1], different experts learn to fit distinct linear relationships, and the gate implicitly clusters inputs into those linear subspaces.
• DUET [2] explicitly applies clustering on encoder outputs to discover pattern groups, then leverages them for improved prediction. Its strong results confirm that meaningful subspaces exist in the learned feature space.
• Large-scale MoE models (e.g., ** Time-MoE** [3], ** Moirai-MoE** [4]) rely on routing different inputs or segments to distinct experts. Their success implies that the learned representations separate diverse patterns into distinct regions, so that each expert can specialize.
• TEST [5] shows that text prototypes can selectively activate corresponding time-series patterns, indicating that even semantic “meanings” form structured manifolds in representation space.

In summary, these findings demonstrate that, although not obvious a priori, deep encoders do induce separable subspaces for temporal patterns, which our MoPE module then discovers and exploits for expert-guided forecasting.

[1] Mixture-of-linear-experts for long-term time series forecasting. AISTATS, 2024.

[2] DUET: Dual Clustering Enhanced Multivariate Time Series Forecasting. KDD, 2025.

[3] Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts. ICLR, 2025.

[4] Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts. ICML, 2025.

[5] TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series. ICLR, 2024.

KL divergence.

The affinity $s_{ij}$ is computed per sample, but $L_{sub}$ is still applied over the entire batch:

kl_loss = F.kl_div(s, s_tilde)

The supervised MSE loss backpropagates any large forecasting error through both the encoder (into $z_i$) and the expert subspace bases $D$. If a patch is mis‐assigned and yields high MSE, its gradient nudges those affinities so that, in the next iteration, $L_{sub}$ will sharpen toward the correct expert.