Learning to Factorize Spatio-Temporal Foundation Models

Thank you for your insightful questions and suggestions. We have revised and expanded our explanations to address your concerns.

Q1 (Weakness - Scope/Clarity): Could the authors explicitly clarify that the specific research focus of this work is forecasting? Moreover, the term "factorized" is unclear—how does the two-stage framework reflect factorization?

A1: Yes, FactoST is specifically designed for spatio-temporal forecasting. The term factorized refers to our core innovation: unlike existing STFMs that perform joint spatio-temporal pretraining, FactoST decouples learning into two sequential stages—first extracting general temporal patterns (UTP), then injecting domain-specific spatial context (STA). This separation explicitly factorizes temporal and spatial learning, in contrast to standard pretraining/fine-tuning where both dimensions are entangled from the start.

As shown in Table 1, this factorization offers three key advantages: (1) UTP eliminates spatial parameters, reducing pretraining cost; (2) STA adapters constitute only a small fraction of total parameters, enabling fast adaptation; (3) the framework maintains cross-domain versatility while achieving superior few-shot performance. Figure 1(c) illustrates how this design avoids the spatial generalization bottleneck of joint learning approaches.

Q2 (Weakness - Generalizability): Is the STA method limited to GNN-based backbones? Has it been tested on other types of architectures, such as the more common encoder-decoder structures? If restricted to GNNs, the generalizability may be limited.

A2: No, STA is architecture-agnostic and does not require GNN backbones. It operates at the feature level, using ST metadata fusion, dynamic ST filtering, hierarchical domain alignment, and continual memory replay to offer a lightweight spatio-temporal modeling solution. This design allows it to work with any temporal backbone architecture, as it processes feature embeddings regardless of the source model.

To validate this, we integrated STA into PatchTST, a non-GNN, time-series-focused model, and evaluated its few-shot performance on long-term tasks. The results (below) show that PatchTST w/ STA outperforms vanilla PatchTST across all datasets, confirming STA’s ability to generalize beyond GNNs. FactoST itself still achieves superior results, which we attribute to its pretrained temporal backbone’s universal pattern-learning capabilities. This demonstrates STA’s broad compatibility and strengthens the generalizability of our framework.

Q3 (Weakness - Experiments/Baselines): Please include more expert models for a comprehensive evaluation. Why doesn’t the method achieve SOTA on all datasets? Are the dataset protocols standard? How does pretraining data volume affect downstream performance?

A3:

• Additional Baselines: As suggested, we include two recent baselines (TimeMixer, BigST) for a more comprehensive comparison. Results (below) show FactoST tops seven of the eight datasets, demonstrating its competitiveness. This aligns with our core finding that the factorized UTP+STA design outperforms both specialized and foundation models in most scenarios.

  Dataset	FactoST (MAE/RMSE)	BigST (MAE/RMSE)	TimeMixer (MAE/RMSE)
  PEMS-03	28.57 / 46.78	51.87 / 75.56	47.86 / 71.52
  PEMS-04	42.04 / 64.89	52.37 / 80.23	58.44 / 86.67
  PEMS-07	45.60 / 72.47	54.92 / 82.12	67.75 / 100.45
  PEMS-08	35.69 / 56.15	58.68 / 86.76	45.10 / 65.53
  PEMS-Bay	2.96 / 6.21	2.93 / 6.20	4.11 / 8.94
  ETTh2	0.358 / 0.561	1.164 / 1.797	1.013 / 1.646
  Electricity	0.265 / 0.409	0.481 / 2.843	0.459 / 2.833
  Weather	0.087 / 0.276	0.892 / 1.522	0.720 / 1.401

  We will cite the references you provided ([1] OpenSTL by Tan et al., 2023; [2] Deep Time Series Models by Wang et al., 2024) in the revised manuscript and incorporate their insights into our discussion of related work and future directions.

• Performance Gap Analysis: The minor performance gaps on specific datasets highlight important trade-offs in model design. On the PEMS-07 (short-term), FactoST slightly underperforms D2STGNN, which explicitly decouples traffic flow into diffusion and inherent signals—a fine-grained modeling approach highly effective for its dense topology. However, this comes at high computational cost, causing OOM failures in long-term forecasting and limiting practicality. On datasets like ETTh2 that lack spatial structure, Moirai performs better as it is is specifically optimized for general time series without spatial priors, while FactoST is designed for spatio-temporal settings. Nevertheless, FactoST uses significantly fewer parameters (1.4% of Moirai’s), runs faster, and still outperforms other STFMs such as UniST and OpenCity. Future work will explore lightweight spatial modeling for complex scenarios, with further discussions in Q2/A2 and Q4/A4 of Reviewer jNeH.

• Dataset Protocol: We follow strict domain separation. UTP pretraining uses the Monash dataset collection; evaluation uses standard benchmarks (PEMS, etc.) with the widely adopted 6:2:2 train/validation/test split. For few-shot settings, we sample 10% of training data, resulting in a 0.6:2:2 effective split.

• Pretraining Data Impact: To evaluate scalability, we conducted an ablation study on ETTh2 (long-term forecasting), varying the UTP pretraining data volume from 20% to 100%. Results show that performance improves monotonically with more data, confirming FactoST benefits from larger pretraining corpora. Notably, our current dataset (13M points) is much smaller than those used by existing foundation models (see Q4/A4 of Reviewer rPHb), indicating potential for further improvement with more high-quality data.

  Data Volume	Zero-shot (MAE/RMSE)	Few-shot (MAE/RMSE)	Δ (%) Zero-shot vs. Full	Δ (%) Few-shot vs. Full
  20%	0.405 / 0.598	0.394 / 0.592	+2.53% / +1.70%	+2.34% / +1.72%
  40%	0.402 / 0.596	0.393 / 0.591	+1.77% / +1.36%	+1.82% / +1.55%
  60%	0.400 / 0.594	0.391 / 0.589	+1.27% / +1.02%	+1.30% / +1.21%
  80%	0.397 / 0.589	0.389 / 0.586	+0.51% / +0.17%	+0.78% / +0.69%
  Full	0.395 / 0.588	0.385 / 0.580	- / -	- / -

Q4 (Weakness - Model Design): Could the authors explain why only daily and weekly periodicity are considered, while quarterly and yearly periodicities are not taken into account?

A4: FactoST supports flexible multi-scale periodicity modeling. The current implementation focuses on daily/weekly cycles because they are dominant in the high-frequency traffic and energy datasets used for evaluation (e.g., 5-minute traffic data in PEMS-03). However, the architecture is designed to be general and can incorporate longer-term patterns (like monthly or annual cycles) by utilizing timestamp features. The STMF module can encode longer cycles by expanding time-of-day/day-of-week embeddings to include month-of-year/yearly features, with no architectural changes. We will clarify this generality in the revised manuscript.

To validate this flexibility, we conducted a few-shot experiment on PEMS-03 using "month-of-year" features instead of daily/weekly cycles. As shown below, performance degrades significantly (short-term MAE ↑12.94%, long-term MAE ↑15.20%), due to a mismatch between the dataset’s high temporal frequency and the long-term nature of monthly features. This outcome confirms that temporal feature selection must align with the data's inherent frequency—a design principle our framework inherently supports.

Q5 (Weakness - Efficiency): Smaller models usually have faster inference—why is FactoST slower than larger models like Informer and FEDformer?

A5: While FactoST has fewer parameters than models like Informer/FEDformer, its inference is only slightly slower due to the necessary computations in the STA module—spatio-temporal metadata fusion, sparse interaction modeling via filtering, and hierarchical domain alignment—which adapt the universal temporal backbone to specific spatial contexts. This modest cost is justified by the substantial performance gains and the fact that FactoST remains significantly faster than joint STFMs (OpenCity, UniST). The trade-off strongly favors FactoST, as it achieves the best overall performance-efficiency balance among the evaluated models (Figure 5).

We are deeply grateful for your comprehensive feedback, which has greatly helped us strengthen the paper's clarity and rigor. We will carefully revise the manuscript to include the additional experiments and clarifications you suggested.

I acknowledge the authors' response to my comments. The rebuttal has adequately addressed my concerns, and I support the acceptance of the paper.

Thank you for recognizing our work. Below, we hope to resolve some of your doubts.

Q1: The UTP dataset spans six major domains, but data quantity and quality may vary significantly across them (e.g., energy vs. health). Have the authors investigated how such imbalances affect generalization performance?

A1: FactoST addresses potential domain imbalance through dual strategies:

• Data-level: For domains with limited data, multi-frequency augmentation is applied to synthetically expand temporal patterns. Consistent preprocessing (outlier removal, gap handling) is applied across all data to ensure quality consistency.
• Model-level: A multi-domain prompting mechanism (Equation 2) dynamically adjusts the influence of different domains using adaptive weights (α_j). Ablation experiments (Figure 3) and visualizations (Figure 6) demonstrate its effectiveness in learning domain-specific knowledge, mitigating the impact of imbalances and improving generalization.

Q2: The paper notes the limited zero-shot capabilities of current STFMs. While FactoST focuses on few-shot adaptation, can the authors suggest directions for improving zero-shot transfer?

A2: To explore directions for improving zero-shot performance, we integrated the Test-Time Computing (TTC) method proposed in [1] into our framework. This approach performs online adaptive optimization during the testing phase to correct model predictions. It works as follows:

1. It maintains a FIFO memory queue to cache gradually predicted inputs, predictions, and ground-truth labels during testing;
2. A calibrator is constructed to transform queue-based historical predictions into the frequency domain via FFT, using group-learned amplitude/phase offsets to dynamically correct non-stationary temporal deviations;
3. Loss is calculated solely from queue-sampled historical predictions to update calibrator parameters, strictly preserving temporal consistency to prevent information leakage.

The integration of TTC effectively boosts long-term zero-shot performance, as shown in the results below. On average, it reduces MAE and RMSE by 7.72% and 8.79%, respectively, demonstrating that online adaptation is a viable direction.

Future directions for zero-shot improvement include: (1) spatial meta-learning to encode universal topological features like distance decay [2]; (2) semantic-enhanced spatial priors using external knowledge [3]; (3) cross-domain latent space alignment to bridge spatial representation gaps [4].

[1] Learning with Calibration: Exploring Test-Time Computing of Spatio-Temporal Forecasting, arxiv 2025

[2] SPOK: tokenizing geographic space for enhanced spatial reasoning in GeoAI, IJGIS 2025

[3] Geolocation Representation from Large Language Models are Generic Enhancers for Spatio-Temporal Learning, AAAI 2025

[4] Unveiling the Inflexibility of Adaptive Embedding in Traffic Forecasting, Arxiv 2024

Q3 (Weakness): While the proposed adapter improves efficiency, the potential gains from more complex spatial modeling are not fully explored.

A3: STA employs feature-level dependency modeling: it dynamically identifies key correlations and suppresses noise through interactions between input features and learnable identifiers, while reducing computational complexity via low-rank projection and sparse strategies.

Its advantages include: simultaneously capturing spatial proximity (S_s), temporal coherence (S_t), and delayed causalities (S_d), breaking the limitation of traditional models that only model single or two dimensions; dynamically adjusting the contribution weights of spatial, temporal, and delayed relationships through learnable parameters to adapt to different data characteristics.

Complex spatial modeling remains underexplored, as more sophisticated graph modeling mechanisms (e.g., D2STGNN) incur extremely high resource consumption (OOM) in long-term prediction (Table 3), restricting practical usability. As shown in our added experiment (Q2/A2 of Reviewer UUgZ Q2), the current STA design is already effective and architecture-agnostic, offering a good balance. Future work will explore more lightweight methods for modeling high-order spatial dependencies.

Q4 (Limitation Discussion): The paper briefly mentions limitations related to dependency on the pretraining corpus and handling evolving spatial structures. A more thorough discussion would strengthen the work.

A4: We appreciate this valuable feedback. Addressing the limitations related to pretraining corpus dependency and evolving spatial structures is crucial for future work. Our current framework offers several avenues for extension:

1. Integration of Exogenous Variables: Enhancing FactoST to effectively integrate and adapt to external factors (e.g., weather, events) in a few-shot manner is an important direction, as this capability remains underexplored in STFMs.
2. Handling Dynamic Spatial Nodes: Future research could focus on making FactoST robust to real-time changes in the number and identity of spatial nodes (e.g., sensors added/removed), ensuring generalization to new environments while preserving pretrained knowledge.
3. Adaptive Model Composition: Leveraging the factorized design, FactoST could be extended to adaptively select components based on data characteristics. By evaluating temporal stability and spatial distinctiveness, the system could dynamically choose between full spatio-temporal modeling, temporal-only modeling, or specialized modules for highly dynamic scenarios, enhancing robustness.

We sincerely appreciate your thoughtful questions and valuable suggestions on future directions. We will expand the discussion on zero-shot learning and model limitations as recommended.

Thank you for recognizing the effectiveness, novel design, and rigorous evaluation of our work. Below we endeavor to address your questions.

Q1: How does the proposed method effectively handle inputs with different temporal scales, such as 12 and 96 time steps?

A1: FactoST employs a Transformer encoder-decoder architecture. During the UTP stage, it utilizes both MLP prediction heads and linear reconstruction heads to support multi-horizon forecasting (e.g., 12/96 steps), enabling robust temporal modeling. This flexibility is achieved through patch-based tokenization, where input sequences are divided into non-overlapping patches, allowing adaptive processing of both short-term and long-term inputs. In the subsequent STA stage, only the encoder and prediction heads are fine-tuned. Horizon-specific prediction heads are initialized based on the target sequence length, allowing the model to effectively manage varying temporal scales.

Q2: Does the spatio-temporal adaptation module participate in the pretraining process, or is it only involved in few-shot learning?

A2: The STA module participates only in few-shot fine-tuning and is explicitly excluded from pretraining. During UTP, FactoST learns universal temporal patterns without any spatial modeling. As detailed in Section 3.2, spatial information is injected later via lightweight adapters in the STA stage, preserving the pretrained temporal capabilities. This factorized learning approach is further explained in Q1/A1 of Reviewer UUgZ.

Q3: During pretraining, are all models (including foundation models and expert models) (re)trained using the same pretraining dataset? Or do the foundation models require only few-shot learning?

A3: No, the models are not retrained on the same pretraining dataset.

• Foundation Models (e.g., FactoST, UniST, Moirai): These are pretrained on their respective, distinct cross-domain corpora (e.g., FactoST's UTP uses Monash; UniST uses grid data; TimesFM uses large-scale time series data). They are then evaluated via few-shot adaptation on downstream datasets.
• Expert Models (e.g., STID, PatchTST): These are typically trained from scratch on individual target datasets, without any cross-domain pretraining.

Therefore, foundation models undergo extensive pretraining on their specific corpora and are subsequently adapted (not retrained from scratch) via few-shot learning on target tasks.

Q4 (Weakness): The analysis of the model's zero-shot experiments is limited, with the model's zero-shot performance being comparable to some existing foundation models. What are the potential reasons for this?

A4: The zero-shot performance of STFMs is fundamentally limited by domain-specific spatial heterogeneity. Notably, FactoST (i.e., UTP without STA) and TimesFM, which lack explicit spatial learning, already outperform STFMs like OpenCity and UniST in zero-shot scenarios. This validates our core motivation: joint spatio-temporal modeling during pretraining can hinder generalization across domains with differing spatial structures.

Furthermore, the pretraining corpora vary significantly in domain and scale:

Despite its significantly smaller scale, FactoST's zero-shot performance remains competitive, demonstrating the effectiveness of our UTP's multi-frequency augmentation, multi-domain prompting, and multi-task pretraining.

We also acknowledge that there is significant room for improvement in zero-shot performance. To this end, we have explored future directions, such as Test-Time Adaptation (TTA), which can further boost zero-shot results. For a detailed discussion on these potential extensions, please see Q2/A2 of Reviewer jNeH.

Q5 (Weakness): In few-shot learning experiments, almost all test dataset task types are included in the pretraining set, which limits the validation of the model's cross-task performance.

A5: Although the pretraining and few-shot datasets share overlapping semantic domains, they differ substantially in spatial granularity (e.g., regions vs. sensors), temporal resolution (e.g., hours vs. minutes), and task objectives (e.g., pedestrian count vs. traffic speed). This significant difference in data characteristics demonstrates the model's true cross-task generalization ability, even if the high-level semantics overlap. Detailed dataset statistics are provided in Appendix A.1.1.

Furthermore, UTP's multi-frequency augmentation explicitly enhances temporal pattern diversity, improving cross-task transferability. The factorized design (UTP for universal patterns, STA for domain-specific adaptation) inherently improves cross-task adaptability compared to joint pretraining approaches.

Q6 (Weakness): The experimental configuration details are insufficient. For example, specific configurations for both foundation and non-foundation models during the pretraining phase are not clearly described.

A6: Comprehensive experimental configurations for both the UTP and STA stages are detailed in Appendix A.1.2, including model architectures, training procedures, and parameter settings (e.g., 3 encoder/decoder layers, 16 attention heads, latent dimension 128). For baseline models, hyperparameters followed the original papers' defaults or were tuned via grid search on validation sets (Appendix A.3). All baselines were evaluated within a unified framework using identical data splits to ensure fair comparison.

Thank you again for your constructive feedback. We will incorporate all the suggested clarifications and improvements into the revised manuscript.

Thank you for recognizing the good motivation, clear design, comprehensive evaluation, and unique innovation of our paper. Below are our responses to your major questions.

Q1 (Weakness): What is the contribution of L_pred and L_recon during pretraining? Are there any weights associated with these losses in Equation 3 and Equation 4?

A1: L_pred supervises future sequence prediction, capturing both short-term dynamics and long-term trends. L_recon enhances multi-frequency pattern modeling through masked frequency reconstruction. The two losses are complementary, so we adopt equal-weighted joint optimization (L_pretrain = L_pred + L_recon) without separate weighting.

To validate their effectiveness, we conducted an ablation study on zero-shot and few-shot long-term forecasting for ETTh2. Results show that removing L_recon degrades performance: zero-shot MAE/RMSE increase by 3.54% and 5.27%, respectively, while few-shot MAE/RMSE increase by 0.78% and 0.87%. This highlights the importance of self-supervised multi-frequency reconstruction for learning robust temporal representations.

Q2 (Weakness): Zero-shot evaluation is a crucial aspect of assessing foundational models, but FactoST has not been evaluated in this setting.

A2: Zero-shot evaluation is indeed crucial and has been performed. Comprehensive results are presented in Appendix A.4 (Table 6), covering both short- and long-term forecasting. We further discuss the limitation of the zero-shot capability in Q4/A4 of Reviewer rPHb and propose some future directions in Q2/A2 of Reviewer jNeH.

Q3 (Weakness): The batch size of 16,384 was selected for pretraining to ensure stable optimization. However, what does "stability" specifically refer to in this context? Given the large and computationally intensive nature of this batch size, was the impact of different batch sizes on performance explored?

A3: "Stability" refers to a balance between training speed and the observability of the optimization process. We tested batch sizes of 8,192, 16,384, and 24,576. All converged to similar loss values with no significant impact on final downstream performance. We chose 16,384 because: 8,192 resulted in too many iterations (slow convergence), while 24,576 resulted in too few iterations, making it difficult to monitor loss changes. Thus, 16,384 provided the optimal balance for managing the training process.

Q4 (Weakness - Typo/Misc): Abbreviation MAE is used (line 11) before its full form "Mean Absolute Error" (line 547). On line 262, "Mean Square Error (MSE)" is used instead of MAE.

A4: Corrections have been made in the revised manuscript: "Mean Absolute Error (MAE)" is now defined at its first appearance (line 11), and the typo on line 262 (MSE corrected to MAE) has been fixed. All technical terms are now consistently defined upon first use.

Thank you for your meticulous suggestions. We will address all your comments, including the experimental details and typographical corrections, in the revised manuscript.