VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters

Thank you for your encouraging response. We are delighted that you find our paper novel, with solid experiments and the performance is noteworthy. Below are our responses:

Claims And Evidence: However, it is not very convincing to me since the domain gap between these two modalities. Further empirical evidence may be needed to justify why the pre-trained MAE can be used even in a zero-shot way to perform time series forecasting.

• As you noted, we are the first to leverage a pre-trained vision model for zero-shot forecasting. We understand that ground-breaking ideas often require time to gain community endorsement. As an initial exploration, we've conducted extensive experiments to validate VisionTS's effectiveness. To our knowledge, our evaluation benchmark is the largest among existing TSF foundation models.

• To explore the domain gap between the modalities further, we visualize their similarities in Fig 7. We find notable heterogeneity within time-series data across domains, with images potentially "bridging" these isolated time-series representations, which might explain why VisionTS outperforms some cross-domain TSF models.

• We welcome any suggestions that could further strengthen the persuasiveness!

Essential References Not Discussed: The experimental comparison lacks one highly related work CALF [CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning, AAAI2025], which is the existing SoTA LLMs-based time series forecasting works. How is the performance of the proposed VisionTS compared with CALF?

• We compare the full-shot results reported by CALF with VisionTS's zero-shot results below:

• We can observe that VisionTS, even in a zero-shot scenario, proves comparable to CALF in full-shot settings. It highlights the greater transferability of visual modality to time series over textual modality.

Other Comments: The LLM have been recently proved not very necessary for time series [Are Language Models Actually Useful for Time Series Forecasting? NIPS2024]. So I wonder whether the vision model also suffers the similar situation?

• We agree that text modality may offer limited benefit to time series. Following the paper's ablation study, we removed or replaced VisionTS's visual backbone with simpler modules. Appendix D.3 shows these changes degrade performance, indicating visual knowledge is indeed beneficial for TSF, unlike textual knowledge.

We hope these responses can fully address your concerns. Thank you once more for your detailed feedback!

Thank you for your positive comments on our work. We are pleased that you find our paper novel and well-experimented, aiding in understanding the workings of TSF foundation models. Here are our responses to your concerns:

E1: Gift-Eval is the most comprehensive benchmark of the utilized benchmarks.

E2: Recent work further highlights problems of the respective long-term benchmark. Therefore, I would suggest emphasizing and discussing the gift-eval and monash in more depth instead of mostly highlighting the long-term benchmarks.

• We fully agree that LTSF has limitations. Since this ground-breaking viewpoint has recently emerged, before it is widely accepted by the research community, we believe it is still necessary to test widely-used benchmarks. As you can see, other reviewers are still interested in the LTSF datasets.

• We also agree on the need to enhance presentation of solid benchmarks. We will add the GIFT-Eval results to the right side of the teaser (Figure 1) to further emphasize them and expand the discussion of their experimental results. Below is further discussion on the GIFT-Eval leaderboard which will be included in our final revision:

W1: Evaluation reporting might miss certain metrics and models (see Experimental Design & Analysis). I suggest the author should update the mentioned results.

For GIFT-EVAL, only the MASE metrics, but no WQL or average rank metrics are reported.

• We report MASE but not CRPS (WQL) or average rank, since these are probabilistic metrics, not point metrics (note that the average rank is based on CRPS-sorted results). Due to limitations of the MAE model, VisionTS is not a probabilistic forecasting model, as mentioned in Section 6. Comparing the CRPS metric of point-forecasting models with probabilistic models is unfair, as the former's CRPS tends to be significantly worse. If we only consider the point forecasting model (e.g., TTM r1/r2 and Timer) on the leaderboard, VisionTS significantly outperforms them in both MASE and CRPS.

Some methods that appear in the evaluation (e.g. TTM and Chronos) are not cited. Although the leaderboard is even linked in the paper, some models outperforming VisionTS (ttm, chronos bolt) are not reported. Although some might be concurrent work, updating the results would be beneficial.

• Thank you for your reminder and suggestion. We will cite all the referenced papers and update our results to include those concurrent works, even though some lack published papers, as noted by Reviewer TiNp and mentioned in Line 297. We also highlight that TTM's "superior results over VisionTS" were achieved by fine-tuning on the GIFT-Eval training dataset, not as a zero-shot model. Its zero-shot capability is significantly weaker than VisionTS, as shown in the leaderboard and already referenced in Figure 4 of our paper.

• We would also like to note that there may be data leakage issues for these concurrent works. For instance, both TimesFM 2.0 and Chronos-bolt used the M4 dataset, while TimesFM also utilized the Weather for pretraining. In contrast, visual MAE was trained on ImageNet, long before gift-eval, which can ensure no leakage.

W2: Approach seems to not to scale with bigger models (see results on different MAE size)

• One explanation is that larger vision models tend to memorize image-related details, leading to overfitting and harmful to time series forecasting. Moirai (ICML 2024 oral) also exhibited similar behavior, where larger models perform worse on out-of-distribution data (See Table 6 in [1]), with even more severe degradation than VisionTS. This is understandable given the disparity between image and time series modalities. We believe future adaptations in time series domain could alleviate this issue.

• Additionally, we found that larger models are not without merit. For example, MAE (large) performs well on ETTh1, and MAE (huge) shows good results on Electricity. Exploring the scenarios for different MAE sizes is a promising research direction.

Q1: How would you use VisionTS to model time series data that exhibits multiple periodicity?

• We would assess potential periodicities based on sampling frequency and select the optimal P using the validation set. For time series without clear periodicity or complex multi-periodicity, we can try P=1, which can be effective in our experiments (Appendix C.5).

We hope these responses can fully address your concerns. Thank you once more for your detailed feedback, which greatly enhances the robustness of this paper.

[1] Unified Training of Universal Time Series Forecasting Transformers

Thank you for your invaluable review. We are delighted that you believe that our paper is novel and the experiment results are impressive. Below are our responses to the concerns:

W1: ablation studies for each of the new mechanisms.

Q1: An empirical study when other methods are fed with the time series of some good alignment and the results are rescaled.

• If "other methods" refers to zero-shot pretrained models, we kindly note that the alignment is applicable only to visual models, as it is meant to convert 1D time series into 2D formats. To the best of our knowledge, no existing TSF foundation models accept direct 2D input, making this alignment step inapplicable to existing models.
• If "other methods" refers to models trained from scratch, we conducted an ablation study using the same alignment but substituting the vision backbone with various models. Table 20 (Appendix D.3) indicates these substitutions significantly hurt performance and fail to achieve zero-shot forecasting, underscoring the vision backbone's crucial role in VisionTS. The mentioned SparseTSF (or TimesNet using similar alignment) cannot achieve such zero-shot forecasting as well.
• Beyond the alignment mechanism, we also introduce a new mechanism: using smaller standard deviations ($r$) during normalization. Thanks for your suggestions and we would like to investigate its contribution. The following table summarizes Moirai's performance with different $r$ (average MSE across four ETT datasets), indicating that $r$ significantly higher or lower than 1 lead to notable degradation. The reason is that image and time series distributions differ; the former is limited by color range while the latter is not. Therefore, this mechanism is unnecessary for other TSF foundation models. |$r$|0.4|0.6|0.8|1.0|1.2|1.5| |-|-|-|-|-|-|-| ||0.474|0.494|0.387|0.372|0.370|0.380|

Q2: An empirical study when a universal or an improper P is fed to VisionTS.

• Thank you for your suggestion. We have compared the results using P=1 on the ETT datasets in Table 7 (Appendix B.2). To further validate this, we tested P=1, 7, and 24 on the Monash dataset: |Proper P|P=1|P=7|P=24|LLMTime| |-|-|-|-|-| |0.729|0.931|0.957|1.102|1.041|

Results indicate that selecting an appropriate P based on sampling frequency is crucial for zero-shot forecasting.

Q: if considered part of the VisionTS, they undermine the zero-shot claim of the proposed method, as these practices are very lookback window-specific.

• We kindly note that existing zero-shot foundation models also have their own mechanisms to incorporate sampling frequency based on the specific lookback window. For example, Moirai selects an appropriate patch size based on the sampling frequency (see Appendix B.1 of [1]). TimesFM [2] even includes the sampling frequency as model input. We believe that leveraging prior data characteristics (e.g., sampling frequency and periodicity) to further enhance the performance of zero-shot models is also a promising research direction.

Comment 1: consider explaining why the current GIFT-Eval leaders are not mentioned (e.g., no corresponding publications).

• Thank you for your suggestion. As you noted, current leaders are concurrent works, some without publications (mentioned in Line 297). However, we plan to include them in the final paper version, as discussed with Reviewer cUEq. We would also like to note that there may be data leakage issues for these concurrent works. For instance, both TimesFM2 and Chronos-bolt used the M4 dataset, while TimesFM also utilized the Weather for pretraining. In contrast, visual MAE was trained on ImageNet and ensures no leakage.

Comment 2: Table 3 is misleading, as the speed up from VisionTS is due to the alignment and the reconstruction step, and it only shows up for >1k forecast length, possibly for batches that use a same P. Moirai and TimesFM are not the best decoder only reference either, e.g., TimesFM does not implement cached decoding properly.

• Thank you for your suggestion. For efficiency testing, our experimental settings are consistent with Moirai (refer to [1] Table 23), using longer forecast lengths. We choose Moirai and TimesFM for comparison since they are our baselines, and we will note in our final revision that VisionTS's evaluation uses a same P. To further address your concerns, we test shorter lengths, as shown in the table below.

The table shows that VisionTS's runtime is similar to these two models.

We hope these responses can fully address your concerns. Thank you again for your detailed feedback!

[1] Unified Training of Universal Time Series Forecasting Transformers

[2] A decoder-only foundation model for time-series forecasting

Thank you for your invaluable response. We are delighted that you found our paper novel, well-motivated, and with sufficient experiments and insights. Below are our responses to the concerns:

W1: How does VisionTS obtain the complete temporal information of visible patches during the alignment process?

• During the alignment process, the temporal information is encoded one-to-one to the spatial information after transformation. Specifically, the patch at the $i$-th row and $j$-th column corresponds to the time step $j\times N+i$ (with the input window scaled to $N(N-n)$ time steps in total). During the ViT processing, each patch receives a unique 2D positional encoding, enabling the model to capture spatial information and, consequently, the corresponding temporal information.

W2: In the zero-shot experiments in Tables 1 and 9, VisionTS underperforms compared to Moirai on half of the datasets (ETTm2, Electricity, and Weather). The authors should provide a detailed discussion and analysis of the underlying reasons.

• A possible explanation is that Moirai's pre-training data significantly overlaps with the domains similar to Electricity and Weather (e.g., Moirai pretraining containing 60% energy and 15% climate data). This possibly leads to data leakage. In contrast, VisionTS, using an MAE model trained solely on ImageNet, is free from such potential leakage.
• Additionally, this experiment is not to prove VisionTS is superior to Moirai; instead, we aim to show a purely visual model can achieve performance comparable to a native time series pretrained model. Maybe we can rephrase this statement as: In the zero-shot experiments, Moirai (trained on time series) underperforms compared to VisionTS (trained on images) on half of the time series datasets. This underscores the promising potential of vision models in TSF.

W3: Table 1 is suggested to include TTM [1] and Time-MoE [2] as baselines as well.

Thank you for your suggestion. We report the base and large model results for Time-MoE since the ultra model weights are unreleased. For TTM, we used the official HuggingFace model for replication. The following table summarizes the performance of various zero-shot foundation models.

We hope these responses can fully address your concerns. Thank you once more for your detailed feedback!