Multi-Modal View Enhanced Large Vision Models for Long-Term Time Series Forecasting

Dear Reviewer Ymur,

Thank you so much for the constructive feedback. We sincerely appreciate your valuable suggestions and questions. The following are our responses.

Q1: The claim that "applying LVMs to LTSF poses an inductive bias towards forecasting periods" is unclear. I am not convinced that LVMs are inherently good at seasonal modeling.

Thank you for pointing it out. By re-reading the draft, we became aware of this non-rigorous wording. Yes, we shouldn't say "LVMs poses an inductive bias towards forecasting periods", which may cause confusion since LVMs are not inherently designed for seasonal modeling. Instead, the bias comes from the period-based imaging method. We will clarify this difference in a revised version of the draft and emphasize the source of the bias is the period-based imaging method.

Q2: The sources of baseline results are inconsistent. It appears that the results for Time-LLM and GPT4TS are copied from VisionTS, while VisionTS results differ from the original paper. It would be better if the author could explain their selective use of results.

Yes, we'd like to clarify our criteria of reporting results. For Time-LLM and GPT4TS, we collected the results reproduced by the work in [*], which uses the standard evaluation protocol and re-implement the methods to confirm their reported results, adding confidence to the reliability of the results. Also, we have re-run both methods on selected datasets (ETT, Illness) and observed <1% differences, confirming consistency. For VisionTS, since it didn't use the standard evaluation protocol (its look-back window length varies, e.g., 1152 and 2304), we implemented it following the protocol (fixed look-back winodow as 336) for fair comparison. For CycleNet, we collected its results from its paper which uses the standard evaluation protocol. For other methods, we collected their results reproduced by [**] under the same protocol, which are generally better than those reported by their original papers. Since CycleNet didn't report results on Illness, we run it and report it below, which will be added to a revised version of the paper. Time-VLM didn't use Illness as well and released code recently. We will run it on Illness and add the results to the paper.

CycleNet performance in Illness datasets:

[*] M. Tan et al. "Are Language Models Actually Useful for Time Series Forecasting?". In NeurIPS, 2024. [**] M. Chen et al. "VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters". In ICML, 2025.

Q3: Compared to the most competitive baseline VisionTS, the performance gains are marginal. I would suggest conducting a statistical significance test or providing the standard deviation for both methods to better understand the performance gap.

Thank you for highlighting this concern. Yes, the challenge of improving MSE and MAE over the SOTA methods on the widely used benchmark datasets may raise doubts about the significance of improvements. To address this concern, we ran our method five times with different seeds and calculated standard deviations. In our appendix, Table 9 reports the standard deviations of DMMV on the same four datasets used for ablation and performance analysis (section 4.2, 4.3). In the following, we summarize the comparison between DMMV-A and the most competitive baseline, VisionTS.

From the results, both methods show small standard deviations across multiple runs, suggesting their stability and the reliability of the results. We ran two-sample t-tests for each dataset under different forecasting horizons (i.e., 96, 192, 336, 720). DMMV-A exhibits significant improvements over VisionTS in 13 out of 16 cases using MSE (p-value<0.05) and 9 out of 16 cases using MAE (p-value<0.05), totaling a majority of the cases. Also, considering the non-trivial difference in the overall number of wins between DMMV-A and VisionTS in Table 1, i.e., 43 and 9, we think the improvements are not a matter of randomness. We will extend this analysis to all datasets and add the results to a revised version of the appendix.

Additionally, we'd like to highlight another advantage of introducing TS view by our method: It alleviates the sensitivity to period estimation. When the period $P$ is perturbed (i.e., varying by P/6, 2P/6, ..., P), VisionTS's performance drops more (e.g., ETTh1: 0.409 → 0.582 (42.3%)) than our model (e.g., ETTh1: 0.395 → 0.450 (13.9%)), suggesting the better robustness of our method.

Q4: I am curious about the results if seasonal modeling is replaced by other numerical methods, like linear module used in TimeMixer. Would there still be significant gains from the usage of LVMs?

This is an interesting question, which can help validate the effectiveness of LVMs in our framework. We performed an ablation by replacing the LVM in DMMV-A (Figure 1(b)) with the seasonal mixing module of TimeMixer [*]. Specifically, we use this module to extract seasonal components from the look-back window and use DMMV-A's numerical forecaster to model the residual component. The forecasts from the numerical forecaster and the seaonal mixing module are aggrated by the gate function. The ablation results are summarized as following. As can be seen, the original DMMV-A outperforms the variants, suggesting the effectiveness of visual forecasters.

[*] S. Wang et al. "TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting". In ICLR, 2024.

Q5 in Limitations: As the authors acknowledge, the triple usage of LVMs could cause some computational overhead, though they present it as the best solution available at the time of submission.

Yes, we acknowledged this limitation in our appendix A. The triple usage includes two passes of LVMs for backcasting and one pass for forecasting. At the time of submission, we thought two passes were possibily the minimum usage of LVM for backcasting. Compared to other LVM-based models, which include at least one pass for forecasting, our method adds two backcast passes. In the following, we compared the inference time of our method and VisionTS. From the results, the computational overhead of DMMV-A looks practically acceptable. We will add these results and the discussion to a revised version of the paper. Moreover, we'd like to improve the model by optimizing the backcast-forecast mechanism using LVMs. Some thoughts are provided in appendix A.

The Averaged inference time for each sample in ETTh1:

Dear Reviewer Ymur,

Thank you for reviewing our response!

We humbly think the statistical tests with p-values in our previous response to Q3 are sufficient to validate that the improvements are not marginal.

Could you please let us know what might be missing in our previous response to Q3 so that we can take the opportunity to address the remaining concerns?

We hope by clarifying the concerns our work can meet the reviewer's standards for a positive rating. Thank you for your attention.

Dear Reviewer Ymur,

Thank you for the further response! We agree with the reviewer that the proposed model is not a panacea for each case and each dataset, like many SOTA models in this domain. However, we humbly think the overall improvement that the proposed method exhibits in the majority of the evaluation cases (across datasets, forecasting horizons, and evaluation metrics) is a reasonable indicator of non-trivial advancement, which we believe the reviewer also values.

We sincerely appreciate the reviewer's recognition of this work's merits, including the finding on the bias of the SOTA LVM-based forecasters and the effective solution to address the limitation by a backcast-residual based adaptive decomposition framework, which may inspire further developments. We also thank the reviewer for reconsidering the rating based on our discussion.

Dear Reviewer KH6N,

Thank you so much for the constructive feedback. We sincerely appreciate your valuable suggestions and questions. The following are our responses.

Q1: Comparison on the GIFT-Eval benchmark: I would encourage the authors to compare their model on the GIFT-Eval benchmark.

Thank you for this valuable suggestion. We acknowledge the limitations of the widely used eight benchmark datasets. We used them mainly for their widespread adoption in related works, so as to ensure fair comparison and reproducibility. While our method is not intended as a large-scale foundation model and was therefore not initially pre-trained on datasets like GIFT-Eval or Time-MMD, we fully agree that evaluating on standardized and diverse benchmarks such as GIFT-Eval is helpful for demonstrating generalization and robustness.

Due to time constraints, we have currently finished running the solar dataset from GIFT-Eval. We present the performance of DMMV-A and compare it with VisionTS on this dataset in the table below, which demonstrates the superiority of DMMV-A. We'd like to extend the results to more GIFT-Eval datasets and include them in a revised version of the paper.

Q2: SoTA compared methods: The model is not compared against SoTA forecasting methods such as N-HITs, iTransformer, etc., and foundation models such as TimesFM, MOMENT, MOIRAI, Chronos, LagLlama, etc.

Thank you for raising this concern. Our baseline selection aimed to cover diverse models by categories (Table 4 in appendix): (1) multimodal (Time-VLM), (2) vision-based (VisionTS), (3) language-based (GPT4TS, Time-LLM, CALF), (4) numerical methods (PatchTST, CycleNet, FEDformer, etc.).

Models like N-HiTS and iTransformer were not included because they belong to numerical methods similar to baselines reported in that category (e.g., PatchTST), which empirically perform no better than the cross-modal baselines[*]. Time series foundation models (TimesFM, MOMENT, Moirai, Chronos, Lag-Llama) rely on pre-training on large-scale time series datasets, which differs fundamentally from our experimental setting that didn't use time series pre-training data, which may lead to unfair comparisons.

Averaged Results for N-HiTS

[*] M. Jin et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models". In ICLR, 2024.

Q3: Efficiency: The method is not efficient, and there should be research on.

We agee with the reviewer that our method is not designed to be the most efficient but rather to balance performance and complexity. The additional numerical forecaster in DMMV-A is lightweight (0.2% of total weights) and adds only a small number of parameters compared to the LVM backbone. The table below compares the total parameters and training time per epoch on the ETTh1 dataset.

In practice, our method only incurs a negligible overhead compared to VisionTS, making the it suited for real-world applications while still delivering significant performance gains, especially when dealing with non-periodic datasets or cases where the predefined period P is inaccurate. Consider the benefits, we think the small cost overhead is paid off. We'd like to include this discussion in the revised draft.

Q4: Value of multimodality: How does the model do without explicitly modeling the linear components (only using the vision model), versus only using a transformer / linear component, versus model decomposed time series with a linear model (DLinear) or a transformer (say decomposition + PatchTST)?

Yes, we agree with the reviewer that comparing with the suggested ablations will help understand the advantage of multimodal views.

For "model only using the vision model", the VisionTS baseline in Table 1 serves for this purpose since DMMV-A's visual forecater alone is similar to VisionTS. For "model only using a transformer / linear component", the PatchTST and DLinear in Table 1 serve for this purpose since DMMV-A's numerical forecaster alone is similar to either of them. However, please let us clarify that DMMV-A's technical novelty lies in the multimodal view fusion framework, i.e., the backcast-residual mechanism in Figure 1(b), by leveraging the inductive bias of the visual forecaster.

From Table 1, DMMV-A outperforms the aforementioned baselines, suggesting multimodal fusion brings complementary advantages.

For "model decomposed time series with a linear model or a transformer", we replaced the visual forecaster in DMMV-A with a linear model, and report the results in the following. As can be seen, the original DMMV-A outperforms the variants, suggesting the effectiveness of visual forecasters.

Q5: Why is 336 chosen as the look-back window? In most recent studies, lookback windows are either 512 (e.g., MOMENT, PatchTST), or 96 (e.g., iTransformer).

We appreciate the opportunity to clarify this point. While 512 and 96 are commonly used look-back lengths, 336 is also a standard choice adopted by models such as PatchTST (PatchTST/42 version), DLinear, CycleNet, ETSformer. The key reason for adopting 336 is that we tried to follow the standard evaluation protocol of our baselines as specified in [1][2][3]. This facilitates fair comparisons and reproducibility.

To validate robustness, we conducted experiments with multiple window sizes (Illness: {104, 156, 208, 312, 416}). The averaged MSE results in the table below indicates DMMV-A outperforms the baseline in most cases. Also, in Figure 9 of the paper and Figure 15 of the appendix, our method consistently outperforms baselines under various input windows, confirming its robustness to look-back length.

Models Averaged performance under different look-back windows:

[1] M. Tan et al. "Are Language Models Actually Useful for Time Series Forecasting?". In NeurIPS, 2024.

[2] A. Zeng et al. "Are Transformers Effective for Time Series Forecasting?". In AAAI, 2023.

[3] S. Lin et al. "CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patterns". In NeurIPS, 2024.

Q6 in Limitations: The authors have a section on the limitations of their in the appendix. I would encourage the authors to reflect on the efficiency of your method.

This question is related to Q3. The response is summarized in the reponse to Q3. We'd like to follow the suggestion and include it the discussion of limitation in Appendix A.

Dear Reviewer KH6N,

Thank you for reviewing our response! We appreciate your recognition of our new experiments on the GIFT Eval framework. We will complete these experiments to ensure comprehensive comparisons. Meanwhile, please let us address your remaining concerns in the following.

1. Improving the benchmark (by evaluating the model on GIFT Eval) does not make the comparison unfair or non-reproducible.

We totally agree with this statement. In our previous response to Q1, we stated "We used them (i.e., the 8 datasets) mainly for their widespread adoption in related works, so as to ensure fair comparison and reproducibility" because we wanted to provide the reason for our earlier focus on the 8 benchmark datasets used in our experiments. Yes, adding GIFT Eval to the 8 datasets will enhance the experiments.

2. TimeLLM is evaluated on the same datasets with known issues. Therefore, I humbly disagree with this claim.

Thank you for pointing this out. Considering the importance of N-HiTS and iTransformer, we compared them with our model on three datasets given the time limitation. The results in the following table validates DMMV-A's effectiveness in leveraging multimodal views to improve performance over N-HiTS and iTransformer. We'd like to extend the experiments to all datasets, including GIFT Eval, and add them to our current 14 baselines in Table 4 in the appendix.

3. Besides, it is unclear to me what the goal of the paper is. Is it to propose a method that is practical, or just interesting?

We apologize if our purpose causes any confusion. Please let us clarify it. The goal of the paper includes (1) highlighting a finding about the SOTA LVM-based forecasters: bias towards forecasting periods caused by imaging technique; (2) proposing an effective solution to address the limitation by a backcast-residual based multimodal view framework, so as to inspire further developments; (3) providing a practically useful model.

We think the reviewer focuses on the third point, which we value a lot. To this end, we'd like to complete our experiments on GIFT Eval and add N-HiTS and iTransformer to our current 14 SOTA baselines (including models released in 2024 and 2025) in Table 4 in the appendix. As for time series foundation models (TSFMs), we humbly think comparing a model (e.g., DMMV-A) leveraging cross-modal knowledge (e.g., visual knowledge) with a TSFM leveraging intra-modal knowledge (i.e., obtained by pre-training on large-scale time series data) is unfair to some extent since the sources of knowledge are different. However, we think it is promising to combine vision models and TSFMs for fusing cross-modal knowledge and intra-modal knowledge due to their complementary modalities, which may further enhance time series modeling.

In addition to quantitative comparsions, we think analyses to support the first and second points are worth attention, which corresponds to our evaluations in Section 4.3.

As for efficiency, in addition our previous response to Q3, we'd like to provide the average inference time of our method DMMV-A, using VisionTS as a reference point, in the following. Since DMMV-A only adds a light-weight numerical forecaster to the visual forecaster, the inference overhead is small and the overall inference time looks practically reasonable.

As such, we hope the full perspectives of our goal, including the intellectual merits and the practical solution, could be fairly assessed. Meanwhile, we'd appreciate it if the reviewer could let us know what else might be missing so that we can take the opportunity to clarify it. Thank you!

The Averaged inference time for each sample in ETTh1:

Dear Reviewer srxJ,

Thank you so much for the constructive feedback. We sincerely appreciate your valuable suggestions and questions. The following are our responses.

W1: The model's forecasting performance primarily relies on VisionTS (the visual forecaster). For DMMV-A, the authors only introduce a commonly used moving average decomposition mechanism, which limits the overall novelty of the framework.

Please let us clarify a misunderstanding in this comment. DMMV-A's decomposition is based on a backcast-residual mechanism with our proposed BCmask strategy, which is different from moving average decomposition. Specifically, in Section 3.1 and 3.2, we proposed two variants of DMMV, Figure 1(a) and (b) present their structures: (a) DMMV-S (simple decomposition) adopts moving average decomposition; (b) DMMV-A (adaptive decomposition) was designed with a learnable backcast-residual based decomposition, which is adaptive. By comparing them, we find DMMV-A is better in Figure 5 (also in Table 4 in appendix (supplementary files)), thus is more recommended.

We acknowledge the important role of LVMs in our method, and reflected it in the title of the paper. However, we'd like to highlight our contribution to explore multimodal views of time series by discovering and levarging a bias of the LVM-based forecaster, which could lay the groundwork for future multimodal models on time series. Also, we think the difference in the overall numbers of wins between DMMV-A and VisionTS in Table 1, i.e., 43 and 9, is remarkable, suggesting a non-trivial improvement. In particular, DMMV-A addresses VisionTS's bias toward periodic data, thus is generalizable to broader applications. As such, we hope our contributions could be fairly assessed.

W2: Compared to DMMV-A, DMMV-S involves a more complex design, yet its performance is actually worse, raising concerns about the effectiveness of the added complexity.

We guess this question is caused by the same misunderstanding in Q1. We apologize for the confusion. From Figure 1, DMMV-A's is more crafted than DMMV-S by introducing a backcast-residual decomposition, which is more adaptive. From Figure 5 (and Table 4 in appendix), this design makes DMMV-A perform better than DMMV-S. In the following, we present the average performance comparison. The full results are in Table 4. Additionally, our ablation analysis in Table 2 validates the design choices of DMMV-A.

Average Performance of DMMV-A and DMMV-S across different datasets:

W3: DMMV-A appears to be well-suited for datasets with periodicity. However, it remains unclear how the model would perform on non-periodic datasets.

Quite the contrary, LVM-based forecaster alone, like VisionTS, is suited for datasets with periodicity, overlooking global trend to some extent. This is explained in our Section 3 ("An Inductive Bias"). In contrast, our method, DMMV-A, was designed to address this limitation and enhance LVM-based forecaster with multimodal views, thus is more suited for less periodic or non-periodic datasets such as Weather and Illness, as presented below.

Average Performance on Non-Periodic Datasets:

Q1: This makes the model highly sensitive to the choice of segment length P; if P is not properly set, the performance may deteriorate significantly.

In fact, segment length $P$ can be set as a period by Fast Fourier Transform, releasing the burden of manual tuning. However, if $P$ is perturbed, LVM-based forecaster alone is more sensitive than our DMMV-A model since DMMV-A reduces reliance on period accuracy by adaptive decomposition. We did the following test: when the period $P$ is perturbed (i.e., varying by P/6, 2P/6, ..., P), VisionTS's performance drops more (e.g., ETTh1: 0.409 → 0.582 (42.3%)) than our model (e.g., ETTh1: 0.395 → 0.450 (13.9%)), suggesting the better robustness of our method.

Q2: Based on the results in Figure 5, DMMV-A outperforms DMMV-S. Does this suggest that additional design components of DMMV-S may be ineffective or unnecessary?

This question is the same as W2. The response is summarized in the reponse to W2.

Q3: Table 1 does not report the results for DMMV-S. Could the authors include them for completeness and comparison?

Yes. Currently, we put the results of DMMV-S in Table 4 in the appendix for keeping Table 1 succinct while highlighting salient methods. Now we are aware of the interests in DMMV-S, and we'd like to update Table 1 to include it in a revised version of the paper. Thank you for the suggestion.

Q4: On datasets with strong periodicity, such as Electricity and Traffic, the performance gains of DMMV-A over VisionTS are limited. Could the authors explain this limited improvement?

This question relates to W3. In fact, VisionTS is suited for datasets with strong periodicity. On such datasets, VisionTS performs well, while modeling global trend may only bring small gains. This explains why DMMV-A's gain over VisionTS is limited. In contrast, on datasets with less periodicity, such as Weather and Illness, DMMV-A achieves more notable improvements over VisionTS, validating its effectiveness in addressing VisionTS's limitation. Considering both cases, periodic or non-periodic, DMMV-A is no less than VisionTS, indicating its broader applicability.

Q5: According to Figure 6, the visual component's weight in DMMV-S is smaller than in DMMV-A. Would manually adjust the gating value improve performance?

In DMMV-A, $g$ is automatically learned to avoid human bias, allowing adaptive aggregation of the decomposed components. Manual tuning $g$ may be inefficient thus is not recommended.

To check whether manual tuning $g$ works for DMMV-S, we set the visual weight as same as DMMV-A for DMMV-S. The following table compares the results. As can be seen, manual weights lead to performance drop. This is because the imposed manual $g$ cannot align with other parts of the model, which are aligned with the leared $g$ during model training. We'd like include this result in a revised version of the appendix.

Q6: Unlike VisionTS, DMMV-A cannot perform zero-shot forecasting and requires fine-tuning. Since fine-tuning latent variable models (LVMs) may incur additional computational costs, how can one balance these increased costs against the performance gains?

Although VisionTS can be used for zero-shot forecasting, we wanted to highlight its performance is limited. The following table presents its zero-shot performance, which is even inferior to baslines DLinear, Time-LLM, PatchTST, etc. (Table 1). Thus fine-tuning is still important for VisionTS to perform optimally, like DMMV-A.

On the other hand, DMMV-A allows freezing LVMs while fine-tuning other weights -- a light-weight numerical forecaster (0.2% of total weights) -- in the model, thus can be efficient. However, to reach optimal performance, fine-tuning is still recommended and it only incurs a negligible overhead compared to VisionTS, as indicated in the second table below. Consider the benefits of DMMV-A, we think the small cost overhead is paid off.

VisionTS zero-shot averaged performance

Model size and training time

Dear Reviewer SJEM,

Thank you so much for the constructive feedback. We sincerely appreciate your valuable suggestions and questions. The following are our responses.

W1: Is the performance improvement achieved by the proposed method too incremental? A significance test may be necessary to validate its effectiveness.

Thank you for highlighting this concern. Yes, the challenge of improving MSE and MAE over the SOTA methods on the widely used benchmark datasets may raise doubts about the significance of improvements. To address this concern, we ran our method three times with different seeds and calculated standard deviations. In our appendix, Table 9 reports the standard deviations of DMMV on the same four datasets used for ablation and performance analysis (section 4.2, 4.3). In the following, we summarize the comparison between DMMV-A and the most competitive baseline, VisionTS.

From the results, both methods show small standard deviations across multiple runs, suggesting their stability and the reliability of the results. We ran two-sample t-tests for each dataset under different forecasting horizons (i.e., 96, 192, 336, 720). DMMV-A exhibits significant improvements over VisionTS in 13 out of 16 cases using MSE (p-value<0.05) and 9 out of 16 cases using MAE (p-value<0.05), totaling a majority of the cases. Also, considering the non-trivial difference in the overall numbers of wins between DMMV-A and VisionTS in Table 1, i.e., 43 and 9, we think the improvements are not a matter of randomness. We will extend this analysis to all datasets and add the results to a revised version of the appendix.

W2: Are visual models sensitive to the visual representation of time series data, such as line color, pixel conditions, the proportion of the time series line in the image, line smoothness, etc.?

This is an interesting question. If time series is imaged by line plots, we saw a survey paper [*] discussed it (in its Section 3.1). Its conclusion is that there is no consensus on whether graphical components, such as lines, colors, legend, grids and tick labels, could provide extra benefits in any task.

In our method, since we didn't use line plots, our method can be think as not being influenced by graphical components. Section 3 (Preliminaries) describes our imaging method. Time series is segmented by period. The segments are stacked to form a matrix, which is resized to a 224x224x3 gray-scale image. The image is gray because the 3 channels are the same. This is intuitive because we don't have extra information to be encoded by color. The most informative pattern is the texture of the image, which is determined by time series values. As such, we think our method will not introduce sensitivity to graphical components to vision models.

[*] J. Ni et al. "Harnessing Vision Models for Time Series Analysis: A Survey". In IJCAI, 2025.

W3: Do multimodal models have potential applications in the spatiotemporal domain?

We are thankful to this great comment. We believe spatiotemporal data modeling is a notable direction to us for its wide applications in domains such as traffic, climate and neuroscience. One key challenge is modeling spatial dependency across channels together with their dynamics. In our humble opinion, our method can be extended to spatiotemporal data by introducing a spatial encoder, which could be a Transformer-like architecture that learns the attention among channels through intermediate embeddings. However, a bottleneck in computational cost may appear when there are hundreds or thousands of channels, which is worth further research. The key advantage of using multimodal view models, like ours, includes their capabilities in capturing complementary patterns across views and leveraging cross-modal knowledge transfer from LVMs, especially when task-specific data are scarce. We'd like to add this discussion to the revised appendix.