MoAT: Multi-Modal Augmented Time Series Forecasting

C1. Text already contains a wealth of information, so using patching is unreasonable here.

As the reviewer commented, text provides sufficient information at each timestep while time-series data does not. However, we would like to emphasize the necessity of patching texts in our architecture. Specifically, we aggregate text embeddings within corresponding time intervals as the time series patches. This alignment is particularly crucial for feature-wise augmentation, where we concatenate temporally aligned time series and text patches as input for the shared multi-modal augmented encoder of MoAT.

C2. Decomposition of text data is unclear.

We acknowledge the reviewer's comment and modified the attention pooling function for texts by using time series patches as query vectors during the attention computation for text aggregation. From the modification, texts are aggregated differently based on the seasonal and trend time sereis patches. For more details, please refer to the common response above.

C3. Table 3 shows that the largest dataset, Bitcoin, contains only 3K scalars. This raises concerns about the suitability of training complex neural networks with such limited data.

As noted by the reviewer, the datasets we collected comprise a limited number of time series data points. To mitigate this scarcity in time series data, we employ multi-modal information, particularly leveraging the abundance of information within the text data. Specifically, text data contains a substantial amount of information that supplements the limited time series data. Moreover, the superiority of transformer-based models (e.g., PatchTST and Crossformer) over simpler linear models (e.g., DLinear and LightTS) suggests that complex neural networks like transformers indeed outperform simpler models within our datasets.

C4. Both input and output series is very short. While most selected baselines are for longer term forecasting. So the comparison is not so fair. It is recommended to add traditional and straightforward models as baselines.

While we incorporated basic linear models like LightTS and DLinear as baselines, we appreciate the reviewer’s feedback regarding the need for a fair comparison. We will take this into account and include baseline models to ensure a more comprehensive evaluation.

C1. It requires further clarification why the proposed textual decomposition components map to the trend and seasonality aspects of the time-series data.

We acknowledge the reviewer's comment and modified the attention pooling function for texts by using time series patches as query vectors during the attention computation for text aggregation. From the modification, texts are aggregated differently based on the seasonal and trend time sereis patches. For more details, please refer to the common response above.

C2. The paper focuses on short-term forecasting.

As commented by the reviewer, we intentionally chose a shorter forecasting length compared to some other time series forecasting works. This decision stems from the fact that the considered datasets are relatively smaller than benchmark datasets, and opting for a longer lookback or forecasting window inevitably leads to a reduction in the number of available training samples, potentially resulting in a significant performance decline. In addition, we would like to emphasize that forecasting time series even with a shorter lookback window presents a new challenge compared to existing works, which is often more practical and thus important.

C3. The datasets primarily feature monthly or weekly sampling intervals. This raises concerns about the applicability of the approach to datasets with higher frequency sampling.

As raised by the reviewer, testing on datasets with higher frequency is indeed an important consideration. However, we would like to emphasize that we collected six distinct datasets encompassing various sampling intervals, ranging from daily to monthly frequencies. In addition, elevating the sampling frequency can potentially introduce timesteps with missing text data, posing an additional challenge that requires careful attention. In our study, we assumed that at least one text is available at each timestep and leaved addressing this challenge as our future work.

C4. In Table 2, MoAT_time, even without textual data, appears to outperform many baseline models.

Thank you for highlighting this valuable insight. As noted by the reviewer, MoAT_time demonstrates superior performance compared to most baseline methods. We conjecture that his advantage can be attributed to the combined utilization of decomposition and patching within our framework. While there are existing time series forecasting methods that solely employ decomposition (e.g., DLinear and Autoformer) or patching (e.g., PatchTST and Crossformer), to our knowledge, none of these methods have incorporated both components simultaneously. In contrast, MoAT_time integrates and leverages both decomposition and patching to maximize their effectiveness.

C5. The results in Table 2 suggest that dual augmentation does not significantly enhance performance.

We would like to clarify that each variant of MoAT in Table 2 is fine-tuned within the same hyperparameter search space. The comparison between MoAT_sample and MoAT_feature aims to examine which augmentation method contributes more significantly. MoAT represents a comprehensive framework that encompasses the overall approach.

C9. Is the forecasting in Figure 5 autoregressive?

The experiments are conducted in an autoregressive manner, as correctly mentioned by the reviewer.

C10. The motivation for offline prediction synthesis discusses improved parameter efficiency, but parameter efficiency is not discussed in the rest of the paper. Why is modularity desirable in this setting?

Our intention was not to claim that modularity is desirable in our setting but rather to mention its importance in the referred paper. We apologize for any confusion caused and will make sure to clarify this point accordingly.

C11. Any intuition as to why MoAT_time is the best performer on the Fuel dataset?

It is important to note that we fine-tuned each variant of MoAT, including MoAT_time, within the same hyperparameter search space. In the case of the Fuel dataset, text behaves somewhat like noise and does not contribute to performance improvement. MoAT is a more universally applicable configuration across all datasets, providing a more generalized approach.

C12. The ablations in Table 2 suggest that the augmentation are not really helpful, considering the relative improvement compared to MoAT_time and MoAT_text.

The comparison between MoAT, MoAT_time, and MoAT_text aims to highlight the effectiveness of leveraging multiple modalities for time series forecasting. Conversely, MoAT_sample and MoAT_feature are intended to demonstrate how employing both augmentation methods using these multi-modalities enhances performance. Therefore, these different variants are designed to confirm distinct aspects rather than competing with each other.

C13. Ridge regression includes a loss penalty. What weight did you choose for this?

The weight for the regularization term in ridge regression is selected based on its performance evaluated on the validation set.

C14. Why are the prediction lengths so short?

We intentionally opted for shorter forecasting lengths due to the relatively smaller size of the datasets under consideration. A longer forecasting length would unavoidably reduce the number of available training samples, potentially causing performance degradation.

C1. This paper is mostly applied and combines several existing techniques, e.g., patch-wise embedding and pattern decomposition. The authors are expected to better position this work by clarifying the technical novelty, contribution, or new insights.

We would like to emphasize the technical contributions and novel insights our work brings to both the multi-modal and time-series domains as follows:

• We have reformulated existing augmentation techniques within the multi-modal domain, categorizing them into sample-wise (modality-specific) and feature-wise (modality-fused) augmentations. This configuration of multi-modal augmentation provides novel and structured viewpoints on how multiple modalities can be effectively fused.
• Surprisingly, despite its potential, there was a lack of benchmark datasets incorporating both time series and text (or other modalities). Thus, we gathered six real-world datasets, and we plan to make them publicly accessible. We strongly believe these datasets will serve as valuable resources for future researchers in this field.
• We proposed MoAT, a time-series forecasting method that leverages multi-modal information. Specifically, MoAT consists of a multi-modal augmented encoder, which benefits from two distinct augmentations applied to both time series and text inputs. Note that most existing multi-modal approaches have typically utilized only one of these augmentation methods.
• As the reviewer commented, patching and decomposition are existing techniques in time series forecasting. However, we would like to emphasize that, to the best of our knowledge, no prior work has combined both patching and decomposition for time series forecasting. Our approach, integrating these techniques, has notably improved accuracy.

In summary, we would like to emphasize that our work makes substantial contributions to (1) redefining methodologies, (2) introducing new datasets, and (3) presenting technically novel methods. As the reviewer suggested, we will emphasize these aspects in the revised version.

C2. The evaluation is mostly on financial datasets. But for finance, the error metric MSE is not the main interest, since in the real-world, the prediction is to serve downstream tasks, e.g., portfolio construction, risk management, etc, and practically MSE does not directly translate to the improvement for downstream tasks.

The reviewer’s perspective is interesting, and we recognize the relevance of various downstream tasks tailored specifically for financial datasets. Nevertheless, we would like to emphasize that MoAT is a versatile time series forecasting framework that leverages multi-modal information. Thus, our evaluation primarily focused on measuring performance through MSE, which is a conventional metric used to assess the accuracy of the model.

C3. On page 5, the part “multi-modal augmented encoder” is essentially a combination of time series and text modalities, and what does the “augmented” refer to?

In Section 2, we restructured two orthogonal approaches for augmenting multi-modal data. Figure 3 (a) illustrates the multi-modal encoder in MoAT, which utilizes both augmentation methods. This integration allows MoAT to leverage both modality-specific and modality-fused information extractable from time series and text data.

C4. For text data, especially the embedding of news text, applying trend-seasonal decomposition is not intuitively understandable.

We acknowledge the reviewer's comment and modified the attention pooling function for texts by using time series patches as query vectors during the attention computation for text aggregation. For more details, please refer to the common response above.

C5. In Table 1, MSE are mostly low-magnitude values, while the real price of the finance assets of the data used in the experiment differs greatly in magnitudes. It is better to report the errors in the original scale of the data, because the prediction on the standardized domain would be less useful for downstream tasks in many cases.

In line with existing practices in time-series forecasting research, we standardized the entire time series dataset using the training set and conducted training and evaluation accordingly. Although the standardized values may vary in magnitudes, it is important to note that the relative rankings among different methods remain consistent. In addition, it is straightforward to revert to the original scales through de-standardization using the mean and standard deviation obtained for standardization.