Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop

Question 1 and Weakness 1,2: Innovation and scope:

We appreciate the reviewer’s comments and the opportunity to clarify our contributions. Our work is motivated by the need for effective explanation, multi-modal time series understanding, and contextual reasoning—areas that are underexplored in current literature. Compared with existing LLM-based methods, our approach offers a unique advantage through: (1) explicit, case-based explanations grounded in multi-modal time series inputs, and (2) closed-loop interactions between time series models and LLMs that enable iterative understanding and refinement of real-world contexts. We will revise the related work section to more clearly articulate these distinctions.

Question 2: Meaningful text segment:

Thanks for the question! As discussed in Section 3.2.1 (Page 3), capturing meaningful text segments (i.e, text prototypes) relies on two components, the pretrained language model producing text embeddings for input texts, and the sequence encoder mapping text embeddings and performing prototype learning. 1) The first component decides the granularity of prototypes at the input level. For example, Bert represents each text input as multiple tokens (token-level embeddings), while S-bert represents it as multiple sentences (sentence-level embeddings). 2) The second component further encodes these text embeddings. We use the convolution-based encoder to get segment representations by applying sliding windows over the text embeddings. With multiple consecutive segment representations, we perform prototype learning with regularizations and projections to identify the most typical segments.

Question 3 & 4 and Weakness 3:

Event time prediction and temporal point process: Thanks for the question! We agree with the reviewer that this is an interesting extension. However, we would like to clarify that our setting is fundamentally different from the temporal point process (TPP) setting and its event time prediction task. TPP models are specifically designed for irregular event sequences, but our problem setting and datasets are for regularly sampled multivariate time series with categorical or continuous outcomes at each step, following the standard time series prediction paradigm. Our objective is not to predict when the next event will occur, but rather to predict what will happen at each regularly observed time step. Therefore, event-time prediction is not applicable in our setting, and TPP models are not directly relevant for comparison.

Adapting our method to this task would require major changes: the encoder must handle irregular timesteps, and the output layer must predict inter-event intervals or model intensity functions. While our prototype design and LLM agents can be modified, it is outside the scope of the current work. Nevertheless, we appreciate the reviewer’s insight and agree that TPP is a valuable framework for modeling temporal dynamics, and will discuss representative works and their differences in the updated version.

Beyond discrete label prediction: We provided regression results in Appendix F (pages 23-25). We also provide baselines comparisons and model analysis to demonstrate its efficacy for regression tasks. Please refer to the rebuttal for reviewer g9bw for detailed results and discussions.

Question 5 and Weakness 3: Base LLMs

Thank you for this question! To explore the effect of different base LLMs (Gemini-2.0 Flash and GPT-4o-mini), we provide the experiment results and analysis on Healthcare (Test-Positive) data. For fair comparisons, we used the same prompts and ran the same number of iterations and followed the same settings detailed in Appendix A.2 and A.4 (pages 13-14). We still use the same temperature setting (0.7 for content generation, 0.3 for prediction), as it yields the best performance empirically. The results are shown below.

GPT-4o clearly outperforms both GPT-4o-mini and Gemini-2.0-Flash, due to its better reasoning and contextual understanding capabilities (larger model size & pre-training corpus). Both GPT-4o-mini and Gemini-2.0-Flash are still competitive compared with baselines listed in Table 1, Page 7. It reveals the impact of the LLM capability on the effectiveness of our framework, especially in tasks demanding real-world context understanding.

We also provide the plot of iterative analysis in this link, where we can also observe the performance improvement over iterations for different base LLMs.

Question 7: Detailed prompt designs

As indicated in Section 3.3.2 (Page 5), we provided the detailed prompt templates in Figures 13-17 in Appendix D (Pages 18-21). We also provided the code containing specific prompts in the supplementary materials. Please refer to them.

The regression-based setting:

We sincerely appreciate the reviewer for the comments. In Appendix F (Pages 23-25), we implemented a regression-based variant and provided a demonstration on the same finance dataset with numerical ground truths to show its capability for numerical value forecasting. To further address the reviewer's concern, we provide more details and additional experiments below.

As introduced in Appendix F, we adapt TimeXL for regression with two minor modifications. First, we add a regression branch in the encoder design, as shown in Figure 23. On the output of time series prototype layers, we reversely ensemble each time series segment representation as a weighted sum of time series prototypes, and add a regression head for prediction. Accordingly, we add another regression loss term to the learning objectives. Second, we adjust the prompt for prediction LLM by adding time series inputs and requesting numerical forecasts, as shown in Figure 24. As such, TimeXL is equipped with regression capability. Next, we evaluate our method on the same finance dataset with the same task as classification, despite that the main prediction target is the raw material stock price instead of trends. Here we follow the same settings detailed in section 4.1, and Appendix A.2-A.4. We update the results by comparing state-of-the-art baselines in the table below..

The main observations are consistent with the classification setting: The multi-modal variants of state-of-the-art baselines (MM-iTransformer and MM-PatchTST) benefit from incorporating real-world contexts; Our method achieves the best results, highlighting the advantage of synergizing multi-modal time series encoder with language agents to enhance interpretability and thus predictive performance.

Moreover, we provide the component analysis based on the refined texts that achieve the best validation performance across iterations. It includes evaluation results of the multi-modal time series encoder, input ablations for LLM-based predictions, and the fusion of both components (TimeXL). The results are summarized in the table below, which also show patterns consistent with those observed in the classification setting. (1) The results clearly demonstrate that real-world financial texts provide complementary information to the LLM, leading to improved accuracy in numerical prediction. (2) The identified prototypes provide contextual guidance to prediction LLM, leading to clear performance gains. (3) By fusing the predictions from both encoder and prediction LLM, TimeXL further improves the prediction and outperforms all variants, underscoring the effectiveness of mutual enhancement between the two components.

In general, (1) highlights the importance of multi-modal inputs, while (2) and (3) highlight our proposed interaction between encoder and LLM for more accurate numerical predictions.

We also provide an iteration analysis to show the effectiveness of reflection and refinement LLMs, as shown in the table below. The prediction performance quickly improves and stabilizes over iterations, which underscores the alternation steps between predictions and reflective refinements.

We sincerely thank the reviewer for the insightful feedback and provide our response below.

Q1 and Weakness 2: Regression tasks

We appreciate the reviewer’s interest in understanding how our approach performs on regression tasks. Below are the results of our method and state-of-the-art baselines on the same finance dataset in our paper, but with numerical ground truths.

Please kindly refer to our response to reviewer g9bw for more experimental results (or this link) and discussions. We would also like to emphasize that we focus on classification-based prediction as many real-world multi-modal applications naturally involve discrete decision-making. This formulation also better highlights the interpretability benefits of our case-based explanations and the LLM’s reasoning capabilities. As for the baselines, TimesNet, OFA, TimeCAP inherently support classification, and the commonly-used TSLib (Appendix A.2) also uses most methods for classification. We also acknowledged regression tasks and provided results in Appendix F (Pages 23-25), including initial results and method designs.

Q2: Encoder choice

As discussed in Section 3.2.1 (Page 3), the model choice affects the explanation granularity, and we used convolutional neural networks followed by prototype layers for both modalities to capture segment-level prototypes for prediction and explanation.

Q4: Iterative process

The iterative process relies on training supervision to improve the text quality of training, validation, and testing sets. As detailed in Section 3.3.2 (Pages 5-6), TimeXL iteratively generates reflective feedback (by reasoning on text, prediction, and ground truths) to refine training texts, where validation data evaluates feedback quality per iteration. Feedback with the best performance is then applied to refine testing texts, which mimics applying a trained model to testing data (Page 6, Lines 284-288).

Q3 and Weakness 1 & 3:

Refined input from LLMs: In table 2, the ablation study on testing data is based on the refined texts, where the refinement is guided by reflective feedback from the best iteration selected by validation data (Page 6, lines 284-288).

Text and prototype only used for ablation study of LLM, not encoder: We clarify that the prototypes are outputs of the encoder (Section 3.2.2, Prototype Projection, Page 4). As predictions and explanations cannot be derived without these prototypes, the ablation is not applicable. However, text and prototypes can both be input to the prediction LLM, and we perform such input ablation to show the performance gains from the contextual guidance provided by prototypes (Section 4.6, Page 8).

Importance of LLM components: We show the importance of reflection and refinement LLMs via iterative analysis in Figure 5, Section 4.5. The original performance means no usage of reflection and refinement LLMs. It is clear that the text quality improves over the iterative reflection and refinements (upper two subplots). TimeXL’s prediction also improves (lower two subplots), because of the improved texts. The importance of prediction LLM is shown in Table 2. The multi-modal encoder indicates no usage of prediction LLM. After fusing with prediction LLM (text+prototype), its performance further improves (TimeXL). Appendix B includes more ablations .

Reference Discussion

Due to space limits, we summarized the insights of these works in the related work due to their differences from ours. News to Forecast constructs a large news database for method development and performs text-to-text prediction by fine-tuning LLM with selected news. Our method uniquely provides explicit explanations (standard case-based framework) from both modalities, showing key time series and text segments that contribute to the prediction. Besides, we emphasize the mutually augmented prediction between the multi-modal time series model and LLM for better performance. The most recently arxived ChatTime and ChatTS differ in scope: ChatTime examines modal translation capability, targeting zero-shot forecasting on common benchmarks, and reasoning tasks. ChatTS targets time series understanding and reasoning tasks instead of prediction. However, we thank the reviewer for pointing them out, and will provide detailed discussions in the updated version.