Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data

Thank you very much for your feedback on our work. We hope that the general rebuttal above and our comments below answer any questions you may have.

W1:
While the TS Instruct dataset and benchmark are comprehensive, they primarily consist of synthetic data. Real-world time-series data can be more complex and noisy, which might affect the model's performance in practical applications.

Response:
The time-series data in our benchmark comes from real-world datasets in the LOTSA archive, ensuring grounding in realistic scenarios. While synthetic text data is used for instruction generation, the model evaluations are firmly rooted in practical time-series applications.

We would like to thank you for your very constructive feedback on our submission. We have addressed each of your commnets and concerns below and eagerly await your feedback.

W1:
There lacks of dataset quality check. Since the instruction tuning dataset is generated by GPT-4o-mini with images, it may introduce errors. Although the authors have filtered out bad cases (as introduced in the appendix), it is not sure whether hallucinations do not exist. Maybe I missed some details, please feel free to have a discussion.

Response:
We conducted an additional manual evaluation on 100 random samples from the dataset. This subset was graded for quality, correctness, and relevance. Out of 100 samples, 92 were correct, with 8 cases being ambiguous or open to interpretation. This indicates that the synthetically generated dataset is of high quality. To address concerns, we have also created and will release a refined dataset called TS Instruct QA Gold, which underwent manual improvements to enhance diversity and accuracy.

Please refer to the general feedback above for a complete analysis of this data.

---

W2:
Limited modality interleaving: since the trained model could only generate text (as the dataset is constructed to do so), it limits the application to time-series data understanding and ignores the time-series forecasting. Maybe a more fine-grained instruction dataset should be constructed with multi-modal responses.

Response:
We acknowledge the limitation and clarify that the scope of this work is focused on time-series reasoning since it is severely understudied. However, we do think it is a very valuable future direction to create interleaved multi-modal datasets for time-series forecasting and imputation tasks.

---

Q2 and W3:
Lack of baselines: Maybe visual language models could be considered as baselines to take images and instructions as input and output their options. This could be a reference to tell readers why numerical data pretraining is useful for relevant time-series data understanding.

Response:
We thank you for the constructive suggestion. We have added new baselines, including vision-based models such as Llama-3.2-11B Vision-Instruct and Phi-3.5 Vision-Instruct, as well as strong text-based baselines like GPT-4o and GPT-4o-mini. These additions provide a robust and diverse comparison that highlights the strengths of our approach.

That said, we want to acknowledge the inherent caveats of vision-based baselines in this context. Vision models have inflated performance on our benchmark due to their alignment with the dataset generation process, as the visual components were involved in creating the synthetic data. This alignment may give these models an unintended advantage, and their performance should be interpreted with caution. Moreover, the vision based models tested (trained by Microsoft and Meta) have extensive multi-modal training data which would give them advantages over our models and other models for which there is very little multi-modal time-series training data.

Despite these caveats, we have included these vision-based models to ensure a comprehensive evaluations. Additionally, we included multiple baselines of varying sizes and architectures. These efforts represent a significant enhancement to the paper's experimental rigor and provide the best possible comparison within resource constraints.

Please kindly refer to the general feedback for a full disection fo these new results and addition information on the visual instruction tuning baselines.

Q1:
For Table 3: Why TS training could improve the BBH and GPQA results while harming the MMLU-Pro results? Is it just performance fluctuation or is there an inner difference between these benchmarks?

Response:
The observed variations arise from differences in task alignment objectives. We would expect the model alignment to shift since we are doing full parameter finetuning. These fluctuations, less than 2%, do not indicate a degradation or improvement in language processing performance and are within expected a reasonable range for model finetuning. Note that the Orca model is finetuned with text-only data and its performance differs from that of the baseline LLama model.

---

Q3:
Open question: What affects the time-series data pretraining? e.g., context length, the vision ability, the reasoning ability, etc.

Response:
Time-series data pretraining described in this paper is used as a way to initialize the new time-series token embeddings. The alternative is to take the mean of the existing text embeddings to initialize them. We then pretrain the model on purely time-series data as a way to further refine the added time-series embeddings. Since the loss is only computed on the new time-series embeddings, we have found that this has minimal impact on the natural language performance of these models.

We thank you for this feedback and hope we have adressed your questions and conerns. Please refer to the general rebuttal above for more details on the extensive experiments and data validation we have performed.

W1:
Generalization: It is unclear whether the synthetic dataset can generalize effectively across diverse types of time-series data or to sequences longer than those encountered during training. Experiments regarding this concern are highly recommended.

Response:
The dataset incorporates real-world time-series data from the LOTSA archive, ensuring its applicability to practical scenarios. We also tested the models on sequences of varying lengths, demonstrating reasonable generalization. Future work will include longer context evaluations and larger models which are unfeasible in our current setup.

---

W2:
Dataset Alignment Process: The alignment between time-series data and natural language relies solely on GPT-4, without additional quality control to validate this process. This raises concerns about potential inconsistencies or inaccuracies.

Response:
Within the time-series dataset generation process we also include images of the time-series and meta-data to improve the accuracy and relevance of the generated samples without solely relying on GPT-4. However we do understand the concerns about additional quality control to validate this process particularly for the testing data. To address this, we conducted additional manual quality checks which showed that of 100 random samples 92% were of reasonable quality and created the TS Instruct QA Gold dataset. Please refer to our general feedback above for a complete analysis.

---

W3:
Reporting results from common multimodal baseline models would provide readers with a clearer understanding of the current state-of-the-art performance for these tasks. A direct comparison with GPT-4o is also missing.

Response:
Thank you for your constructive comment. We have added several multi-modal baselines, including Llama-3.2-11B Vision-Instruct and Phi-3.5 Vision-Instruct. A direct comparison with GPT-4o and GPT-4o-mini has also been included, providing a comprehensive performance context.

However, it is important to note that vision-based models may have an advantage in our benchmark due to their alignment with the dataset generation process, as visual data was utilized to generate the the text synthetically. Please refer to our general feedback where we discuss this in-depth.

---

Q1:
Why does the base model show strong performance on the TS Instruct QA benchmark but perform less well on TS Math Analysis and Decision-Making tasks?

Response:
Math and Decision Making as reasoning tasks for time-series is very new. It is expected that the base-modal has little exposure in this regard and would struggle to perform these tasks compared to models which have been trained specifically with augmented time-series reasoning. While the test samples are held-out and were not in our training datset, there would be similar style examples included there.

---

Q2:
Do you expect that the improvements from your method would still hold with larger models? A scaling experiment with a larger model would be beneficial.

Response:
We expect that the observed trends will generalize to larger models. Resource constraints limited extensive scaling experiments, but scaling experiments remain a priority for future work. Our added base-lines do show that scaling model size and reasoning capabilities generally results in more powerful models for both time-series and nlp analysis tasks.

---

Comment:
The manuscript has some formatting issues, particularly with the margins not conforming to ICLR's template specifications. It could benefit from a more in-depth analysis of the model's limitations, potential biases, and practical applications, as well as more rigorous experimental validation.

Response:
We respectfully argue that Reviewer xWsL’s feedback lacks specificity and provides generalized critiques that could apply to nearly any submission, without engaging meaningfully with the content of our work. Below, we provide a point-by-point response to highlight the unfounded nature of these comments:

1. Formatting Issues:
   Our submission adheres to the ICLR Latex template specifications, we did not modify any style files or packages. If there are specific formatting issues beyond margins that the reviewer feels compromise the readability of the paper, we are happy to address those.

2. Model Limitations and Biases:
   Reviewer xWsL suggests that the manuscript would benefit from a more in-depth analysis of limitations and biases. However, the paper already includes a dedicated Limitations and Future Work section that addresses:

   • The scope of time-series reasoning versus forecasting.
   • The trade-offs of time-series normalization.
   • The challenges of scaling multi-modal instruction tuning for zero-shot classificaiton.

   Without specific concerns or omissions identified by Reviewer xWsL, it is unclear what additional analysis is expected. This comment fails to acknowledge the substantial discussion already present in the manuscript.

3. Practical Applications:
   The critique that the paper does not address practical applications is particularly surprising. Our work explicitly connects time-series reasoning to real-world use cases by:

   • Training models on data derived from real-world time-series datasets (from the LOTSA archive).
   • Evaluating model performance on practical reasoning tasks that are critical for domains such as healthcare, finance, and engineering.
     Reviewer xWsL provides no counterexamples or evidence to substantiate their claim, making this feedback unsubstantiated.

4. Rigorous Experimental Validation:
   The comment regarding experimental rigor fails to account for the extensive baseline comparisons already included in the paper. These include:

   • Comparisons across differnt model permutations and baselines. We have expanded it to include 8 open-source models and 2 closed-source models (GPT-4o and GPT-4o-mini).
   • Detailed quantitative evaluations on multiple benchmarks, including TS Instruct QA Gold, MMLU-Pro, BBH, GPQA, and TS Math Analysis and Decision-Making tasks.
   • Discrete tokenization validation experiments.

   Reviewer xWsL does not identify specific shortcomings or suggest additional experiments that would improve the work. Without clear guidance, the assertion of insufficient validation appears baseless.

We believe Reviewer xWsL’s comments do not substantively engage with the contributions or content of the paper. Their feedback consists of generic critiques that could be applied indiscriminately to most submissions.

3. QA Gold Task and Vision Model Performance

We included several new baselines in our evaluation to provide a more comprehensive comparison and better context for the performance of our models. Notably, we added two vision-based models, Llama-3.2-11B Vision-Instruct and Phi-3.5 Vision-Instruct, as well as strong text-based baselines such as GPT-4o and GPT-4o-mini. These baselines help highlight the trade-offs between vision-based and text-based approaches for time-series reasoning tasks.

Table: Model Performance Metrics

In the table below we show the results of the added base-lines on the new TS Instruct QA Gold dataset. This evaluation shows not only the effects of model size but also of different architectures. It also highlights the importance of work such as ours that focus on building models for time-series reasoning analysis. Despite the size of our model it outperforms model's such as Phi-3-medium-4k-instruct which are both larger and show stronger reasoning capailities on NLP benchmarks. These results also show the importance of scale since scaling up model performance (for example gemma-2b to gemma-9b) generally increases performance on both the MMLU_PRO and time-series benchmarks. We also evaluated powerful "closed-source" models such as GPT-4o and GPT-4o-mini. These results should be interpreted with caution as the GPT-4o-mini (which likely shares training data with GPT-4o) was used in the origional dataset generation process.

Open Source Models

In additional to providing training and testing data, a key contribution of our work was providing a flexible model training process through discrete tokenization which has minimal impact on nlp performance and easily integrates time-series reasoning. The table below shows the importance of training models for time-series analysis. The PreOrcaTS(ours) model performs proportionally better than the other baselines when it comes to the time-series reasoning compared to MMLU_PRO performance. The table is sorted by TS Reasoning performance.

In our updated paper version this will be converted to scatter plot to more clearly visualize these results.

@inproceedings{merrill-etal-2024-language,
    title = "Language Models Still Struggle to Zero-shot Reason about Time Series",
    author = "Merrill, Mike A  and
      Tan, Mingtian  and
      Gupta, Vinayak  and
      Hartvigsen, Thomas  and
      Althoff, Tim",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.201",
    pages = "3512--3533",
}