Modeling Asynchronous Time Series with Large Language Models

W:Although the model in this paper has shown a significant performance improvement in the selected dataset, there is a concern that the performance of the model on all tasks seems to be low … the current task definition is not perfect enough, whether it is too difficult for the model, or usable enough.

Thank you for your feedback regarding the performance levels of our model. Asynchronous time series (AST) are inherently challenging due to irregular timing and sparsity of events. Most existing studies focus on datasets with a small number of event types, each presenting different levels of difficulty. In our Table 2, we reference datasets like Amazon, Retweet, Taxi, Taobao, and StackOverflow, which have been extensively studied in AST forecasting and are included in benchmarks such as EasyTPP [3]. These datasets are known for their complexity; for instance, prior to our work, the state-of-art Macro-F1 score for the StackOverflow dataset was as low as 0.0661, underscoring the difficulty of achieving high performance on this task

Studying these challenging datasets is important as they represent real-world scenarios where event occurrences are irregular and sparse. Our contributions aim to advance the field by:

• Enhancing Traditional Datasets: We have expanded upon traditional AST datasets by introducing datasets with larger numbers of event types, increasing task complexity and relevance.
• Establishing Baselines for Future Research: By providing baseline results on these more complex datasets, we enable future work to build upon our findings.

We hope this addresses your concern.  We would be happy to provide further clarifications.

W: One of the main experiments in the paper lacks more credible baselines… Finding more baselines will help reflect the performance of the method.

Thank you for your feedback on the need for more credible baselines. We have addressed this in Global Question 1 with the addition of more LLM-TS baselines. Please let us know if we can provide further clarification.

Q: Scaling laws have been widely demonstrated in LLMs, … whether this approach would generalize to larger models

While this was listed as a weakness, it sounded to us more like a question. If that is the case, we appreciate it if you can clarify. Our method is not specific to small models; we used Llama-3-8B-Instruct primarily due to computational resource constraints. We fully expect that our approach would generalize well to larger models if more computational resources were available. In fact, Reviewer 8vh2 also noted that "this technique will continue to improve as LLMs improve, and as in-context learning techniques improve." To explore this further, we conducted additional experiments with smaller models, specifically Llama3.2-1B and Llama3.2-3B. We observed a consistent trend of increasing performance as the model size increases, supporting the idea that our approach scales positively with model capacity. We have included these new results and analyses in Appendix A.8 of our manuscript. These findings suggest that our method would likely benefit from even larger models, reinforcing its generalizability and potential for improved performance with access to more powerful language models.

W: Related work lacks LLM for TS

Thank you for highlighting the gap in discussion on Large Language Models (LLMs) for time series in our related work section. We have revised the subsection titled "LLMs for Time Series," in the related work section to include a more comprehensive exploration of LLMs applied to time series analysis.

W: Performance is not competitive.

Thank you for your feedback regarding the performance comparison between StoP prompt learning and QLoRA. However, we respectfully disagree - as detailed in our results, StoP leads to substantial improvements across various tasks and datasets:

• StoP outperforms QLoRA in Forecasting by +9.97% Macro-F1 (MF1), in Imputation by +29.15% MF1, and in Anomaly Detection by +1.55% MF1, when averaged over our three text based datasets - Breakfast, MultiThumos, Epic Kitchen.
• Similarly, when averaged over all datasets and all tasks, StoP outperforms QLoRA by  +13.55% MF1.

We will include a detailed breakdown of these results in Appendix A.7 before the end of rebuttal period. These results prove StoP's competitive performance across diverse settings. We hope this addresses your concern and provides clarity on the advantages of StoP over QLoRA.

W: Lacks comparison with related work.

Thank you for your feedback on the need for more credible baselines. We have addressed this in Global Question 1 with the addition of more LLM-TS baselines. Please let us know if we can provide further clarification.

W: Figure 1 lacks many details.

Thank you for your feedback regarding Figure 1. We have updated the figure to clarify that it focuses on the tasks explored in our paper, while details about the framework are presented in Figure 2. We are happy to make additional adjustments based on your suggestions.

W: Written is weak.

Thank you for your feedback regarding the writing quality. We were surprised to read this after reviewer 8vh2 praised the paper in this regard “At a meta level, this paper’s strongest feature is how well it was written. I wish that more AI papers were written with this much clarity and intention.” Based on your comment, we have revised the abstract to make it clearer and have clarified the language in several portions of the paper. We would be happy to make further improvements if you could point out specific areas that remain unclear. Thank you for your feedback.

Q: Concrete example of Asynchronous Time Series

Thank you for your thoughtful suggestion. We clarified the language in Figure 1 to make it more informative. Additionally, we added a new paragraph (Paragraph 2) in the Introduction that highlights the differences between asynchronous time series (ATS) and traditional time series, providing insight into ATS using a concrete social media example. We hope these changes address your concerns and make the paper more self-contained and reader-friendly.

Q: There is a line of research (see e.g. AntGPT...), which does text-based next action prediction using in-context learning. Perhaps the authors could add more citations to other papers that use a similar ICL strategy to process sequences of actions ...

Thank you for your insightful feedback and for bringing AntGPT (https://arxiv.org/abs/2307.16368) to our attention. While AntGPT and similar works leverage in-context learning with LLMs for next-action prediction in video-based action recognition, forecasting, or videographic memory tasks (e.g., question answering and retrieval on underlying video data, as in https://arxiv.org/pdf/2312.05269), our work focuses exclusively on textual asynchronous time series and extends beyond forecasting to include anomaly detection and imputation. We have revised our novelty statement to:

"To the best of our knowledge, this is the first work to explore the capabilities of LLMs to process textual asynchronous time series data across multiple tasks such as forecasting, anomaly detection, and data imputation." We have also added citations to these relevant works in the introduction to acknowledge their contributions.

Q: Motivation for why it is reasonable to only take prefix-slices of the prompt

Thank you for your feedback on the motivation for using prefix-slices in StoP. Our approach is inspired by several established techniques in the literature. The idea of introducing randomness during training aligns with methods like dropout and stochastic depth, which enhance robustness by exposing models to varying input or architecture configurations. More specifically, our method is closely related to approaches used in audio models like SoundStream, where training is performed on the first k codebooks where k is randomly chosen each mini batch. This strategy encourages the model to learn a coarse-to-fine structure, allowing hierarchical representation learning and achieving high reconstruction quality at lower bit rates. Similarly, in StoP, randomly truncating the prompt length during training fosters hierarchical learning, improving the model's generalization and adaptability to varying prompt lengths. We modified the manuscript to include this:

"Our approach is inspired by techniques like dropout and stochastic depth , as well as audio models like SoundStream, where randomly selecting the first $k$ codebooks during training enables better generalization."

Q: Did the authors consider doing few-shot versions of their experiments as well

Thank you for your comment. We performed few-shot experiments with 5 examples and have added the results to Table 1. As expected, we generally observe better performance in the few-shot setting compared to zero-shot. Additionally, we plan to include an appendix before nthe end of the rebuttal period to explore the effect of varying the number of examples (k) on performance.

Q: Can the authors please clarify the language used to describe the "anomaly detection" task? In Section 3 it says "the model is tasked with identifying this out-of-place element" but in Figure 1 it says the model has "the goal of predicting the correct event".

Thanks for catching this: we changed the figure to reflect the corrected version. The goal of the anomaly detection task is indeed to identify the incorrect event. We apologize for any confusion.

Q: Did the authors investigate why LASTS + StoP performed so poorly on the Amazon dataset on RMSE, relative to the other models?

Thank you for your question. The poor performance of LASTS + StoP on the Amazon dataset (in terms of RMSE) can be attributed to the dataset's event categorization. Unlike other datasets where event categories are well-defined, the Amazon dataset contains a large number of event types, and the dataset groups a wide variety of event categories into a single bucket, resulting in only 15 event types. This aggregation makes it more challenging for our method to perform well on time prediction, as our approach does not explicitly model time distributions like TPP processes.

We have added the following line to the paper to reflect this:

"The Amazon dataset groups a large number of diverse event types into a limited set of 15 categories to keep the number of event types low. This aggregation makes it harder for our method to perform well on time prediction without explicit time modeling through TPP processes."

Thanks for your detailed answers, you have addressed most of my concerns. Some remarks:

1. I have read through all of the other reviews, and I agree with the need for more baselines. I'm looking forward to seeing the updated results by the end of the rebuttal period.

2. I want to go on the record as strongly disagreeing with some of the criticisms I have seen in the other reviews--

   • I disagree with gifc's (unspecific/non-constructive) criticism that the writing is weak. I stand by my claim that the paper is very clearly written, for the reasons I have already discussed.
   • I don't think that XyhT's criticism about data leakage makes any sense.

3. Another question that I have: where in the paper does it say what the prior SOTA metrics were on these the Asynchronous Time Series tasks for the datasets considered in the paper?

I still think this is a strong paper, and I have raised my score. If the authors can add those additional baselines by the end of the rebuttal period, then I intend to advocate for acceptance of this paper.

Thank you for your comments and suggestions. We have incorporated three additional baselines into our work. The details of these added baselines are covered in the global response above and further elaborated in Section 5.2 and Appendix A.5 of our paper.

Regarding the question about prior SOTA metrics for the Asynchronous Time Series (AST) tasks, it's important to note that while we have robust benchmarks for datasets like Taobao, Taxi, Stackoverflow, Amazon, and Retweet (Section 5.2), the situation is different for the other three datasets we focus on—Breakfast, Multithumos, and Epic Kitchens. These datasets have been explored in isolation for specific tasks, such as forecasting in EPIC Kitchens, but the settings often differ, making direct metric comparisons challenging. Additionally, tasks like Anomaly Detection and Imputation are not widely studied for AST, which limits available SOTA references. We aim to address these gaps by providing a comprehensive evaluation in our work. We will improve our text to better communicate this to the reader.

W: It is not very clear what makes asynchronous time series more difficult than normal time series for LLMs

Thank you for your feedback and suggestion. We have clarified the differences between asynchronous and regular time series in our manuscript to highlight why modeling asynchronous time series is more challenging:

Unlike regular time series, which consist of values at evenly spaced time intervals (e.g., weather measurements), asynchronous time series are composed of multiple types of discrete events occurring sporadically over time. For instance, on social media platforms like Twitter, user interactions (e.g., likes, comments, shares, and follows) happen at irregular intervals. Each interaction type, combined with its timestamp, forms an asynchronous time series. Modeling such data is challenging because of the irregular timing and the diversity of event types, which contrasts sharply with the uniformity and regularity of traditional time series. These differences mean that methods designed for regular time series cannot be directly applied to asynchronous time series without significant adaptation.

W: It seems that many existing methods of LLMs for time series can be easily adapted to asynchronous time series as well. It is recommended that some baselines of the existing LLMs be added for the time series.

Please refer to our  global answer on additional baseline based on LLMs for time series newly included in our manuscript.

W: Although StoP is designed for asynchronous time series, it could also be applied to normal time series … More datasets of normal time series can strengthen the evaluation.

Thank you for appreciating the generality of our proposed StoP. We would like to clarify that StoP has been evaluated on eight datasets in total, as presented in Table 1 and Table 2, including diverse asynchronous time series datasets across different domains. Benchmarks on asynchronous time series like EasyTPP[1]  benchmark five datasets, and we include three additional datasets in our work.  While we agree that exploring the application of StoP to regular time series is an interesting direction, we aim to keep the focus of this paper on asynchronous time series, as it addresses unique challenges such as irregular timing and diverse event types. Evaluating StoP on regular time series would require additional experiments and analyses, which we believe are better suited for a future investigation dedicated to that context.

[1] Xue, S., Shi, X., Chu, Z., Wang, Y., Hao, H., Zhou, F., ... & Mei, H. EasyTPP: Towards Open Benchmarking Temporal Point Processes. In The Twelfth International Conference on Learning Representations, 2024

W: LASTS underperforms in time prediction compared to TPP models for some datasets but lacks sufficient analysis to explain this.

We appreciate the reviewer's feedback. Our method demonstrates competitive performance in time prediction across four out of five datasets, achieving the best results on two datasets and the second-best results on the remaining two. The exception is the Amazon dataset, where our model underperforms. We edited the manuscript to include two analysis for this:

Analysis 1: (algorithmic perspective): “We think that our model is not performing as well as the TPP models, because our model does not have an explicit prior about the time distribution whereas TPP models make strong assumptions about the time distribution (e.g. Poisson process or Hawkes process)."

Analysis 2: (data centric perspective): "In the case of the Amazon dataset, the performance gap is more pronounced because this dataset groups a large number of diverse event types into a single event category, making it harder to model inter-arrival times.” We also responded to reviewer 8vh2 in this regard, who had a similar question.

W: Model architecture Figure 2b only shows "Cross Entropy loss" and how the RMSE calculated.

Thank you for pointing this out. The figure caption indicates that the soft prompts are learned via the next-token prediction loss, which is the standard training objective for LLMs. We modified the figure to make this more clear. RMSE is not a training metric but rather an evaluation metric and does not appear in the figure. It is computed as the root mean squared error between the time intervals predicted by the model and the ground-truth values during inference.

W: While the interpretability of StoP prompts is highlighted … more case studies are needed

Thank you for your feedback. By prompting the model itself, we obtain textual descriptions that offer a broad-level understanding of what the model has encoded in the soft prompts. This approach provides insights into the general structure and task information stored in the learned prompts. We have revised the language in our manuscript to better align with this perspective and included additional examples of these textual descriptions in the appendix A.6 to provide more context.

W: Data leakage across train and test data Thank you for raising this concern. There may be a misunderstanding regarding how event descriptions are handled in our framework. The semantic descriptions of events are indeed consistent between the training and test sets, similar to how category labels remain the same in traditional supervised learning settings. This consistency is intentional and necessary for the model to learn meaningful representations of the event types. The differences between the training and test data lie in the frequency and ordering of events, which reflect the underlying temporal dynamics. These differences ensure that the model is evaluated on its ability to generalize across varying temporal patterns rather than memorizing specific sequences.

Q: Given the zero-shot claims … LASTS be applied to tasks outside .. non-linguistic event sequences

The Table 2 in our manuscript includes results on five datasets that are not textual in nature: Amazon, Taxi, Taobao, StackOverflow, and Reddit. These datasets treat event types as categorical labels rather than relying on natural language descriptions and shows that our framework outperforms various baselines. Additionally, our newly added LLMTime baseline converts our textual datasets into non textual datasets by treating event names as simple category labels and applying Zero-Shot prompting.

Q: Testing the LASTS framework with smaller LLM backbones and non-LLM transformers

We have added results for 1B and 3B LLM backbones in the Appendix A.8, showing performance improvements consistent with scaling laws, where larger models typically perform better. Additionally, Table 2 provides results for TPP models using non-LLM transformers across 8 datasets, highlighting performance differences. However, we clarify that LASTS is specifically designed for large language models, leveraging natural language descriptions of events or categories as text. This reliance on natural language makes it unsuitable for direct testing on non-LLM transformers without significant changes to input representation. These models assume that the inputs are regularly sampled. We hope this addresses your question and clarifies the design and scope of our framework. Please see our response to reviewer G2wj below for additional details.

Q: How is the risk of data leakage mitigated? Please see our response above. We believe there was a misunderstanding; there is no elevated risk of data leakage in our setup.

Thank you for your continued feedback and for acknowledging the improvements we've made to the manuscript. Regarding your concern about the interpretability of learned prompts, we'd like to provide further clarification.

The interpretability we derive from the soft prompts is obtained by probing the model itself. Since soft prompts are continuous vectors without inherent human-readable meaning, this method offers a practical way to assign meaning to them. Previous attempts to interpret soft prompts have involved mapping the learned prompt embeddings back to the nearest tokens in the model's vocabulary (e.g., [1]). However, as shown in [2], this results in sequences that lack meaningful content. In Appendix D of [2], the authors demonstrate that the closest words to the learned embeddings are mostly meaningless, several tokens are mapped to the same word, and the cosine similarity between the tokens and the closest word embeddings almost always falls below 0.16. This highlights the challenges in extracting useful information using this approach.

Our method tries to understand the learned prompts by probing the model to generate textual descriptions describing them. By probing the model in this way, we obtain a clearer understanding of the relevance and content of the learned soft prompts. This provides a better view of the dataset and task information encoded within them. To further address your suggestion for more case studies, we have included multiple examples of model probing results in Appendix A.6, covering different tasks and datasets.

We hope that this additional clarification and the expanded examples addresses your concerns.

References:

[1] Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv preprint arXiv:2104.08691, 2021.

[2] Zhaozhuo Xu, Zirui Liu, Beidi Chen, Shaochen Zhong, Yuxin Tang, Jue Wang, Kaixiong Zhou, Xia Hu, and Anshumali Shrivastava. Soft Prompt Recovers Compressed LLMs, Transferably. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. https://openreview.net/pdf?id=muBJPCIqZT

We sincerely thank the reviewers for their comments and suggestions. We have incorporated three additional baselines in our work:

1. Time Series Foundation Model-Based Baseline: We adapt Chronos [1], a state-of-the-art pretrained foundation model for time series forecasting, as a baseline for forecasting and imputation tasks on asynchronous time series. This provides a stronger and more relevant comparison than heuristic-based baselines.
2. LLMTime [2]: We adapt LLMTime, a large language model-based time series forecasting method, as a baseline for forecasting, imputation, and anomaly detection on asynchronous time series.
3. LLM Processes [3]: We adapt LLM Processes, another LLM-based approach for time series forecasting, as a baseline for forecasting and imputation on asynchronous time series.

Results for these methods are included in Table 1, along with a detailed discussion of their selection, limitations, and performance in Appendix A.5.

With these additions, our work now includes the following sets of baselines:

• Random baseline
• Time Series Foundation Model-Based Baseline (state-of-the-art for time series forecasting)
• LLMs-for-Time-Series-Based Baselines
• TPP Model-Based Baselines [4] (state-of-the-art for asynchronous time series forecasting, Table 2)

We believe these additional baselines provide comprehensive coverage of the key lines of work in the literature that we reviewed in Section 2. If there is a crucial baseline the reviewers feel we left out, we would be happy to investigate it time permits.

References:

[1] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024. https://openreview.net/forum?id=gerNCVqqtR

[2] Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G. Wilson. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 2024.

[3] James Requeima, John F. Bronskill, Dami Choi, Richard E. Turner, and David Duvenaud. LLM Processes: Numerical predictive distributions conditioned on natural language. In ICML 2024 Workshop on In-Context Learning, 2024.

[4] Siqiao Xue, Xiaoming Shi, Zhixuan Chu, Yan Wang, Fan Zhou, Hongyan Hao, Caigao Jiang, Chen Pan, Yi Xu, James Y Zhang, et al. EasyTPP: Towards Open Benchmarking the Temporal Point Processes. International Conference on Learning Representations (ICLR), 2024