LAST SToP for Modeling Asynchronous Time Series

We appreciate the reviewer's positive evaluation, thoughtful suggestions, and recognition of our contributions, including the LASTS representation and the Stochastic Soft Prompting (SToP) mechanism. We respond to each comment below:

Random Initialization: Thank you for highlighting this relevant literature. We evaluated pretraining on the Breakfast dataset's forecasting task and observed clear benefits: fine-tuning a randomly initialized model yielded F1-score 0.14 vs 0.26, Accuracy 0.21 vs 0.39, and MAE 39.29 vs 32.55. This highlights the value of text pretraining in our asynchronous setting, given the rich natural language input. We’re happy to include this as an additional baseline in the main table of the camera-ready version if the reviewer recommends it.

Redundant Notation (Lines 210–215): Thank you for pointing this out. We will remove the redundant notation.

Clarification on LORA: Our goal in this section was to show that LASTS integrates easily with existing PEFT methods like LoRA and Soft Prompting for adapting an LLM backbone. We acknowledge the wording may have caused confusion and will revise it to clarify that we use the original LoRA implementation from [1]

[1] https://github.com/huggingface/peft

Questions

1. We answer this in two parts:

• SToP Generality: We agree that SToP may have broader applicability, but the focus of our current work is specifically on asynchronous time series. Exploring SToP in other NLP domains is beyond the scope of this paper and is left for future work.
• Comparison to Other Prompt Tuning Techniques: We include the most widely-used PEFT methods—Soft Prompting and QLoRA—as strong representatives for LLM adaptation; please see our response to Reviewer L5nG titled "Comparison to Other Prompt-Based Adaptation Methods" for further details.

2. Clarification on use of semantic information: We appreciate the reviewer’s careful reading

• The three datasets in Table 1—Breakfast, MultiTHUMOS, and EPIC-KITCHENS—contain semantic information that is available to the model and used in our experiments. (L270-274)
• For the five TPP datasets shown in Table 2, event names are replaced by categorical indices, and neither our model nor any TPP baseline uses semantic information. (L264-267)
• The experiments in Appendix A.4 using gibberish and textual descriptions are controlled ablations designed to isolate the effect of semantic content.

These experiments demonstrate that our model is flexible and effectively utilizes semantic information when it is available. We will make this clarification more prominent in the camera version of the manuscript.

3. Finetuning LLMTime, LLM Processes: We focused on using these models in their zero-shot capacity, as they were specifically proposed and pretrained for that purpose. Since these methods are compared against LASTS in the same zero-shot setting, we believe our comparisons are fair and meaningful, especially to assess generalization without additional tuning. We also highlight this zero-shot comparison in our main paper (L381–384), Figure 5, and Appendix A.6. We agree that fine-tuning LLMTime and LLMProcesses could be interesting baselines, but it may require additional effort to determine the optimal fine-tuning recipe and ensure a fair evaluation between models. Therefore, we will consider it as part of our future work..

4. QLoRA vs Full Finetune: We chose QLoRA because it allows parameter-efficient adaptation with low memory cost, aligning with our goal of scalable deployment. As Table 1 and Appendix A.9 show, QLoRA performs competitively, and SToP improves upon it while using only 0.02% of model parameters. While full fine-tuning could potentially yield slightly improved performance, it would be computationally impractical given the limitations of our current hardware.

5. Simpler non neural baselines: We thank the reviewer for this suggestion. Our primary baselines were state-of-the-art TPP models and recent LLM/PEFT techniques. Most recent literature on asynchronous time series modeling has shifted toward neural approaches, and we follow this trend to maintain consistency with comparable works. In non-neural TPPs, the features and functional forms used to model event intensities, dependencies, and histories often need to be manually designed or based on assumptions. This restricts the model's ability to generalize to complex datasets. In contrast, neural models offer greater flexibility in modeling the intensity function, enabling them to capture intricate relationships within the data. This adaptability makes neural TPPs better suited to handle diverse and complex datasets, improving their generalization and performance.

Once again, thank you for your encouraging and constructive review. We are grateful for the thoughtful feedback.

Thank you for recognizing the creative integration of concepts, practical applicability, innovation in training techniques, and comprehensive analysis in our work. Responses to key points raised:

Chronos: Limited analysis; inclusion of TEMPO: Chronos performs poorly as expected, given its reliance on time series–specific augmentations and synthetic data, which make it ill-suited for asynchronous time series due to their fundamental differences (L046–L020). We included Chronos as a representative general-purpose TS model to highlight how such assumptions hinder performance, unlike models like LLMTime and LLMProcesses that incorporate fewer biases. which likely contributes to their stronger performance.

Thanks for pointing us to TEMPO; it faces similar limitations due to its reliance on seasonal and trend decomposition—concepts not meaningful for asynchronous sequences. These comparisons underscore the need for purpose-built approaches like LASTS. We will revise the manuscript to clarify this discussion.

NeuralODEs, LipCDE: Thanks for bringing up NeuralODE-based approaches like LipCDE. NeuralODEs are generally ill-suited for modeling asynchronous time series due to two key limitations:

1. In ODE systems, the future trajectory is entirely determined by the initial state, which implies that modeling long asynchronous time series would require the initial state to encode all future observations—an unrealistic assumption for most real-world data.
2. The continuous trajectories assumed by NeuralODEs make them ill-suited for capturing abrupt changes or irregular time gaps, which are common in asynchronous time series with discrete, sudden shifts.

Thus, NeuralODEs are only appropriate when underlying dynamics are deterministic and continuous, which is rarely true in practice. To our knowledge, no NeuralODE-based models have been evaluated on the datasets used in this paper.

Weaknesses

1. Domain-Specific Analysis: We agree that domain-specific analysis can offer additional insights. In our work, we chose to focus on generalizability across 8 diverse datasets spanning 7 distinct domains given the broad ICML audience which makes it difficult to derive domain-specific conclusions. However, we make following observations to be included in teh camera-ready version:

• Online Shopping (amazon): As discussed in L371–373, the Amazon dataset includes a mix of unrelated event types grouped under one label, which possibly hurts time prediction.
• Cooking: Datasets like Breakfast and EPIC-KITCHENS show strong performance, as our model benefits from rich natural language descriptions and meaningful event sequences.
• Sports (MultiTHUMOS): This dataset features more Markovian event transitions, making forecasting relatively easier but anomaly detection harder.

2. We answer this in two parts:

• Theoretically, why SToP is better than SP: Our focus in this work is to demonstrate empirically, across multiple tasks and datasets, that SToP outperforms soft prompting—both in performance (Table 1) and in the structure of learned tokens (Section 4.5). We agree that a theoretical foundation would be valuable, but given the complexity, we are exploring this as future work.

• Coarse-Fine Structure Emergence: This emergence is a direct consequence of random prefix length selection during training which encourages early tokens to capture general patterns, while later tokens refine the representation. Similar behaviours are observed in prior work discussed in our manuscript (L254-260) :

  • Soundstream's residual vector quantization (RVQ), where selecting a random number of quantizers during training leads to coarse-to-fine audio reconstruction, and
  • Matryoshka Representations, which explicitly optimize for informative prefixes.

  We will incorporate this discussion in the final version of the paper.

3. Practical Implementation Details: We're happy to clarify any remaining implementation details, revise the manuscript as needed, and will release the code upon publication.

Other Comments: Examples and Visualization: Thank you for the suggestion. Due to space constraints, we cannot include this analysis in the rebuttal but will incorporate it in the camera-ready version.

Questions

1. What do we learn form Fig 5 & 13: We hypothesize the emergence of a coarse-to-fine structure in SToP, where earlier tokens capture diverse high-level features, and later tokens refine them:

• Fig5 the t-SNE projection shows that the first 100 tokens in SToP are more spread out compared to the clustered tokens in standard soft prompting. This indicates that SToP learns more diverse representations in earlier tokens.
• Fig13: shows that the cosine similarity between adjacent tokens is lower at the beginning of the SToP prompt and gradually increases, consistent with a coarse-to-fine pattern. No such organization is observed in standard soft prompting.

Thank you for your thoughtful review.

We sincerely thank the reviewer for their thoughtful summary and generous assessment of our work. We are glad that the core contributions—LASTS and Stochastic Soft Prompting (SToP)—were found meaningful and well-supported. Below, we address the specific questions raised:

Train/Validation/Test Splits

We follow the standard protocol for each dataset as adopted in prior work (e.g., [Xue et al., 2024]). The splitting is done at the sequence level—given a dataset of N independent sequences, we use 80% for training, and 10% each for validation and testing, sampled independently from a shared distribution. The model thus learns patterns governing sequences of events and their interarrival times from the training set, and generalizes to unseen sequences in the validation and test sets. We use the standard protocol because it enables us to compare our results with those already published.

We acknowledge that traditional time series tasks require chronological splits to account for drift. However, in our datasets (e.g., EPIC-KITCHENS, MultiTHUMOS), the sequences are short, self-contained, and typically do not exhibit long-term temporal drift. This mitigates the need for chronological splitting. Nevertheless, we appreciate the reviewer’s concern and will make our data split methodology more explicit in the final version of the paper.

Comparison to Other Prompt-Based Adaptation Methods

Thank you for this suggestion. Many adapters have been proposed recently, and it would not be feasible to compare all of them. Therefore, we decided to focus our comparison on widely-used PEFT techniques: Soft Prompting and QLoRA. QLoRA is the standard adapter-based method and strong baseline, and we highlight that LASTS is compatible with such adapter techniques. Although we have not yet tried prefix tuning, there are no methodological limitations that would prevent its use with stochastic training strategy (SToP), and we plan to investigate this in future work.

Once again, thank you for your encouraging and constructive review. We are grateful for the thoughtful feedback.

We sincerely thank the reviewer for their detailed review. We appreciate the recognition of our method's novelty, the thoroughness of our experimental evaluation, and the relevance of our chosen tasks and benchmarks. We also thank the reviewer for highlighting how our work aligns with broader scientific trends—adapting LLMs to new domains, advancing parameter-efficient fine-tuning techniques, and challenging the reliance on specialized architectures.

We acknowledge the two specific areas of improvement identified and address them below:

1. Relatively weaker time prediction performance compared to TPP models: We agree with the reviewer, however, our design prioritizes simplicity, general applicability across tasks (forecasting, imputation, and anomaly detection), and effective use of diverse natural language event descriptions—without explicitly modeling time. This leads to our model ranking as best on 13 and in the top 2 on 17 out of 18 evaluations in Table 2. Introducing explicit time modeling is a valuable next step, which we are actively exploring as future work.

2. We answer this in two parts:

• Anomaly detection baselines: This task remains underexplored in the context of asynchronous time series. While we have adopted someTime Series methods (e.g., Chronos, LLM Processes) to the asynchronous setting for forecasting and imputation, they cannot be easily extended to anomaly detection as they are very forecasting focused and anomaly detection is not easily recast as a forecasting problem. We see our work as an early step toward bridging this gap.

• Inclusion of industrial datasets: We used publicly available datasets from standard benchmarks widely adopted in the literature to ensure fair comparison with both traditional TPP models and modern time series forecasting methods. We agree that incorporating additional real-world industrial datasets could further strengthen the evaluation.

We thank the reviewer again for their insightful feedback, and are happy to answer any additional questions.