HoTPP Benchmark: Are We Good at the Long Horizon Events Forecasting?

We sincerely thank you for your thoughtful and comprehensive feedback. Your insights and questions have provided us with an opportunity to clarify and further strengthen our work. Below, we address each of your points in detail, highlighting the revisions and improvements made in response to your suggestions.

W1.a Structured descriptions of each event. Thank you for highlighting this ambiguity. We have clarified this point in the updated version of the paper. Specifically, the field of Marked Temporal Point Processes (MTPP) pertains to tabular data processing. Tabular data is structured, meaning it consists of multiple columns of different types [1].

W1.b Challenges of autoregressive prediction. The mentioned sentence directly follows a discussion outlining one of these challenges, specifically the dependency of autoregressive methods on their own prediction errors. We identify and analyze the challenges associated with autoregression in Section 6.3 (Autoregression vs direct horizon prediction).

W1.c How subsequences are handled (line 99). As noted in the text, details on MTPP modeling are provided in Appendix A. Modeling MTPPs involves a deep understanding of probability distribution parameterization techniques and sampling. Rizoiu [2] offers an excellent introduction to these concepts. We chose to include these details in the Appendix because they can be complex to grasp initially and are not essential for understanding the core contributions of our work.

W2. The motivation for the long-term event prediction? The paper provides both theoretical and empirical justification for the importance of long-horizon prediction and evaluation. First, achieving high next-event prediction quality does not necessarily result in good long-horizon predictions. In Section 6.1, we demonstrate that HYPRO, a method specifically designed for long-horizon prediction, often performs worse in terms of next-event quality. This highlights the need for distinct evaluation approaches for different prediction horizons. The theoretical foundations are presented in Section 3. Notably, next-event quality assessment is analogous to OTD with a prefix size of 1, which leads to incorrect accounting for false positives and false negatives. This issue becomes especially important when a model predicts too many or too few events over the horizon.

W3. Long-tail prediction. T-mAP employs macro averaging of individual scores for all classes, ensuring that the prediction quality of each class contributes equally to the final metric. This makes T-mAP particularly effective for studying long-tail prediction, in contrast to metrics like error rate and OTD, which may overlook performance on less frequent classes. To illustrate this capability, we added Appendix I, where we demonstrate how T-mAP effectively evaluates long-tail prediction on the Transactions dataset, which includes a total of 203 classes.

W4. Measuring time quality. Time prediction is a core challenge in the Marked Temporal Point Processes (MTPP) domain. All metrics, including next-event MAE, OTD, and T-mAP, evaluate predicted timestamps. In Section 3, we highlight the advantages of using T-mAP for time prediction assessment. Specifically, we show that T-mAP accurately accounts for false positives and false negatives, a key property lacking in OTD. To address the challenge of irregular timestamps, we included an additional example in Appendix H. In this example, we generated a toy dataset with highly irregular timestamps and demonstrated that both next-event MAE and OTD assign the lowest error to a simple baseline that repeats the last observed event. In contrast, T-mAP favors a baseline that captures time step statistics. This result suggests that T-mAP is better suited for evaluating model performance in scenarios with high irregularity.

Thank you once again for your thoughtful feedback. Please let us know if we have adequately addressed your questions. If so, we would greatly appreciate it if you could consider updating your final score for the paper.

[1] Ryan M. Deep learning with structured data. – Simon and Schuster, 2020.

[2] Rizoiu M. A. et al. Hawkes processes for events in social media //Frontiers of multimedia research. – 2017. – С. 191-218.

As the discussion period is coming to a close, we kindly ask if we have adequately addressed all your questions and concerns. If our responses are satisfactory, we would greatly appreciate it if you could consider adjusting the final score accordingly. Thank you for your time and thoughtful feedback!

We deeply appreciate your detailed review and the time you have dedicated to evaluating our work. In the responses below, we address your comments point by point, highlighting the changes made in the updated version of the manuscript and providing additional context where needed.

W1 / Q1. Why T-mAP is better than OTD. We discuss the limitations of OTD and the benefits of T-mAP in Section 3. There are two main issues with OTD. First, previous works used OTD with fixed-length prefixes. This approach incorrectly counts false positives and false negatives. T-mAP, instead, uses time horizons to compare all events within a specific interval. This ensures accurate counting and is especially useful for models that generate events too rarely or too frequently. Second, OTD relies on hard labels and needs calibrated model outputs to reduce errors. This calibration step was often missing in earlier studies. T-mAP does not have this issue because it is invariant to linear calibration. This makes T-mAP more robust and reliable for comparisons. In the updated version of the paper, we also included Appendix H, which illustrates T-mAP's ability to evaluate highly irregular event sequences, and Appendix I, which highlights its importance in assessing long-tail prediction, to further highlight the benefits of the proposed metric.

W2 / Q2. Experiments covers equally on the topic of the next-item forecasting. The main focus of our work is developing an evaluation procedure for long-horizon prediction. To highlight its importance, we demonstrate how it differs from prior approaches like next-event prediction and OTD. Analyzing metrics such as next-event MAE and error rate is essential to this comparison. Additionally, reporting widely-used metrics improves the benchmark's quality, supports reproducibility, and enables meaningful comparisons in future.

W3 / Q3. Domain importance. For a discussion on the domain of event sequences, please refer to the general response (Domain and benchmark importance).

W3 / Q3. Baselines. Our experimental evaluation includes a wide range of popular baselines that represent diverse methodologies. These include RNNs and Transformers, intensity-based and intensity-free approaches, next-event and horizon prediction methods, discrete and continuous-time models, including ODE-based approaches. Unfortunately, many recent works face challenges with formulation, reproducibility, or scale. For instance, the implementation of ContiFormer [1] is limited to a toy example (a spiral dataset) which requires over 2 hours of training on an RTX 4060 GPU, questioning the actual effectiveness of the approach. Furthermore, the authors have not addressed or commented on reproducibility concerns raised on GitHub. In contrast, we provide implementation, hyperparameters, and full evaluation results for ALL methods considered, achieving exceptional reproducibility in the event sequence domain.

Thank you once again for your thoughtful feedback. Please let us know if we have adequately addressed your questions. If so, we would greatly appreciate it if you could consider updating your final score for the paper.

[1] Chen Y. et al. “Contiformer: Continuous-time transformer for irregular time series modeling”, NeurIPS, 2024

As the discussion period is coming to a close, we kindly ask if we have adequately addressed all your questions and concerns. If our responses are satisfactory, we would greatly appreciate it if you could consider adjusting the final score accordingly. Thank you for your time and thoughtful feedback!

We sincerely thank you for your detailed and constructive feedback. Below, we address each of your points thoroughly, referencing the corresponding updates made in the revised text.

W1. T-mAP motivation. Optimal Transport Distance (OTD) can indeed be computed between sequences of varying lengths. However, previous works have only utilized fixed-size prefixes [1, 2], which lead to the challenges discussed in Section 3, particularly illustrated in Figure 2.1. One of our key contributions is addressing time horizons rather than relying on prefixes of predefined lengths. While it is theoretically possible to implement a form of "horizon" OTD, such a metric would only address one of the two issues outlined in Section 3. The second issue, related to model calibration, requires the use of label distributions rather than hard predictions. Based on these considerations, we conclude that T-mAP is a superior alternative to both OTD and "horizon" OTD. Additionally, T-mAP offers more intuitive hyperparameters, such as horizon length and maximum allowable time error, compared to the abstract insertion and deletion costs associated with OTD. In the updated version of the paper, we included Appendix H, which illustrates T-mAP's ability to evaluate highly irregular event sequences, and Appendix I, which highlights its importance in assessing long-tail prediction, to further highlight the benefits of the proposed metric.

W2. Precision & Recall. Indeed, precision and recall are inherently in a trade-off relationship when varying the decision threshold, as thresholding directly controls the balance between true positives and false positives. However, this trade-off does not apply during the matching process, where the goal is to establish an optimal alignment between predictions and ground truth events. In matching, precision and recall grow together because the alignment maximizes the number of true positives (i.e., correctly matched pairs). Importantly, the total number of predictions and ground truth events remains unchanged, as these are independent of the alignment process. Consequently, the denominators in the precision and recall formulas are constant, allowing both metrics to be optimized simultaneously through matching.

W3. References to datasets and methods in Section 6.1. We introduce both datasets and methods at the start of the Experiments section (Section 6). This structure allows us to maintain a clean and concise presentation for individual experiments, including subsection 6.1. All the methods considered are implemented in our source code, eliminating the need for additional references. Exact links to the datasets are provided in Appendix C.4, and our source code includes data preparation scripts for all datasets (located in the experiments folder). All datasets, except MIMIC-IV, are downloaded automatically. Due to licensing constraints, MIMIC-IV must be obtained individually by each user.

Q1. Deep learning suboptimality and the proposed T-mAP metric. We address this question at the beginning of Section 6.1. Unlike most machine learning models, rule-based methods do not predict label distributions; instead, they produce hard labels. This limitation prevents rule-based methods from being adjusted to specific precision or recall levels. In our experiments, we demonstrate that T-mAP, unlike OTD, effectively captures this distinction between baselines and deep learning models.

Q2. Message of section 4.3. The primary objective of this section is to provide guidance on selecting hyperparameters for the T-mAP metric. We conclude that the T-mAP delta parameter can be set to either twice the cost of OTD or just beyond the initial slope of the delta-quality curve, as smaller delta values impose overly strict constraints on time prediction accuracy. Additionally, we demonstrate that T-mAP is robust to the choice of delta, with small variations in this parameter having minimal impact on the evaluation procedure.

Q3. Sparse and highly irregular data. In the MTPP domain, each event is represented as a point in time. The sparsity of events can be adjusted by simply altering the unit of time measurement. To address irregularity, we have included an additional example in Appendix H. In this example, we generate a toy dataset with highly irregular timestamps and show that both next-event MAE and OTD assign the lowest error to a simple baseline that merely repeats the last observed event. In contrast, T-mAP favors a baseline that learns the average time interval. Based on this analysis, we conclude that T-mAP provides a more accurate assessment of model performance in scenarios characterized by high irregularity.

Please let us know if we have adequately addressed your questions.

[1] Xue S. et al. “EasyTPP: Towards Open Benchmarking Temporal Point Processes”, ICLR 2024

[2] Xue S. et al. “Hypro: A hybridly normalized probabilistic model for long-horizon prediction of event sequences”, NeurIPS 2022

As the discussion period is coming to a close, we kindly ask if we have adequately addressed all your questions and concerns. If our responses are satisfactory, we would greatly appreciate it if you could consider adjusting the final score accordingly. Thank you for your time and thoughtful feedback!

We thank you for your thoughtful and detailed feedback. Your comments have provided valuable insights and have given us the opportunity to clarify and strengthen our work. Below, we address each of your points thoroughly, referencing updates made in the revised paper and providing additional explanations where necessary.

W1/Q1. Limited Justification of Metric Selection. Metrics are a fundamental component of experimental design and cannot be easily validated solely through experimental results. One potential approach is to link the developed metric to human evaluation, effectively providing a proxy for a complex evaluation process. However, in the domain of TPPs, there is no well-established procedure for human evaluation, and even defining a straightforward target for such assessments is challenging. As a result, we justify the proposed metric based on its intrinsic properties, which can be demonstrated both theoretically and through toy examples, as provided in Section 3. In the updated version of the paper, we included Appendix H, which illustrates T-mAP's ability to evaluate highly irregular event sequences, and Appendix I, which highlights its importance in assessing long-tail prediction, to further highlight the benefits of the proposed metric.

W1/Q1. Order-Preserving Wasserstein Distance (OPW). Regarding the OPW metric, it is important to note that it is based on event indices and does not account for timestamps. Since timestamp modeling is a critical objective in the TPP field, we excluded OPW from our comparisons.

W2/Q2. Lack of Fine-Grained Analysis for Different Domains. In the revised version of the paper, we included a comparison across different domains in Appendix F. We demonstrate that T-mAP is a suitable evaluation measure in domains where accurate timestamp forecasting is more critical than precise event ordering. However, in cases where datasets contain a large proportion of zero time steps and the ordering of events with identical timestamps becomes significant, T-mAP may miss some important aspects. Nevertheless, we believe that ordering events with identical timestamps should generally be avoided in practical applications.

W3/Q3. Computational Constraints. We discuss the implemented optimizations in Section 5 (computational effectiveness) and Appendix C.5. HoTPP introduces highly efficient parallel inference procedures, as well as optimized implementations for NHP and ODE. These enhancements achieve up to a 17x improvement in performance compared to previous implementations, which is crucial for handling larger datasets. To the best of our knowledge, this is the first time approaches like NHP and ODE have been evaluated on large datasets such as MIMIC-IV and Transactions. For scenarios with limited computational resources, we recommend using mixed-precision training, which reduces memory consumption and improves training speed.

W4/Q4. Limited Exploration of Next-K Models. Next-K models, such as those introduced in our paper (e.g., IFTPP-K and RMTPP-K), have not been previously applied in the domain of MTPPs. As a result, our work establishes a baseline for the future development of these methods. In Section 6.3, we highlight the potential benefits of this class of models. However, the further advancement of Next-K approaches lies beyond the scope of this work, which focuses on establishing a benchmark and validation procedure for the long-horizon prediction problem.

W5/Q4. Lack of Qualitative Error Analysis. We included a qualitative analysis and a discussion of common issues in Appendix G. As observed, most methods tend to produce constant or repetitive patterns, whereas rescoring with HYPRO generates more diverse and natural outputs. We believe these challenges can be addressed in future research. Although these results are not directly related to our core contributions—namely, establishing a benchmark and metric—we have chosen to include them in the appendix for completeness.

Thank you once again for your feedback. Please let us know if we have adequately addressed your questions. If so, we would greatly appreciate it if you could consider updating your final score for the paper.

As the discussion period is coming to a close, we kindly ask if we have adequately addressed all your questions and concerns. If our responses are satisfactory, we would greatly appreciate it if you could consider adjusting the final score accordingly. Thank you for your time and thoughtful feedback!