EasyTPP: Towards Open Benchmarking Temporal Point Processes

For reproducibility purposes, it would be useful to the community to know the exact accuracy values (and confidence bounds) from the various methods. Therefore in my opinion all plots showing empirical results should be replaced with tables. Is this possible in the camera-ready version?

We agree that empirical results should be better illustrated with numbers instead of figures. We have replaced the figure of the main prediction result with a table in the main context and all the other tables are in Appendix D.3. We are committed to replace all the figures with tables in nice formats in the camera-ready version. See [Presentation Improvements].

I may have missed this in my reading, but in Fig 1, I don't quite know what is meant by "Temporal event sequence" as opposed to "temporal point process". Furthermore, isn't a Hawkes process a version of a temporal point process? It seems that these publication sets may overlap somewhat. I'm wondering if this is clarified in the text, or if the authors can clarify.

In fact we have clarified this search process in Appendix E.1. In the revised version, we have added one line to refer to this section for details of how we count the articles. See [Presentation Improvements].

We clarify again here. To find out the number of point process research papers, we search in arxiv with the three terms (Temporal point process, Hawkes process, Temporal event sequence) that are widely used in related publications, respectively. To avoid the overlap of searched results, we search with the following advanced rule:

• Temporal point process: it corresponds to the articles whose abstracts contain the term ‘temporal point process’.
• Hawkes process: it corresponds to the articles whose abstracts contain ‘hawkes process’ but not the term ‘temporal point process’.
• Temporal event sequence: it corresponds to the articles whose abstracts contain the term ‘temporal event sequence’ without neither the term ‘hawkes process’ nor ‘temporal point process’.

In Section 4's "Data Preprocess" section... To my understanding (which may be incorrect), some TPP datasets consist of a single sequence, and some models may process the sequence recurrently, i.e. each state embedding h_t depends only on the previous embedding h_{t-1}. For these models, why are variable-length sequences needed? On the other hand, datasets with events of multiple types (and models that can handle multi-type events) will need variable-length sequences. It would be very useful to the reader if the authors could describe these multiple scenarios, and specifically how EasyTPP handles each one at the implementation level.

Firstly, the dataset each contains thousands or tens of thousands of sequences, where each sequence usually corresponds to the sequence of one user’s behavior (e.g., one user’s reviewing behavior on Amazon). As a result, the variable-length sequence exists (not ‘needed’) in every dataset. See Appendix C for dataset descriptions and Table 2 for statistics of the dataset.

Secondly, because the input is fed batch-wise into the model and there may exist sequences of events that have unequal-length in the same batch, we pad them to the same length batch-wise. This padding mechanism is the same as that in NLP when we pad text sentences.

• Users can choose to decide whether padding to the beginning or end, padding to max length of batch or a fixed length across the dataset. The function interface and usage are almost the same as those in huggingface/transformers package.
• The sequence_mask is used to denote whether the event is a padded one or a real one, which will be used in computation of log-likelihood and type/rmse evaluation as well. An detailed explanation of dataset preprocessing operation can be found at our documentation https://github.com/Anonymous0006/EasyTPP/blob/main/docs/source/user_guide/dataset.rst#expected-dataset-format-and-data-processing

Lastly, to clarify, the input sequence format is the same no matter how many event types the data have, e.g., a sequence is $[(t_1, k_1), (t_2, k_2), (t_3, k_3),...]$, where $k_i$ is a non-negative integer denoting the event type. For all the models, we use an embedding layer (nn.Embedding in torch) to map the event type $k_i$ of each event to a dense vector in higher space; then pass it to the following modules to construct the state embedding $h_i$ recurrently (with $h_{t-1}$). We do not differentiate whether it is a single event type sequence or multi-type events because the codebase can handle both.

The corresponding code of event type embedding can be found at https://github.com/Anonymous0006/EasyTPP/blob/main/easy_tpp/model/torch_model/torch_basemodel.py#L25.

In sum, we have rephrased the paragraph and added Appendix C.1 for a more detailed explanation in the revised paper. See [Presentation Improvements].

Dear Reviewer 2AV3,

As a friendly reminder, we are waiting for your valuable feedback to our responses. In your summary of review, we appreciate recognizing the contribution of our work, and understand the concerns about clarity and presentation. We believe they have been addressed in the rebuttal and revised paper, so please check them out and let us know if you have any remaining questions or concerns. We are more than happy to discuss any further issues. We look forward to hearing from you. Thank you.

Would you mind elaborating further on the broader impact this benchmark will have on the community?

Please see [Technical Contribution and Broader Impact]

As a reminder, the discussion session will end in few hours. We believe we have addressed your concerns on technical contribution. If you have any further doubts or suggestions, we are willing to answer them until the last minutes. So, please let us know if our responses are satisfying or need more revision. We look forward to hearing from you. Thank you.

This paper lacks technical innovation. I don't think a software-level abstraction is a good fit for ICLR.

Please see [Technical Contribution and Broader Impact]

The description of the system design and interface lacks details.

In section 3, we discuss the modules in detail and use figure-3 to illustrate the pipeline of the system.

In section 4, we discuss the interface with two examples of usage (how to customize a model and how to run the pipeline) in Listing 2 and Listing 3.

In Appendix A, from page 13 - 16, we discuss the details of design and interface such as

• High level software architecture (section A.1): figure-8 depicts the detailed architecture with each module’s functionality annotated / explained in text.
• Why and how the system supports both Tensorflow and PyTorch optimization framework (section A.2 - A.3).
• Pseudo implementation to customize a model using the interface of the code base (section A.4)
• Pseudo implementation to run an existed model using the interface of the code base (section A.5)

Besides, an even more detailed description of system and interface could be found at the document provided in our code base: https://github.com/Anonymous0006/EasyTPP/tree/main/docs/source/ref. Please see Readme.md to find out how to generate htmls for better visualization. The explanations of the interfaces at any level are well documented and easy to search and read.

In all, around ¼ of the paper contents, plus pages of online documents, relate to the system design and interface, could you please elaborate which part is still unclear to you？

It's unclear what points the authors intend to demonstrate in the experiment section.

We want to demonstrate the following key insights from the experiment:

• Insight 1: The performance of the classical model MHP is worse than the neural models across most of the evaluation tasks.
• Insight 2: There is no single winner against all the other methods on time and type prediction. Notably, all the best to-date models make poor predictions on time of future events. Moreover, on type prediction, attention-based models only outperform other architectures by a small margin.

These insights have strong implications on our benchmarking work and future research:

• Based on insight 1, we decide that our code base focuses on neural TPPs rather than classical TPPs.
• Insight 2 signals that we should think beyond architectural design for future directions. For the past decade, this area has been focusing on developing new architectures, but the performance of new models on the standard datasets seem to be saturating. Looking into the future, we advocate for a few new research directions that may bring significant contributions to the field, summarized in Section 6 [FUTURE RESEARCH OPPORTUNITIES], including building foundation models of event sequence, empowering the model with large language models etc.

Please let me know if these points are clear for you.

Dear Reviewer MAje,

As the author-reviewer discussion is coming to its end in a few hours, we are looking forward to knowing whether our responses have resolved all your concerns so far, and whether you would like to adjust your evaluation. Thank you.

The authors compare against the classical Multivariate Hawkes Process (MHP) but don't specify what type of kernel they use (I assume exponential) or structure of the excitation matrix.

We are indeed using the exponential kernel for MHP whose dimension equals the number of the event types of the dataset.

We have added the specification for MHP in the revised version. See [Presentation Improvements].

Some typos, e.g. Multivariate Hakwes Process (MHP)

Thanks for pointing this out. We have fixed this typo in the revised version. See [Presentation Improvements].

Is there an easy way to incorporate marks into your package? From the description in the paper, it seems like you can easily handle different event types through a multivariate TPP, but I don't see any discussion of marks. In many application settings, we can have a feature vector for each event, which can be modeled as a mark.

Yes. It is easy to incorporate it in our package. We can modify the input data structure to include a feature vector for each event, and in the input module, we apply a feature encoder to produce a feature embedding and concat it with the event type embedding. This modification needs only a small amount of work and is compatible with the current framework.

We are committed to adding this functionality in the future version. See [Commitment].

It seems like the main concern from other reviewers is regarding the lack of technical innovation. Considering that the paper is submitted to the primary area of datasets and benchmarks, I disagree with this concern. The amount of technical innovation is roughly what I would expect for a datasets and benchmarks paper.

Furthermore, I would argue strongly for the potential significance and impact of this contribution to the growing machine learning community working on TPPs. As a researcher in this community myself, I am much more likely to make use of the contributions from this paper than, say a paper proposing a new neural network architecture for TPPs (which perhaps the other reviewers would rate more highly on technical innovation). There is a tremendous need for high quality software packages for neural TPPs, and the authors are providing just that.