Decomposable Transformer Point Processes

Thank you for your feedback. We respond to your concerns below:

• Even though we do not present the full thinning algorithm in the main text due to space restrictions, we present the exact algorithm in the appendix on page 17. As we explain in lines 76-83, the expression of the two log-likelihoods in Eq. (1) and (2) are equivalent. Actually, there is a closed-form formula that relates the intensity function with the CPMF/CPDF; a derivation can be found in Section 2.4 in JG Rasmussen, 2018, "Lecture notes: Temporal point processes and the conditional intensity function"

• We have included a table (attached pdf) with an ablation study on the influence of the number of mixture components M. The optimal M is chosen based on the log-likelihood of the held-out dev set as we explain in the Appendix (Section A.2).

• One could potentially interpret the large (or low) log-lkl values as a proxy of the ability of the models to capture the complex dynamics of the event sequences. Larger values may indicate that the model provides a good approximation of the latent generative mechanism of the process however this might not be always true since it is heavily based on the quality of the training/test data and how representative is the sample at our disposal.

• Thank you for spotting the typo in line 80. We will correct this in the revised version of the paper.

Thank you for your feedback and your suggestions. We respond to your concerns below:

1. Notice that we do not learn any intensity functions since our framework is based on the decomposition in Eq. (2). Given the black-box nature of the transformer-based architecture we refrained from including visualizations of the learned representations. This is a general problem for black-box models such as transformers or LSTM/RNN even though there has been some progress in the last few years [*] but in the context of computer vision tasks. To the best of our knowledge, previous works on MTPP using the transformer architecture do not provide such visualizations due to this reason. Only [44](Zuo, 2020) provides a visualization of attention patterns of different attention heads in different layers; however, we believe the results are more confusing rather than providing clarity since they have been arbitrarily chosen from one of the datasets without any insight regarding the configuration of the tranfsormer architecture.

2. This is an interesting point and something that we aim to investigate in the future regarding the modification of the architecture. Nevertheless, tha currently used architecture is already tailored for MTPP since it shares components with [44] amd [41]. We are happy to include this as future direction in the paper. Thank you.

3. We respectfully disagree with your assessment of lack of novelty. We indeed utilize ideas from previous works to build our final novel model but we fail to see why this is absence of novelty. For instance, [30] (Panos, 2023) used the same decomposition which is already known from [6] (Cox, 1975) and the functional form in [27] (Narayanan, 2023) to model the mark distribution. [35](Shchur, 2019) combined a mixture of log-normals with LSTM; all well-known ideas at that time. The continuous-time Transformer architecture for modeling point processes was first adopted by [44](Zuo, 2020) and [43] (Zhang, 2020) independently while [41] (Yang, 2022) later used the same transformer architecture with a modified way to model the intensity function. We also feel that the reviewer has overlooked our experimental evaluation which provides strong evidence of the state-of-the-art performance of our model over well-established baselines. Both the RMSE and ERROR metrics highlight the efficiency of combining a simple mixture of log-normals with a transformer architecture. We believe this is an important contribution and something that was not known to the community until now. The other contribution is the ability of our model to significantly outperform (in a fraction of time) the state-of-the-art HYPRO baseline on the long-horizon prediction task. This result showcases the importance of using a simple yet robust model for the inter-even times as the mixture of log-normals. We are also the first who investigate the limitations of thinning-based methods for the long-horizon prediction task. This result becomes more important if you consider that our method was never developed to deal with the more challenging long-horizon prediction task. We believe these results by themselves are novel enough and would be interesting for the community.

[*] Chefer, et al. "Transformer Interpretability Beyond Attention Visualization", CVPR, 2021

Thank you for your feedback. We respond to your concerns below:

1. Using the decomposition in Eq. (2) is equivalent to using the standard log-likelihood based on $λ_k^∗(t)$ in Eq. (1). For more details, see JG Rasmussen, 2018, "Lecture notes: Temporal point processes and the conditional intensity function". We chose the decomposition in (2) because it allows us to freely define different models for the times and marks. Therefore, we can have models with nice properties (eg mixture of log-normals) without depending on the computationally demanding thinning algorithm for generating samples. We discuss this in lines 89-94.

2. We are unsure what is confusing in these lines. In lines 60-63, we just showcase the contributions of this paper. More details regarding the long-horizon prediction task can be found in Section 5.2. In lines 128-136, we introduce shortly the properties of the mixture of log-normal distributions and how this model can be used for modeling the distribution of inter-event times. The notation is quite standard but we encourage the reviewer to point specifically where the confusion comes from; we are happy to modify the text of the paper to increase readability.