Decoupled Marked Temporal Point Process using Neural Ordinary Differential Equations

We thank the reviewer for highly appreciating the novelty and ideas of our work. We provide answers to your concerns below:

• Comparison in computation time across different models? Please see the general response at the top where we provide evaluations for computation time per epoch.
• Reference for THP missing? We appreciate the reviewer for pointing it out, and we have rectified this error with the author name and conducted a thorough review of all other references to ensure correctness.

• Limitations? A major limitation is that we assume that each $\mu(t)$ is independent from other events. Even though such an approach is beneficial for interpretability, it may be limited in learning inter-event relationships. Also, even though Dec-ODE shows comparable training time, it requires longer training time for an iteration than most conventional methods due to the flexible but complex nature of ODEs. However, notice that the actual computation time per epoch is quite reasonable and even better than ANHP (shown in the common question) given our proposed training scheme. Finally, the experimented version of the Dec-ODE can only express a self-excitatory effect, where each event only increases the intensity but not decreases, which makes the model less flexible. Actually, if we design the model to both increase and decrease the intensity simultaneously, the performance becomes better as we have shown in the Appendix A.1. However, in this case, the parallel training scheme in Section 4.3 cannot be adopted as it violates eq. (16) in the paper, and therefore we had to leave it in the Appendix.

• Technical errors? We have corrected all the technical issues addressed by the reviewer and went through the paper again to check for other errors if there were any. These errors can be easily corrected in the text and do not affect the soundness of our model. We really appreciate the reviewer for going through the paper thoroughly to improve its quality.

  • Reference errors for Omi et al, NeurIPS 2019 and Zuo et al, ICML 2020?: Corrected.
  • Complex dynamics depending on the past events is not limited to $f^*(t)$?: Our objective was to underscore the challenge of modeling the probability density function $f^*(t)$ with a parametric model. As the intricate nature of the dynamics involved is quite diverse, describing these temporal variations through intensity functions allows for a more flexible modeling, often achieved with nonparametric methods.
  • Do the authors mean that N_g(t) is a counting process?: The $N_g$ is a temporal point process with a realization $\\{ t_i \\}_{i=1} ^N$. For clarity, the text regarding the counting process in the 1st paragraph in Section 2 is removed for more precise delivery.
  • In 2nd paragraph in Section 2, $\int f^*(t) = 0$?: It is now $\int_0 ^{\infty} f^*(t) = 1$.
  • In Eq. (9), the definition of $f_\theta(ℎ,t)$ is not found?: It should be $\gamma (\bf{h_t}, t, \mathbf{e_i} ;\theta)$ and not $f_\theta ()$, where $\mathbf{e_i}$ is a vector of events corresponding to $\mathbf{h_t}$.

We appreciate the reviewer for the constructive comments, which will greatly help improve the paper's quality and soundness. Please see our answers to the questions below, and we hope to overturn the reviewer's decision.

• Is Dec-ODE computationally more efficient than others? The reviewer raised a good point and we are sorry for the misleading text in Section 3.2. What we intended to argue was that computation with Neural ODEs is heavy by its nature but we mitigated the issue with the characteristics of the differential equation within our proposed framework, simultaneously solving the multi-dimensional differential equation with respect to the hidden state $h_t$. The actual computation time per epoch is given in the comment to all reviewers, where Dec-ODE takes comparable or less time than the SOTA approach. Please see the revised text in the updated manuscript highlighted in yellow starting from the bottom of page 4. We would like to make it clear that faster computation is not the advantage of our model (nor we argue it as a contribution in the introduction), but we proposed an efficient learning of a flexible model which would have taken much longer computation without our proposed optimization scheme. However, we still want to mention that numerical computation of integrals often leads to more accurate results than MCMC, which may lead to faster convergence even if sec/iter may be slower. Moreover, other methods that involve transformer do not consider the temporal nature (e.g., Markovian) of the events and need to perform integration iteratively whenever a new event is introduced.

• Decoupled structure is already well-known? We are aware that the decoupling structure of time and event in MTPP is a general concept as we also mentioned in the Background section for MTPP. However, to the best of knowledge, decoupling individual events through separate intensity functions has been rarely studied. Through some investigation, we found a recent work VI-DPP(Panos et al., AISTATS 2023) which uses the decoupled structure of time and event. We are currently performing experiments with VI-DPP to compare decoupling behavior of Dec-ODE, whose results will be updated soon.

• Why not compare with RMTPP? We appreciate the reviewer for pointing us to the RMTPP as a fundamental baseline. We reported the results only from the latest literature so we missed it, but we are more than happy to include RMTPP results in our paper. The results from RMTPP under our experimental setting are given in the table below which is now updated in the revised paper. The results from RMTPP compared with Dec-ODE can be found in the table below. The implementation provided by the author of RMTPP was employed, with minor adjustments made to accommodate length-varying sequences and present the results in a consistent format. The hyper parameters were found using a grid search from the list of options that can be found in RMTPP Section 6.4 paragraph 5. In most cases RMTPP showed a good performance in the time prediction task with best RMSE on StackOverflow data, and its NLL in MIMIC dataset showed a better result compared to Dec-ODE.

• Implementation details? We feel sorry that we missed these important implementation descriptions. We utilized simple 3-layer Multi Layer Perceptrons (MLPs) for implementing the Neural ODE $\gamma()$ with linear activations. Specifically, softplus normalization on the weights is used for parameterizing $\gamma (\theta)$ as in (Chen et al., ICLR 2021). For decoders, 3-layer MLPs with ReLU activations have been used. All experiments were performed with a single NVIDIA RTX A6000.

• Details on deriving the derivative of the likelihood? The derivative of the likelihood regarding the model parameter is calculated using the adjoint sensitivity method in (Chen et al. NeurIPS 2018), which is extensively discussed in the referenced paper. For detailed description and implementation please refer to the Neural ODEs (Chen et al. NeurIPS 2018) and the public repository (https://github.com/rtqichen/torchdiffeq).

Thank you for the authors’ clarification and efforts. I stand by the point that the proposed model is promising but not clearly impactful. Still I appreciate the effort and I will increase my score to 5.

Is Dec-ODE computationally more efficient than others? Thank you for the comparative analysis about computation time, which makes the computational pros/cons of the proposed method clearer.

Note: I cannot understand why the authors mention about MCMC. The authors might to confuse Monte Carlo integration with MCMC?

Decoupled structure is already well-known? MTPP models with decoupled structures have been actively investigated in the context of discovering latent network structures: For example, (Iwata, 2013) and (Yuan, 2019) adopted exponential decay kernels, and (Linderman, 2014) adopted logistic-normal density kernels. The idea of MTPP with decoupled structure is a straightforward extension of the classical Hawkes process, and is itself rather conventional. The comparison with VI-DPP is of course valuable, but the authors should discuss the necessity of the Neural ODE-based model against the references. Indeed, the idea of using neural ODEs to model triggering kernels in marked temporal point processes is new, but there is no value in using Neural ODE itself.

Why not compare with RMTPP? Thank you for the analysis with RMTPP.

Technical errors? If the authors define $\mathcal{N}_g$ as a realization of {t_i}, they should define $\mathcal{N}_g (t)$ separately, which is used in several equations. If the authors define $\mathcal{N}_g (t)$ as the total number of events occurring before $t$, then $\mathcal{N}_g (t)$ should be introduced as a counting process.

We sincerely thank the reviewer for highly valuing the ideas and presentations given in our paper.

• No simulation studies?: We are currently performing a simulation study with synthetically generated data from Neural Hwakes Process (NeurIPS 2017). We will update the results soon, which will also be discussed in the paper (or in the Appendix). Regarding the computing time, the comparisons on the real-world data are given in the general response at the top of this rebuttal.

Changes in Table1 There has been a slight change to the result table 1 after our careful tuning of the baseline methods, which now has been updated in the revised text. Most of the results are the same but the accuracy of ANHP in Stackoverflow data got slightly better which became the best and ours became the second best.