Thank you for taking the time to read and review our submission, and for your high soundness, presentation and contribution scores, and your positive comments about our method!

Your questions were very on-point; so much so that other reviewers echoed all or parts of your questions! We therefore refer the reviewer to the Global Response for our initial response to comments that were shared between reviewers. We then provide more fine-grained responses here.

W1. While log-likelihood is…

Please refer to the global response (“1. Log-Likelihood Results”).

W2. Performing intensity recovery…

Please refer to the global response (“2. Synthetic Experiments”).

Minor: the per-event…

Good spot, we have corrected this.

Q1. If the background…

Yes, it can be through either constraining the background intensity function $\boldsymbol{\nu}_t$ or by rectifying the intensity by passing it into a non-negative activation function, similar to how we do in general for the DLHP in Eq. 13. You are correct, though, that in general, without either of these, it would not be possible to ensure a positive solution. We have added clarification of this.

Q2. Is it possible…

Yes, it is possible to use other discretization methods. This flexibility was explored at length in the original S4 paper [Gu, Goel and Re, 2022, ICLR] and the follow-up work S4D [Gu, Gupta, Goel and Re, 2022, NeurIPS]. It is our understanding that there is comparatively little systematic difference in the performance between the discretization methods. We have added a note of this.

Thank you again for taking the time to read and review our submission. Please do not hesitate to reach out if you have further questions!

-- Submission 3981 Authors

We again thank the reviewer for taking the time to read and review our paper! We refer the reviewer to the Global Response for our initial response to comments that were shared between reviewers. We then provide more fine-grained responses here.

Only NLL was…

Please refer to the global response. We do stress, however, that we presented numerous metrics: the total log-likelihood, both time and mark log-likelihoods, classification accuracy for next event type prediction, and time and mark classification calibration scores. These metrics were presented in full in the appendix. We will add text to the main body highlighting and summarizing these additional results.

Also, as a…

Can the reviewer please clarify what they mean by a “simulation study” in this context? If you are referring to fitting the model to a synthetic dataset with a known ground truth intensity, then we are currently evaluating this (see the global response). If you are referring to something different, then please let us know and we will try and include it!

Q1. In Fig. 4, THP…

THP and IFTPP were not implemented in JAX, instead using the EasyTPP PyTorch implementation. The EasyTPP IFTPP implementation uses the highly optimized, low-level GRU implementation in PyTorch, and so would not benefit from implementing in JAX. We suspect THP may improve in runtime with JAX. However, the main drawback with THP is memory and work scaling; both of which are present in any implementation of the model and are both seen in Figure 4.

At the time of writing, PyTorch did not ship an easy-to-use complex-valued parallel scan method. We therefore implemented a standalone DLHP architecture in JAX to allow us to fairly test the performance with the inbuilt complex-valued parallel scan in JAX.

Since submission, we have actually implemented a pure PyTorch and Triton backend for the parallel recurrence for a real-valued DLHP (since Triton does not support complex values yet). We find that our Triton-PyTorch hybrid implementation is faster than the EasyTPP baselines at longer sequence lengths. We do note, however, our Triton implementation is slower at shorter sequence lengths. We think this is to do with an inefficiency in the compilation/deployment of our Triton kernel. We are actively working to improve this now. Even still, this highlights that DLHP admits faster inference on long sequences in the same environment.

Q2. Demonstration of the…

Can the reviewer please clarify what they would like us to explore in this comment?

If you are asking if there is an accuracy drop through discretization, then the answer is that there is no drop in accuracy. The model is instantiated and learned using the chosen discretization strategy (i.e. discretized at event times) and hence learns representations that eliminate inaccuracies (as opposed to discretizing an existing model, which may incur approximation error and a subsequent performance drop). If the reviewer is asking us to benchmark different discretization strategies, then this is something we could certainly look into. However, we note that most SSM papers do not find systematic performance differences with different discretization strategies [Gu, Goel and Re, 2022, ICLR; Gu, Gupta, Goel and Re, 2022, NeurIPS].

Thank you again for taking the time to read and review our submission. Please do not hesitate to reach out if you have further questions!

-- Submission 3981 Authors

Thank you for your reply! You are right, on reflection, our wording was a little bit strong and this was not our intention. What we intended was that both NHP and DLHP better recover the ground truth intensities when visually inspecting the results (compared to other baselines). We included these plots (with the ground truth intensities) at the end of the Appendix (page 31-32). That said, it is definitely true that all models are doing a decent job of recovering the ground truth intensities (just that NHP and DLHP are slightly better visually). We have updated our initial response to reflect your suggestions.

It is a great idea to also assess the synthetic application quantitatively. We have computed MSE and per-event log-likelihood on a held-out test set. Please find both results in the table below. We find that DLHP is again competitive on both metrics.

We really appreciate your input, which has definitely strengthened our paper and further highlighted the performance of our DLHP model!

Dear Reviewer 15be,

Thank you for your engagement with the updated material. With the discussion period closing soon, we're wondering if you happened to have a chance to review the additional experiments you suggested. We've addressed all your comments and have uploaded a fully revised manuscript for your reference. If you have any further questions, please don't hesitate to ask.

If you're satisfied with the additional experiments and have no further questions, we would be grateful if you would consider raising your score.

Thank you again for your time and helpful feedback.

-- Submission 3981 Authors

Thank you for taking the time to provide a number of really insightful observations and questions! We were very glad to hear you enjoyed reading our submission!

We refer the reviewer to the Global Response for our initial response to comments that were shared between reviewers. We then provide more fine-grained responses here.

W1. I am not very convinced …

Thank you for these great questions. We started this work by trying to incorporate deep SSMs into MTPPs. Fully leveraging the continuous-time dynamics of deep SSMs — as is so natural in an MTPP context to handle variable inter-arrival times — necessitated using discrete impulses and converting the ODE into a stochastic jump process. The resulting process bore a strong resemblance to the canonical LHP. This provided a lens to situate our new model in the existing literature by matching the parameterization wherever possible. ⁠Crucially, these parameterizations were not “design decisions”, but rather natural touchpoints to mathematically define how our model is similar and where it differs.

You are then absolutely right that certain matrices can be factorized in several different ways; but this possibility does not materially affect the theoretical complexity, implementation, etc. of the model. This was a byproduct of drawing connections to the LHP. Indeed, any such factorization could be used – we do not claim this factorization is “better” for performance. For instance, $\mathbf{ABu}_t dt$ resulted from mapping $\boldsymbol{\beta}$ to $\mathbf{A}$ and $\boldsymbol{\nu}_t$ to $\mathbf{Bu}_t$ from the original term $\boldsymbol{\beta}\boldsymbol{\nu}_t dt$ term in the LHP.

We did however introduce $\mathbf{E} \boldsymbol{\alpha}$ to facilitate a low-rank factorization of the mark embeddings (i.e. $\mathbf{E}$ is a tall, narrow matrix). If there is a large mark space (as in EHRShot) then the parameter count in the embedding matrix would be enormous.

W2. On the other hand, ...

We believe this is a little bit of a misunderstanding. In the core formulation in Equation (10), the latent dimensions can interact, as $\mathbf{A}$ is dense. We then diagonalize the system through a linear rotation into (and out of) the eigenbasis. This diagonalization does not modify the underlying dynamics.

Put another way: while each dimension of $\mathbf{x}$ is independent in time after diagonalization, the channels are mixed through the input matrix $\mathbf{B}$, the emission matrix $\mathbf{C}$, and the position-wise nonlinearity. Furthermore, events with a specific mark $m$ affect all dimensions of the hidden state through the mark-specific impulse $\mathbf{E}\boldsymbol{\alpha}_m$.Therefore, although each latent dimension is independent (i.e. they are in the eigenbasis of the layer), the marks actually do communicate because the eigenbasis of different layers is different, because of the mixing in the projection matrices, and because of the position-wise non-linearities.

Marks would only be independent if every DLHP layer, in the notation of Equation (10), were restricted to have: a diagonal input matrix $\mathbf{B}$, diagonal output matrix $\mathbf{C}$, diagonal feedthrough matrix $\mathbf{D}$, and diagonal effective impulse matrix $\mathbf{E}\boldsymbol{\alpha}$ for all layers (as well as a diagonal $\mathbf{W}$ to transform the last layer’s output into intensity values). To be clear, this would be a very severe set of additional constraints on our proposed model.

As for interpretability: you are correct that $\boldsymbol\Lambda$ and $\boldsymbol\alpha$ have limited interpretability in and of themselves. That being said, we have included a more general discussion of interpretability in the response to Reviewer PxVB (search for “W2. The proposed method”).

We have clarified all of these points at the end of the methods section. We are more than happy to answer any more questions you have on this topic!

W3. I would suggest…

Please refer to the global response.

W4. No standard deviation…

Please see the global response. The key takeaway is that DLHP still performs comparably or better than the baseline models across multiple random seeds. Thank you for highlighting this.

(Continued)

W5. It would be better…

This is a great suggestion. We have included a table below (and in the manuscript) defining the work, memory and parallelism of the methods in Big-O notation. $L$ denotes the sequence length, and $M$ denotes the number of Monte Carlo samples used in evaluating $\int \lambda_t d t$ in Eq. (2). As IFTPP is an intensity-free method with parametric assumptions, it does not need to estimate $\int \lambda_t d t$ as the other methods do.

The crucial point is that DLHP matches the best-performing baseline in every category. The empirical speed evaluations presented in Figure 4 of the original submission corroborate these results.

Q1. The main text…

Thank you for highlighting this. This is a slight oversight in Figure 2. The input, $\mathbf{u}_t$, is $\mathbf{0}$ for layer one (the input solely comes through the mark impulses $\mathbf{E}\boldsymbol{\alpha}d\mathbf{N}_t$ on layer one). We used $\mathbf{u}$ to indicate inputs and events here, but in hindsight, this is not clear. We have updated the caption to explicitly state this.

Q2. Do the authors…

At the time of writing, we do not have insight on this. We explored multiple avenues, e.g. is one preferable or larger/smaller mark spaces, heavier-tailed inter-arrival time distributions, etc, but found no compelling insights. We then treated it as a hyperparameter search (see Table 9), and then selected backwards discretization (as it was slightly better on average) to use as a single, unified architecture for our main metrics table. We have added exploring this as an avenue for future work in the conclusion.

Q3. What do the…

The 1.4 corresponds to the average multiplicative improvement in the likelihood between our method and the best baseline, so higher is better. For the tables that we produce, higher log-likelihood values are better, whereas a lower rank is better (with rank = 1 being the best). We have clarified both in the text.

Thank you again for taking the time to read and review our submission. We hope our comments have allayed your concerns. Please do not hesitate to reach out if you have further questions!

-- Submission 3981 Authors

Firstly, we thank the reviewer for their positive comments towards our work, particularly highlighting how we are the first to fully leverage the opportunities of deep SSMs for TPPs and the presentation of our work.

We refer the reviewer to the Global Response for our initial response to comments that were shared between reviewers. We then provide more fine-grained responses here.

W1. As the authors clarified ...

We apologize, there might have been a lack of precision in our language for our submission. In line 480, we use the complexity of the implementation to refer to the fact that the fast implementation requires a parallel scan implementation, as well as a conventional sequential scan, i.e. a for-loop-based implementation. This was in contrast to other models, such as the neural Hawkes process (NHP), which only supports a sequential scan. This is purely an implementation detail, and doesn’t change the expressivity of the model family or parameter counts of the model. We have refined the language throughout the paper when discussing these points. Thank you for highlighting this oversight.

It is worth also noting that throughout our experiments we have controlled the parameter count of different models to ensure roughly equivalent model capacity across methods and datasets. If neither of these points addresses your original concerns please let us know and we can try to clarify!

W2. The proposed method …

Thank you for this question, it is something we actually thought about prior to submission. Similar to other neural MTPP methods, interpreting the model parameters, latent states etc is extremely difficult. However, the unique SSM formulation does offer some limited opportunities for interpretation: due to the LLH layers effectively writing to and from a residual stream with mark-dependent impulses, we can build a coarse approximation of the instantaneous interaction of mark types by summing the impulses across the layers analogous to the original interpretation of the $\boldsymbol{\alpha}$ parameters of linear Hawkes processes. However, doing so requires neglecting certain non-linearities and temporal dependencies, and hence is a very crass approximation.

However, your question on interpretability has an opportunity beyond just interpreting the parameters:

• As the DLHP is stateful, we can approach interpretability by dimensionality reduction and clustering/decoding of the latent states (see e.g. Mitchell-Heggs et al [2022, J Comput Neurosci] or Maheswaranathan et al [2019, NeurIPS]). This type of analysis is not afforded by attention-based methods. These methods provide richer descriptions and analyses of trajectories and model computation.
• As the LLH is a deep SSM, as interpretability and understanding of deep SSMs grows, our DLHP automatically inherits these interpretability methods.

We will include a discussion of these points in any camera-ready version of the paper.

Q1. The authors may …

We did briefly discuss the contemporaneous (under ICLR’s definition here) Mamba Hawkes Process (MHP) in the related work section. To clarify our statement, we were specifically referencing how our model both encodes events and decodes intensity values with the same SSM, whereas the MHP uses Mamba only as an event encoder. MHP intermediate intensities are decoded as a fixed parametric function of time and the hidden state. Contrasting this with our model, the same latent dynamics are used to both encode events through impulses and evaluate intermediate intensities through continuous-time differential equations. This is why we describe the DLHP as being the first method to fully leverage the capabilities of deep SSMs, as opposed to using a deep SSM as a highly performant encoder. We have expanded this discussion in the main paper.

Thank you again for taking the time to read and review our submission. Please do not hesitate to reach out if you have further questions!

-- Submission 3981 Authors