Deep Continuous-Time State-Space Models for Marked Event Sequences

We sincerely thank the reviewer for their positive evaluation and for highlighting the strengths in our model design, visualization, and experiments. We are glad that the inductive bias of the latent linear Hawkes layer and the clarity of the figures came through clearly, even for someone unfamiliar with the specific area.

How does S2P2 handle long sequences?

The memory usage for S2P2 scales linearly with the sequence length, enabling it to handle long sequences efficiently in practice. As shown in Section B.5 of the Appendix, we benchmarked inference latency for S2P2 and other models across sequences ranging from 8 to 500,000 events. In these tests, S2P2 exhibited no memory or stability issues during inference.

We expect this stability to extend to training on long sequences as well. This is due to the way the latent dynamics in the latent linear Hawkes layer are parameterized: the hidden states are guaranteed to converge to a steady state over time, which helps mitigate instability in long-range computations.

Summary

We thank the reviewer again for their positive and thoughtful comments. We’re glad that our explanations and figures helped make the core ideas accessible, and we hope our response clarifies S2P2’s scalability and stability on long sequences. Please don’t hesitate to reach out if we can provide any additional clarifications or details to assist your understanding further.

-- Submission 20144 Authors

I appreciate the author’s thoughtful responses to my questions. I keep the original score.

We thank the reviewer for their thoughtful and constructive feedback, as well as their positive remarks on the clarity and motivation of our model. We are glad to hear the paper was a pleasure to read and that the core modeling ideas and empirical strengths came through. Below, we address your questions and suggestions in detail.

S2P2’s Relationship to State of the Art is Unclear

We intended to raise comparisons between S2P2 and other baselines at the relevant points throughout the paper. However, as the reviewer points out, we did therefore forego a single explicit block describing the baseline methods (also partly due to space constraints). We will make the descriptions and distinctions between models explicit in the related works and experiments section in any camera-ready version.

Please also let us know if there was anything specific on regarding baselines that you would like further clarified here!

On Hyperparameter Selection

Thank you for this question. Due to space, we deferred much of this discussion to the appendix. We will use part of the extra page in a camera ready paper to (a) bring up a high-level summary of our hyperparameter search from Appendix C to the main paper, and (b) make the pointer to Appendix C more visible in the main paper.

To summarize here: S2P2’s main hyperparameters are: the number of layers $L$, the hidden state dimension $P$, and the residual stream dimension $H$. Intuitively, $P$ controls the model’s memory capacity, while $H$ limits the rank of the final transform from hidden states to marked intensities. For the experiments in Tables 1 and 2, individual to each baseline and dataset pairing, we performed a grid search over all hyperparameters to select configurations maximizing validation log-likelihood (note we report fully held-out test performance in the tables). The search space and best configurations are detailed in Appendix C.1 and Table 6.

Regarding sensitivity, we generally found a range of reasonable hyperparameters to yield similar performance. We found that multiple layers $L$ was key for performance, but that there was not a critical dependence on depth. The dimensions $P$ and $H$ were more sensitive across datasets with a wider range, i.e., 16 to 128, being typically acceptable. A reasonable starting point is $H=P=32$ or $64$, followed by fine-tuning as needed when computational resources allow.

What is ZOH?

Thank you for spotting this! We accidentally cut any mention of the full definition in Appendix B.2 (and Table 3) from the main text. We will add a short definition, discussion, and a reference to the full definition back to the main text.

For completeness: ZOH stands for zero-order hold, a piecewise-constant interpolation technique applied to discrete-time signals to obtain continuous-time approximations. In our model, exact integration of inputs into the recurrent layers is computationally expensive (to the point of being intractable). Approximating the input with ZOH enables a tractable update rule that closely approximates the true continuous-time dynamics. The ZOH approximation is common in the deep SSM literature (see, e.g., S4D, S5).

Summary

We sincerely thank the reviewer for their careful and constructive feedback, as well as their kind words regarding the clarity and motivation of our work. We hope our detailed responses help clarify the points raised, and we welcome any further questions or requests for elaboration.

-- Submission 20144 Authors

We thank the reviewer for their strong support and thoughtful feedback. We are glad to hear the core contributions and empirical results were found novel and impactful. Please find our response to your questions and comments below.

Weakness 1 (Paper is Dense)

We agree that the math can be a little dense in places. As you say, space constraints meant some exposition was deferred to the appendix (see, for instance, B.2). In the camera-ready version, we commit to using the additional space to add more signposting, intuitive explanations, and clarifying examples to improve readability and accessibility for a broader audience.

Weakness 2 (Latent Space is Non-Interpretable)

This is a good observation. Indeed, compared to parametric Hawkes models, which have more immediate interpretability, our latent mark representations trade interpretability for expressiveness. This tradeoff is common between neural MTPP models in general; S2P2 is similar in interpretability to, for instance, neural Hawkes process. While we did initially note this in the conclusion, we will have a more expanded discussion of this and opportunities this presents for future research. For instance, as also noted by reviewer cv8v, recent work on interpretability in SSMs could be extended to S2P2, which we believe is a promising direction for future work.

Question 1 (Non-Categorical Marks)

Yes, S2P2 can naturally be extended to non-categorical marks with architectural adjustments tailored to the structure of the mark space. For instance, in a spatial setting with marks $k \in \mathbb{R}^2$, one could:

• Embed the continuous-valued mark into a dense vector at the input, similar to how categorical marks are handled;
• At output, rather than computing separate intensities $\lambda_k(t)$ for each mark, model the marked intensity as $\lambda_k(t) := \lambda(t) \cdot p(k | t)$, where $\lambda(t)$ is the total intensity and $p(k | t)$ is the conditional mark density over $\mathbb{R}^2$ (for spatial).

The total intensity $\lambda(t)$ can be computed as in the categorical case, while the conditional density $p(k | t)$ could be parameterized by a Gaussian, mixture model, or more flexible methods (e.g., normalizing flows), all conditioned on the latent state $\mathbf{h}(t)$. This allows S2P2’s continuous-time latent dynamics to fully inform both time and mark predictions. These extensions are a fascinating direction to explore and potentially very impactful. We will add this in the discussion section of the camera-ready version.

Question 2 (Sensitivity to Adversarial Observations)

Formal evaluation under adversarial or corrupted conditions is an open direction for S2P2. We evaluated calibration as part of our suite of metrics, finding that, in general, models are well-calibrated—but we acknowledge we evaluated this under 'normal' conditions. Extending this evaluation to adversarial or corrupted is therefore a very natural follow up. We thank the reviewer for pointing to Chakraborty et al. (AAAI’25) as a starting point, and will include discussion of this in the discussion/conclusion section.

Summary

We again thank the reviewer for taking the time to review our submission, for providing such positive and helpful feedback, and some great suggestions for follow-up works! We are more than happy to answer any further questions you may have!

-- Submission 20144 Authors

We thank the reviewer for their thoughtful feedback and kind words about our paper. We appreciate the recognition of our theoretical framing, empirical results, and clarity. Below, we address each question and suggestion in turn.

On Model Interpretability

This is a great point; incorporating or extracting interpretability from S2P2 is an exciting direction for future work. In relation to the referenced paper: Conditioned on the event times, S2P2 computes update matrices similar in form to Mamba, hence S2P2 is eligible to utilize the method in [1] (and likely other techniques being developed for deep SSMs in general). For [1] specifically, it is worth noting that since S2P2 inherently lives in continuous time, it will exhibit a continuously varying form of this implicit attention. Fully taking advantage of this aspect will require special care, but, excitingly, would provide richer insights when compared to discrete-time attention maps.

[1] Ali A, Zimerman I, Wolf L. The hidden attention of mamba models. arXiv preprint arXiv:2403.01590. 2024 Mar 3.

Question 1 (THP/MHP Decoding Head)

Yes, you are correct; this is the parametric decoding head that we referred to. We describe it as “parametric” because, after conditioning on a history $t_1, …, t_j$, the model encodes this sequence into a single hidden state $\mathbf{h}(t_j)$. The intensity function is then computed as

If we combine terms, conditioned on a particular history, the intensity for each mark simplifies to $\text{softplus}(a t + b)$, where $a,b$ are scalar values representing combinations of parameters/values from the initial expression.

In contrast, S2P2 models the intensity as:

We see now that a constituent of the input to the softplus is $\mathbf{h}(t)$ (instead of $\mathbf{h}(t_j)$). As a result, there is no simplification rendering the intensity as the softplus of a linear-in-$t$ function (as before); it is instead the softplus of a non-linear and history-dependent function of $t$, because $h$ is defined by the entire S2P2 stack and event history. This is why we describe THP as being “parametric”; and why S2P2 can represent a much richer class of intensities across $t$. We will add further clarification of this in the methods section and appendix.

Question 2 (S2P2 as a Pretrained Backbone)

Yes, S2P2 can indeed be used as a pretrained model for downstream tasks. One can use the final hidden state $\mathbf{h}(t_n)$ after conditioning on $n$ events, or alternatively, leverage the latent state trajectory $\mathbf{h}(t)$ across time $t \geq t_n$ for richer representations. Given the expressiveness of S2P2, we agree this direction is promising. Exploring these variants systematically, e.g., via ablations on clinical classification in EHRSHOT, will be a valuable extension for future work. We will add this in the discussion section of any camera-ready version.

Question 3 (Empirical Performance of NHP vs. THP)

All baselines used up-to-date PyTorch implementations provided by the EasyTPP library [2]. The library’s implementations for NHP and THP are adopted from Yang et al. [3]. These versions were explicitly confirmed by the original authors as being correct (see footnote on pg. 8 of [3]). As noted in [3] and confirmed in our experiments, NHP often outperforms THP under this setup. We will clarify these implementation details in the final version.

[2] Xue, Siqiao, et al. "EasyTPP: Towards Open Benchmarking Temporal Point Processes." Proceedings of the Twelfth International Conference on Learning Representations (ICLR). 2024.

[3] Yang, Chenghao, Hongyuan Mei, and Jason Eisner. "Transformer Embeddings of Irregularly Spaced Events and Their Participants." Proceedings of the Tenth International Conference on Learning Representations (ICLR). 2022.

Minor Fixes

Thank you for spotting these typos! We will be sure to correct them in a camera-ready version.

Summary

We sincerely thank the reviewer for taking the time to assess our submission, for the encouraging and insightful feedback, and for pointing out areas where our presentation can be refined. We are happy to clarify anything further!

-- Submission 20144 Authors

Here is a gentle reminder: The Author-Reviewer discussions will end by Aug 8, 11.59pm AoE. Please also be aware that there is a "Mandatory Acknowledgement" for the reviewers.