Transformers for Mixed-type Event Sequences

We thank the reviewer for their detailed comments and suggestions, as well as the helpful pointer to the literature:

What does this paper add that is not in [https://arxiv.org/abs/2306.11547 and https://arxiv.org/abs/1909.12127]

In short: Our model generalizes and significantly outperforms both approaches. These are our contributions over the literature:

1. Intensity-free outperforms intensity-based. Previous work argued that intensity-based methods dominate due to "fewer modeling restrictions" [https://arxiv.org/pdf/2412.19634, Section 4]. Our approach brings intensity-free modeling back on the table, outperforming all intensity-based methods (except for DLHP in one dataset, see Table 1), all while avoiding numeric integration at training and rejection sampling at inference time.
2. Point processes beyond homogeneous marks. To the best of our knowledge, FlexTPP is the first universal point process method for heterogeneous mark spaces. Event Stream GPT (https://arxiv.org/abs/2306.11547) does not use a normalizing flow for continuous marks, so we can model distributions more accurately.
3. Generalizing point processes to new tasks. The generalized framework allows modeling structured prediction tasks as point processes, which we show with the successful annotation of events in a dense time series. Event Stream GPT is built to predict continuous marks with a Gaussian, which deteriorates sample quality over time and prevents sampling sequences.
4. Successful extraction of triggering kernels. Based on the suggestion by reviewer M7Vi, we are adding an evaluation on the interpretability of our model to the camera ready version. See our answer to M7Vi for details.

Let us now give more details on the two papers.

Difference to https://arxiv.org/abs/2306.11547

We thank the reviewer for pointing us to Event Stream GPT (ESGPT). The key difference is that ESGPT models model continuous marks using unimodal Gaussians, whereas our FlexTPP uses normalizing flows, which can approximate arbitrary distributions.

This flexibility leads to better generative performance: FlexTPP(-C) achieves lower NLL than ESGPT(-C) on all twenty but one electronic health dataset. For example, on the first three datasets in Table 3:

Median improvement: 0.10 nats, vs. std dev of 0.009 nats. The only case ESGPT outperforms is on Anemia (0.220 vs. 0.238).

The mismatch between a Gaussian and arbitrary distributions becomes especially pronounced when generating longer sequences. Because the model is autoregressive—using its own outputs as inputs for future steps—errors from the incorrect Gaussian assumption compound over time. We confirm this on the event annotation task (Table 4):

These results illustrate the critical benefit of modeling continuous marks expressively. While the data encoding is identical, our approach is both more general and performs better in practice. We will include ESGPT as a baseline in Tables 3 and 4 and cite the paper in the final version.

Difference to https://arxiv.org/abs/1909.12127

Intensity-free Temporal Point Processes (IFTPP), like our FlexTPP, model the arrival time of the next event using a flexible one-dimensional distribution. We adopt their point process framework in Section 3.4, lines 96-98.

However, the authors do not model more complex mark spaces such as variable-length and mixed-type.

In addition, on the EasyTPP benchmark, our approach outperforms IFTPP in all cases.

https://arxiv.org/abs/1909.12127 [...] found that normalizing flows are not helpful

The authors of that paper refer to particular expressive one-dimensional bijections that are expensive for sampling since they require numerically solving for an inverse (see their section 3.1). Neural spline flows [https://arxiv.org/abs/1906.04032] overcome this bottleneck since they are based on RQ splines, which have an analytic inverse function.

Regarding 4.2: How many patients are there per condition? How are you addressing the issue of multiple labels per patient inherent in EHR data? In Table 3 when you say "Lab tests w/o values" are you excluding the values only as prediction targets or as both prediction targets and features?

1. Between 64 and 1515 patients (average: ~700 patients), of which we use 70% for training, and 15% each for validation and testing.
2. We train one model for each condition, and filter the data for the relevant events: top 10 CPT-4 treatments for patients with that disease, top 5 lab tests for patients with that disease (see Appendix A.2)
3. We are also excluding the values as features. The left part of Table 3 evaluates Eq (13), which measures the quality of predicting treatments given an increasing number of longitudinal covariates (first column: only previous treatments, second column: first column + time and type of lab results, third column: second column + lab results, fourth column: third column + demographic information).

No notion of uncertainty or statistic soundness of many of the comparisons. This is especially important for small datasets like EHRSHOT.

We report standard deviations in Appendix A, Tables 6, 9, 15, confirming significance. We will make the wording more precise on the significance of the difference between FlexTPP and FlexTPP-C in Table 3.

Ablation whether particular components (like the normalizing flows part) are helpful

For now, please refer to the data presented in the EasyTPP benchmark in Table 1 of our paper (comparing the setup of IFTPP in https://arxiv.org/abs/1909.12127 which uses a log-normal mixture instead of our spline flow), as well as the new data on https://arxiv.org/abs/2306.11547 for the EHRSHOT data, which uses a regression head for continuous marks instead of the spline. We propose adding a dedicated ablation on the EasyTPP benchmark comparing autoregressive backbones (attention vs RNN) as well as one dimensional distributions (intensity vs regression vs mixture vs normalizing flow) to disentangle the contributions.

The conditional handling seems overly engineered. Why not just add it as an event at the start of the sequence?

Indeed, we do that for the EHRSHOT data. But for feeding time series data, it is inefficient to have 10000 input timestamps each have its own token. The conditional network therefore preprocesses the data.

We hope that this addresses the concerns of the reviewers and we are happy to address any further questions in the subsequent discussion.

We thank the reviewer for the helpful comments. We particularly appreciate the call to list contributions from a point process point of view.

The model conditions each mark component on all previous times and marks in a fully autoregressive manner, which can be unnecessarily restrictive or misrepresent the data. For example, blood pressure and heart rate are recorded at the same time for a patient, but there is no natural dependence on each other.

The described knowledge about the true dependent structure can be integrated into our model: If there is prior knowledge about the (in)dependence between certain predictions, this can be guaranteed to be fulfilled by the model by masking out the corresponding attention weights in the transformer.

For example, assuming we want to predict $p(\text{blood pressure}, \text{heart rate}|\text{past events})$ as independent variables, we can zero out entries of the fully autoregressive attention mask:

\\begin{pmatrix} 1 & 0 \\\\ 1 & 1 \\end{pmatrix} \\mapsto \\begin{pmatrix} 1 & 0 \\\\ 0 & 1 \\end{pmatrix}

This results in an exactly independent prediction $p(\text{blood pressure}, \text{heart rate}|\text{past events}) = p(\text{blood pressure}|\text{past events}) p(\text{heart rate}|\text{past events})$.

While the approach is well-executed, it largely applies standard sequence modeling to flattened temporal data. [...] Can you specially list what has been done more than above, and what's the contribution/innovation from a point process view.

From a point-process view, we contribute:

A new state-of-the-art for intensity-free modeling.

Contrary to the original IFTPP paper, we find that normalizing flows are a highly effective basis for intensity-free modeling (see their comment on normalizing flows in section 3.1 in https://arxiv.org/pdf/1909.12127). We combine a transformer backbone with neural spline flows, achieving closed-form likelihoods and fast sampling. This outperforms their approach (Table 1).

Intensity-free outperforms intensity-based.

Previous work argued that intensity-based methods dominate due to "fewer modeling restrictions" [https://arxiv.org/pdf/2412.19634, Section 4]. Our approach brings intensity-free modeling back on the table, outperforming all intensity-based methods (except for DLHP in one dataset, see Table 1), all while avoiding numeric integration at training and rejection sampling at inference time.

From a high level, this strengthens the connection between the point process and generative modeling literatures.

Point processes go beyond homogeneous marks.

We show that point processes are a natural model for generative models of series of complex events such as patient records and time series annotations.

[Note that reviewer ByqH pointed us to related work [1], which encodes data in a similar way to our model. However, their model regresses continuous marks instead of modeling them with a universal distribution. This is not expressive enough and empirically deteriorates under autoregressive sampling.]

Blackbox methods allow for interpretation. [Thanks to suggestion by reviewer hdCQ]

Despite being a blackbox method, our model allows extracting information about the underlying point process. In detail, we replicated an experiment from [2] by successfully extracting a triggering kernel from our model. Our extraction closely follows the ground truth compared to extracting the relation between events from the attention weights, which is the baseline in that paper.

Again thanks for pushing us to think more about the point process perspective. We propose overhauling the list of contributions in our introduction section.

• [1] Matthew B et al. "Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events", NeurIPS 2023.
• [2] Isik et al., "Hawkes Process with Flexible Triggering Kernels", MLHC 2023.

The likelihood values are reported in the experiments, but not sure how significance the improvement is without statistical test.

We report standard deviations in Appendix A, Tables 6, 9, 15, confirming significance. We will make the wording more precise on the significance of the difference between FlexTPP and FlexTPP-C in Table 3.

We hope that this clarifies the concerns of the reviewers. We are happy to provide further details in the subsequent discussion.

We thank the reviewer for the detailed, diverse, and helpful comments.

We begin with main limitations, followed by specific responses.

The main limitations [...] are its complexity relative to baselines and its lack of interpretability.

Let’s start with interpretability:

The paper should also benchmark the interpretability aspect and compare it with classical Hawkes and interpretable neural Hawkes approaches, e.g., [1, 2].

That is an excellent suggestion! Building on the reviewer’s references [1, 2], we find that our model offers a comparable level of interpretability—albeit in a local, query-based form—by targeted inspection of how past events influence future intensities.

Interpretability through targeted intensity inspection

In Hawkes Process with Flexible Triggering Kernels [1], the method learns a kernel $q_{m_1 m_2}(|t_1 - t_2|)$. It measures how an event $(m_1, t_1)$ influences the intensity of a future event $(m_2, t_2)$.

Similarly, Neural Spectral Marked Point Processes [2] learns kernels $q(m_1, t_1, m_2, t_2)$ that compose the intensity for the next event.

We can extract these kernels using our Eq. (2, left), which computes intensities from the event pdf and cdf.

We experimentally confirm that our model accurately predicts intensities such as $\\lambda(t_2|t_1)$ via Eq. (2, left) for a synthetic example from [1, Figure 2]. We find that the RMSE distance between the true kernel and the one extracted from our method is $0.01$. You can imagine Figure 2 in reference [1] with our line closely following the true kernel.

We will add the above result as an interpretability section to our paper, and are happy to provide any additional details to the reviewer upon request.

Global vs. Local Interpretability

Taking a step back, there is an inherent tradeoff between globally interpretable models and highly flexible ones.

Structured approaches that learn kernel-based pairwise event influences allow for a simple complete understanding of the model’s behavior. However, they are limited in flexibility and can misrepresent data with nonlinear dynamics. For example, in marketing, an effect such as ad fatigue can’t be modeled, where seeing an ad once or twice increases the chance of conversion, but repeated exposures eventually stop having any effect or may even cause disengagement. The ground truth structure of real data is usually not clear and so interpreting a kernel-based model can yield spurious explanations.

While our model is not globally interpretable, it supports local interpretability through queries about how specific past events affect future intensities, allowing fine-grained analysis and uncertainty estimation.

[Regarding Complexity] The proposed approach seems more complex and might not be scalable relative to baselines, which is worsened by the use of normalizing flows [...].

The flow adds negligible cost (one millionth of a second on consumer GPU); most computation comes from the autoregressive transformer, similar to SAHP and NHP

Eqn 9: Could you clarify how the normalizing flow function is specified, e.g., what is different or shared across marker types and event times? Also what is f’?

The normalizing flow models 1D conditional distributions over continuous values, with shared structure over types. Just like categorical logits, it is parameterized based on the history through the transformer.

Eqn (9) is the standard change of variables equation underlying normalizing flows and autoregressive models in particular. Such a model learns an invertible function mapping $x$ to a standard normal in the latent space. The density $p(x)$ implied by the flow is proportional to the likelihood of $z=f(x)$ under the standard normal, and proportional to the volume change $f’(x)$, which in 1D is just the derivative of $df(x)/dx$ with respect to its input.

How does the model capture the influence of event interactions on event time?

The autoregressive transformer takes in all previous events (times, mixed-type marks) as a vector $H_i$ to predict the parameters of the next event time $p(t_{i}|H_i)$.

This is similar to Self-Attentive Hawkes Processes and Transformer Hawkes Processes, although they follow an intensity-based approach, which requires numeric integration at training and rejection sampling at inference time. Additionally, we outperform them on EasyTPP in Table 1. We model the next event time as a one-dimensional probability distribution in an intensity-free fashion, which avoids these issues.

Could you provide a complexity analysis in terms of the number of parameters, training/inference time, and scalability relative to baselines?

Our model is somewhat larger, but trains and runs at comparable speed to competitors.

In detail, let us report the geometric mean for the requested metrics:

• Parameter count: We use a mean of 325k parameters, which is about 3-10x larger competitors.
• Training speed: We achieve a training speed of around 2400 sequences of length 200 per second, over 2x faster than AttNHP and NHP. RMTPP, SAHP, THP and IFTPP are 2-5x faster than us.
• Generation speed: ~70 sequences of length 200 per second, with similar relation to competitors as training. We did not implement a KV-Cache, which speeds up sampling by a factor in sequence length.
• Scalability: Self-attention inherently has access to the entire event history in each prediction. This makes memory scale linearly $O(n)$ with sequence length (one would naively assume that memory use is quadratic, but the attention matrix never has to be realized in memory, e.g., using FlashAttention [3]). Notably, we can fit a sequence of over a million entries on a GPU with 12GB of VRAM for our largest model. Also, transformers with four orders of magnitude more parameters than our biggest model have been deployed on edge devices (e.g., Llama 3-8B on a Raspberry Pi at 500ms/token using quantization [4]).

References:

• [3] Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR 2024
• [4] Ardakani et al. LLMPi: Optimizing LLMs for High-Throughput on Raspberry Pi. CVPR Workshop Paper 2025

Could you clarify the empirical benefits of conditioning on longitudinal covariates, i.e., Equation 10 vs. Equation 12?

Eq. 10 and 12 are not directly comparable since they do not concern the same variables. To assess whether including additional longitudinal variables is useful, we need to look at Eq. (13). It tells us the likelihood of events (e.g. medical interventions for health data) depending on the knowledge of the covariates: $p(t_\text{procedure}, t_\text{procedure}| \text{past procedures}, \text{covariates})$.

Table 3 shows the NLL as in Eq. (13), e.g. the first row (lower is better, std < 0.002 in all cases):

The more longitudinal covariates we include, the better we can predict the next medical procedure.

The proposed approach seems to yield small gains relative to baselines (including FlexTPP vs. FlexTPP-C). The paper should also report error bars of the reported negative likelihood.

We report standard deviations in Appendix A, Tables 6, 9, 15, confirming significance.

For FlexTPP (our model, unconditional; Eq. 11) vs FlexTPP-C (our model, conditional; Eq. 12), the reviewer is correct that only 60% of the winning NLL is at least one standard deviation away from the other model.

Figure 3: The qualitative results in Figure 3 are difficult to follow; could you clarify?

Figure 3 shows a sequence from the time series annotation dataset, where the goal is to infer events affecting the input signal. The ground truth events (blue) consist of a frequency change by a factor of 1.14, and additional noise of 1.63.

The discrete-only model (orange) predicts event start times and types but cannot predict durations or continuous parameters. The depicted duration is chosen to fit the text. In contrast, our model (green) predicts start, end, and parameter values, accurately modeling the full event structure. To our knowledge, this is the first point process model applied to such a task.

Table 4: The paper should report the metrics on event type and event time predictions separately and provide non-likelihood-based metrics, such as RMSE, for event time predictions.

We agree that non-likelihood-based metrics can be helpful for interpreting the model performance. Thus, we reported AUC ROC in Table 4.

Here are the RMSE values for Table 4 (standard deviations in brackets):

While this clearly favors our method, we think that RMSE is not a good metric for this task. A model that samples events autoregressively can have a good RMSE but if one of the generations fails (e.g. because the model is tuned for RMSE via a regression loss), the rest of the sequence cannot be meaningful.

We therefore report AUC ROC in the paper since (a) they are based on a full generated sequence of events from each model and (b) are an accepted quantity in the time series annotation literature [5].

[5] Schmidl et al.: Anomaly detection in time series: a comprehensive evaluation. VLDB 2022.

We thank the reviewer for their helpful feedback!

This study demonstrates significant advancements in marked temporal point process modeling through two core innovations [hybrid-type processes and conditional information utilization]

Thanks for acknowledging these contributions. We would also like to point out these additional contributions of our work:

1. Intensity-free outperforms intensity-based. Previous work argued that intensity-based methods dominate due to "fewer modeling restrictions" [https://arxiv.org/pdf/2412.19634, Section 4]. Our approach brings intensity-free modeling back on the table, outperforming all intensity-based methods (except for DLHP in one dataset, see Table 1), all while avoiding numeric integration at training and rejection sampling at inference time.
2. Successful extraction of triggering kernels (new thanks to reviewer M7Vi). We are adding a successful evaluation on the interpretability of our model to the camera ready version. See our answer to M7Vi for details.

The combined computational load from Transformer and normalizing flow architectures increases parameter count by 5× compared to RNN-based models, potentially hindering edge device deployment.

Our model is several orders of magnitude smaller than transformers that have been deployed to edge devices. For example, recent work quantizes the weights of Llama 3-8B (8 billion parameters, which is four orders of magnitude larger than our largest model) to run it on a Raspberry Pi 5 at ~500ms/token [1].

The normalizing flow itself has at most 88(!) parameters predicted by the transformer, and likelihood computation/sampling takes around one millionth of a second on a consumer GPU for each entry. This is comparable to the compute required for a Hawkes process intensity evaluation.

[1] Ardakani et al. LLMPi: Optimizing LLMs for High-Throughput on Raspberry Pi. CVPR Workshop Paper 2025

Please clarify whether complexity calculations exclude attention weight matrix generation and provide quantitative GPU memory usage data, as the self-attention mechanism inherently requires $O(N^2)$.

Transformers are not memory-bound: This remarkable effect can be achieved by computing the attention matrix in chunks and never storing it in full in memory. Then, the memory consumption of a transformer is linear in sequence length $O(n)$. Computing the attention matrix as needed without ever realizing it in full is one of the tricks in modern attention implementations such as PyTorch’s scaled_dot_product_attention, e.g. based on FlashAttention [2].

Concretely, on a GPU with 12GB VRAM and our largest EasyTPP model, we can fit a maximum sequence length of two million tokens for a batch size of 1.

[2] Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR 2024

We hope that this elucidates the properties and contributions of our approach, and we are happy to address further questions in the upcoming discussion.

Hi Guanglin,

thanks! We are still pending an internal code review, unfortunately. Will keep you posted!

Best, Felix