A Simple Baseline for Predicting Future Events with Auto-Regressive Tabular Transformers

We thank you for your thorough review and address your questions below.

The primary reason for this being described as a "simple baseline" is due to its flexibility and lack of over engineering compared to prior work on event data. The flexibility of our method is described in 4.3: We show that the label column (i.e. the column that we are predicting) can be in any location in the table and similarly, we can change which column serves as the label (i.e. predicting time_to_event or is_fraud) at runtime without the need for any column specific finetuning While we agree that it would be possible to finetune an LLM to perform these tasks, in this work we are primarily demonstrating that you could use a lightweight, autoregressive model to do the same thing. In table 1 we show zero-shot performance using an LLM (Llama3-8B) to show that it is not sufficient to use an out-of-the-box LLM for event prediction tasks. Ultimately our primary contribution is this: There have been a limited number of attempts to use transformers for event prediction tasks, but all of them have used convoluted architectures, features, training regimes, etc. Additionally, all of them have used encoders to encode events before using a second model (often a decoder) to predict future events. In our work, we show that by using a simple decoder-only transformer and removing the special event prediction related features we are able to outperform current methods. Our work serves as a baseline in its flexibility across a number of prediction tasks and its simplicity by removing intricate architectural features.

In response to your questions about metrics and Table 1: Table 1 shows AUC scores and the errors were calculated by averaging over 5 random seeds. We have added information about both of these things to the caption for Table 1. Randomizing does decrease performance slightly, but does not dramatically increase the error rate (in some cases the error is actually smaller when randomizing).

In terms of decreased performance when masking labels, while it is true that STEP with masking underperforms baselines, it would be better to compare STEP with masking to baselines with masking.

Additionally, given that time-to-event is just another feature it can be predicted in the exact same way as the label feature. Similarly, it can be removed from the table at run time and STEP can make predictions without it (this is true with time-to-event as well as with any other feature). Additionally, we agree that there should be further research into the correct way to handle numeric values. The idea to quantize the numeric features was based on the decisions of prior work to do the same. Prior work primarily used 10 buckets, we decided to use 32 (although we tried 64 and saw similar performance). Ideally, numeric values would be embedded directly rather than bucketed but for the purpose of defining a baseline we felt that 32 buckets was sufficient given our comparison to prior work. We use exactly length 10 sequences as in prior work but we do not use the same train/test split (as those splits are not available). Additionally, in order to remove potentially getting a favorable train test split we use a method similar to 5-fold validation to calculate AUC scores.

Regarding your questions about hyperparmeter tuning, we did not hold out a portion of the dataset for validation but rather we choose hyperparameters by running a series of tests on different portions of the data and averaging performance. Specifically, we ran k-fold validation (with $k=5$) to determine the best hyperparameters before actually running all of our training runs. Similarly, we did not have a designated train/test split that we used for every run, but rather each run took a a unique split of the users into train/test sets to be trained on and then evaluated for that model. The performance was then averaged across 5 seeds to avoid any anomalies that might occur from any specific split of the users.

Regarding ESGPT as a potential baseline, while ESGPT is a valuable library, their work primarily focuses on data cleaning and flexibility of the MIMIC-IV dataset rather than focusing on a new method for handling this type of data. Additionally, while they provide some code to train a transformer on this specific domain they specifically consider encoder/decoder models rather than the decoder only models we consider. Further, they use intricate nested attention layers rather than our vanilla causal attention. Our work is differentiated in its simplicity.

Thank you for taking the time to review our paper. We have responded to you points below:

In response to your questions about our experimental details: Figure 4 shows one of the important variations of our method, namely that the model can handle the label in different positions and with different sequence lengths without additional finetuning. This type of flexibility is the primary contribution of our work.

Would you be willing to elaborate more on what you think the experimental section is lacking? In our work we have outlined that there are limited baselines for event prediction tasks and we aim to provide a framework that is more simple than the few existing baselines, but we do compare our method to theirs.

Additionally, we have updated the title of figure 1 for additional clarity.

We thank you for your response and address your questions below.

While prior work has primarily used encoder-type solutions, that does not mean this task requires the use of encoders. In fact, we believe that our primary contribution is showing that a decoder-only model outperforms the encoder-based baselines. Further, treating event prediction as a generative task gives the model the flexibility to not need to be finetunes on specific targets at runtime.

In response to Llama underperforming in the zero shot setting indicating that decoder-only LLM's arent suited for event prediction: performance is the main factor we consider in our comparison, the Llama column in Table 1 is zero-shot performance and is used as a baseline demonstrating that our method outperforms foundation models. Of course, you could finetune an LLM for each task/dataset, but a primary advantage of STEP is the lightweight nature of the method which includes a smaller model, much less training, and no further finetuning for each task on the given dataset. Baseline results are reported from prior work and we agree that baselines results are limited. We believe the lack of strong baselines is further proof that event prediction needs a more robust set of benchmarks and baselines.

In response to questions about our separate vocabularies, we implement the separate vocabularies using a single tokenizer object. We specifically used the term "vocabularies" instead of "tokenizers" to show that the vocabularies for each column are non overlapping. It would be possible to use separate tokenizers per column but we chose to implement it as you said by having a single tokenizer, where different ID's can be assigned to the same string values. The difference between these two things is simply an implementation detail. We have updated the paper to make this distinction more clear.

Additionally, a vocab size of 60,000 was chosen by observing performance on a few different vocabulary sizes. We did not observe a material difference in performance when attempting other vocab sizes. It does not mean that there are only 60,000 unique numbers as the numeric values are quantized into 32 different buckets (this process is described in section 4.2 where we discuss data preprocessing). The choice to bucket numeric values rather than using the direct values themselves is in line with prior methods (although we agree that this would be a potential future area of research).

In response to requesting additional clarify on experimental details: in section 3.3 we walk through the training setup. Our model is trained from scratch and is not a fine-tuned version of a pre-trained LLM. We have added additional information to this section to make it more clear.

Regarding your question about additional LLM's as baselines: We are specifically discussing the event prediction modality, which is a subset of tabular/structured data and has been relatively underexplored. The primary difference between event prediction and general tabular domains is the addition of a temporal feature and a meta-feature (where events are grouped by the meta-feature). We discuss this distinction in section 1, but have added more information there for clarity. While we are not able to make a fair comparison to the closed-source models you mentioned (as we have no way of viewing the output logits to evaluate their performance), we do include a comparison to Llama3 as a baseline.

Additionally, this method does not need to be fine-tuned for each task. Because it is generative in nature it learns the joint distribution across the entire dataset and can predict any label at evaluation time. That being said, our primary goal was not to create a foundation model for event prediction as this would require much more publically available data (we discuss in section 6 that one of the things that makes event prediction difficult is the lack of publicly available training data like there is in time-series or general tabular domains). Additionally, we include compute and training requirements in Appendix A.3. Training and model hyperparameters are included in Appendix A.4. Section 3.3 gives our methodology:

We train STEP in the exact same fashion as GPT-style decoder-only language models (Radford et al., 2019). We employ a standard next token prediction objective via cross-entropy loss and causal masking.