PEMs: Pre-trained Epidemic Time-Series Models

While the paper compares PEM with FUNNEL, a mechanistic model, it does not explore simpler disease-focused mechanistic models or even basic auto-regressive models....COVID-19, limits the model's generalization claims

We do compare against state-of-art epidemic forecasting models that have shown to beat past simpler statistical models on epidemic forecasting tasks and have won past epidemic forecasting competition[1,2,3]. Therefore, we believe this is a fair comparison to provide evidence that PEM indeed provides state-of-art forecasting performance.

The paper already refers to a paper on one of the hubs (Cramer 2022). Without comparison against the models used by the community, it is hard to judge how much value the proposed approach adds.

There are two main reasons why direct comparisons with the models used by forecast hub is not feasible. First, our model only uses past values of the time-series to forecast future dynamics. It does not ingest additional external variables. Therefore, all the benchmarks are univariate time-series forecasting or prediction tasks and we evaluate state-of-art baselines designed for such tasks. Further, most models in the forecasting hubs used additional data sources and the models and datasets used are not public in most cases.

The data that is available today is a much cleaner version compared to what was available at the time a forecast was made. Especially for COVID-19 the data revision history is available, therefore, the authors should test the dataset with the appropriate version, not the latest version. It is not clear if the authors have done so.

All the baselines and the model are evaluated on the latest available version of Covid-19 mortality data for a fair comparison. We agree with the reviewer that data revision is an important problem in epidemic forecasting. Our work doesn't address this problem and instead focuses on modeling and accurately forecasting epidemic time-series. However, analyzing and tackling the data revision problem for our pre-training framework is an interesting future direction.

The paper lacks information on the data preprocessing...

We do not perform any data pre-processing on raw time-series data provided. Since we used Reversible Instance Normalization as part of the model, we do not need explicit data normalization before feeding into the model.

Performance compared to simple transfer learning or doing some training on a single disease dataset (such as pre-training on just past flu data if the disease of interest is flu) would have been good.

Randmask does not perform nearly as well as the others, so does adding this into the combination of tasks contribute meaningfully?

Yes RANDMASK alone is the worse performing SSL task overall. However, we still included it for pre-training because (a) it still outperforms the variant without any pre-training, (b) and using all four SSL tasks together provided the best performance across all benchmarks.

Can the authors verify that the hyperparameter selection was done on a validation set and not on the test set?

Yes, the hyperparameter selection was done during training on validation data for all benchmarks.

Lower RMSE Values for Flu in the US

The influenza incidence time-series extracted from CDC is normalized between 0 to around 10. We direcltly use these values for training and prediction. Therefore, the RMSE values may appear very low compared to other benchmarks.

References

[1] Harshavardhan Kamarthi, Lingkai Kong, Alexander Rodríguez, Chao Zhang, and B Aditya Prakash. When in doubt: Neural non-parametric uncertainty quantification for epidemic forecasting. Advances in Neural Information Processing Systems

[2] Bijaya Adhikari, Xinfeng Xu, Naren Ramakrishnan, and B Aditya Prakash. Epideep: Exploiting embeddings for epidemic forecasting. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining

[3] Logan C Brooks, David C Farrow, Sangwon Hyun, Ryan J Tibshirani, and Roni Rosenfeld. Flexible modeling of epidemics with an empirical bayes framework. PLoS computational biology,

why do you achieve so much better results than PatchTST? Is it just the finetuning?

It is due to our SSL pre-training. Indeed performing fine-tuning on heterogenous time-series data by designing specific SSL tasks for extracting useful background knowledge from wide range of disease datasets during pre-training is our most important technical contribution. We note that most other SSL pre-training methods of past works are not effective on our heterogenous pre-training problem. Therefore, our method of effective fine-tuning enables the pre-trained models to significantly outperform models without pre-training (such as using PatchTST without pre-training) as well as methods that use previous SSL methods.

I'm not sure I understand why inputting each time step as a single token would lack "semantic meaning", seeing as the output representations of a transformer for a given token are influenced by all the other tokens in the input sequence (i.e. "contextual embeddings")

Providing semantically meaningful input tokens can better enable transformers to model relations between tokens. For example, in the context of language, LLMs use specific tokenization methods to capture semantic meaning instead of individual characters to provide significantly better performance. Further, the inputs of time-series are real-valued numbers rather than specific character. Therefore, these individual values of time-series observed in isolation do not provide sufficient context such as local patterns (such as peaks, spikes, etc.) around the time-step. therefore, we use segments of time-series that provide semantic information about local patterns.

How do you normalize the training process of your method wrt baselines? How can you disentangle the architecture's influence from the influence of the training process and the influence of using additional compute?

For all deep learning baselines and PEM, we train using early stopping. We use the same compute hardware for training all models and baselines (appendix B) and we measure the training time till convergence for all models. PEM uses similar training time to other best baselines with similar compute and provide significantly better performance. In fact, it takes 12-50% less time to match the performance of the baselines (Appendix Table 5).

We also measure the importance of various training choices and pre-training choices in Table 3 and show the importance of each SSL task and linear-probing for optimal performance. We also measure the effect of segmentation in Appendix Table 9.

unclear what is being evaluated in appendix D table 6 since it's not known how the baselines are trained. For example, it is clearly remarked that no pre-training is performed for the PT baseline, but not why

Appendix D Table 6 measures the performance of PEM when we remove the training data for downstream tasks from pre-training. This is denoted by PEM-ExcludeTrain. This measures the performance of fine-tuning PEM to unseen diseases. We observe that even is such cases PEM outperforms the best baseline performance.

why are the results around one of the contributions (Significant improvement in data and training efficiency and adaptability to novel epidemics) only in the appendix?

We summarize the main results of training efficiency and data efficiency in the main paper page 8 and provide additional details in the Appendix: PEM takes 47-88% of the time taken by best baselines and uses 60-80% of total training data to outperform them. We could summarize the training times in Table 4 using the bar graph and add to the main paper.

There might be bleed between the influenza data from 2001 to 2010 and the crypto data from 2006 to 2012 given the temporal alignment

This is an interesting question. However, influenza and crypto show very different disease dynamics. Influenza peaks during winter months and spreads through the air. Crypto mostly occurs during the summer and is spread through food and water contamination. While influenza is spread through a virus, crypto is spread through a microscopic parasite. We couldn't find any research linking the incidence of both diseases

How do you choose the value of T, i.e. the length of the input time series?

All the diseases in training and pre-training have a sampling rate of 1 month. Moreover, even seasonal diseases have periodicity within 12 months. Therefore, we choose the total input length to be $T=12$. We will add this information in the Appendix.

How do you justify the default choice of gamma = 0.1 for lastmask ? Seems like 0.2 performs well too.

Yes both 0.2 and 0.1 performs well depends on the downstream tasks. However, even in cases 0.2 performs better, 0.1 is only slightly worse. Therefore, we chose gamma as 0.1.

The baselines in this paper do not represent the SOTA performance.

The chosen baselines are state-of-art general forecasting and epidemic forecasting models at the time of submission. Therefore, we are indeed comparing PEM with the previous best possible performance.

The baselines used have code released publicly. We used the same code with mostly default hyperparameters for evaluation with hyperparameter tuning on learning rate. We can add links to the baseline code in paper to alleviate these concerns.

There is no uncertainty quantification analysis of all the methods. It seems the transformer-based models are so large that they may have large predicting variance

Our method and most other baselines only focuses on forecasting and prediction accuracy and does not provide reliable uncertainty estimates. However, we agree with the reviewer that uncertainty quantification is an important forecasting problem, and extending our methods to produce calibrated probabilistic forecasting is an interesting research direction.

Clarification on SEASONDETECT

We divide the 12 months into four seasons: Dec-Feb, Mar-May, June-Aug and Sept-Nov. For each disease dataset $d$, we assign one of these four seasons as peak season $s_1(d)$. For example, in the case of influenza $s_1(d) =$ Sept-Nov. Based on assignment on $s_1(d)$ we assign $s_2(d), s_3(d), s_4(d)$ chronologically. In influenza's case, $s_2(d)=$ Dec-Feb, $s_3(d)=$ Mar-May, $s_4(d)=$ June-Aug.

Data normalization, will the normalized data be reversed before computing the loss error during training?

Yes. Using Reversible Normalization prevents large gradients during training.

Why perform linear-probing before fine-tuning?

Kumar et. al [1] showed that doing direct fine-tuning leads to lower accuracy as it distorts the learned representations of pre-trained models, especially for downstream tasks with distribution shift, similar to our case where we fine-tune on heterogeneous and unseen disease datasets. Linear-probing alleviates this by adapting the final layer of the pre-trained model to downstream tasks without distorting the representation learning of the pre-trained model.

The related work should be expanded.

We have described recent state-of-art SSL methods and used them as baselines. While there are many methods that perform SSL on time-series, we focus on methods that are applicable to any general time-series domain with specific applications to forecasting tasks. We would gladly add additional references to SSL methods including recent methods described in the survey referred to by the reviewer.

The technical novelty against previous SSL approaches is somewhat limited.

Unlike most previous SSL papers, we are the first to deal with a different broader problem of effectively leveraging multiple heterogeneous epidemic datasets to train a general pre-train model that can be fine-tuned to a wide range of downstream epidemic analysis tasks.

RANDMASK and LASTMASK are indeed widely used pre-training tasks in language domains and are also explored in time-series domain previously. However, to better extract useful epidemic dynamic information, we also introduce two novel tasks: PEAKMASK and SEASONDETECT. PEAKMASK helps identify peaks and their dynamics which is vital for epidemic forecasting since it is important to model and predict epidemic peaks in advance. Detecting seasonal information SEASONDETECT automatically is also important in successfully modeling epidemic dynamics. These tasks are carefully designed to impart useful background knowledge and patterns from multiple epidemic datasets.

To further illustrate the importance of designing and choosing the right SSL tasks, we also compared PEM against sophisticated SOTA self-supervised methods for general time-series and show that using generic tasks that perform well in other domains to epidemic datasets leads to poor performance.

The baselines listed on Page 7 are insufficient.

We note that even most recent time-series SSL methods do not deal with multiple heterogenous datasets. The consisdered SSL baselines (TS2Vec, TNC, TS-TCC) in the paper which provide top performance for specific benchmarks where training data is used for SSL pre-training perform much worse than even some baselines which do not use any pre-training in our setting of pre-training on multiple datasets. This shows that even sophisticated general SSL methods cannot simply used to pre-train on multiple datasets.

authors provide the source code for their approach.

We have added link to the implementation code in the paper at page 7: link. We also plan on releasing the weights of pre-trained models on acceptance.

more results on few-shot or zero-shot learning for epidemic data..

Our work focuses on the impact of pre-training model weights for more performant training of downstream tasks. Additionally, we evaluated the efficacy of fine-tuning from smaller dataset sizes in page 8 (Q4) and Appendix Figure 4. Specifically we measured the performance of PEM with different fractions of training data and found that PEMs in on par or outperform the best baselines using 60-80% of the training data in benchmarks.

If the reviewer is interested in any specific analysis on few-shot learning, we would be glad to add it to the paper.