CHIRon: A Generative Foundation Model for Structured Sequential Medical Data

Lack of Novelty: The approach lacks clear originality, as autoregressive models for medical data, especially those incorporating procedure codes, medications, lab results, and demographics, are well-explored. Several models, such as CLMBR, MOTOR, EventStreamGPT, and ETHOS, also use decoder-only architectures with causal or autoregressive objectives. The one unique aspect in CHIRon is the addition of pretraining tasks (predicting place of service and age at encounter), but even these are present as direct features in models like CLMBR.

Limited Baseline Comparisons: The study does not include critical autoregressive baselines, such as CLMBR, which shares similarities in design as a decoder-only autoregressive model. This makes it challenging to assess the comparative performance of CHIRon accurately.

Questionable Utility of Additional Pretraining Tasks: While the model adds a pretraining task for predicting the place of service, the benefit of this task remains unclear, as embeddings for the place of service are already included. The redundancy and utility of this pretraining task are not well-demonstrated, especially given that other models like CLMBR and EventStreamGPT include similar objectives as part of their standard autoregressive prediction tasks.

Evaluation Limitations with Standard NLP Metrics: Using standard NLP metrics may not accurately capture the performance of medical code generation, as minor variations in medical codes can significantly change diagnostic meaning. Including qualitative results, such as expert-verified generated samples and gold-standard examples, would improve evaluation quality. Additionally, predictive metrics over long-term simulated trajectories (e.g., AUROC for one-year risk prediction) would better capture the model’s generative capabilities and clinical relevance. A generative model could be used to answer long term predictive problems by simulating a year of future data and seeing if a code appears in those simulated histories. One can run multiple trajectories to get a probability.

Suboptimal Empirical Results for Novel Contributions: The causal visit-based masking approach, claimed as a novel contribution, shows suboptimal performance in comparison to simple next-code prediction, as shown in Figure 4. This weakens the empirical support for the proposed masking method.

Potential Bias in Index Time Selection: It is not clear if the same index time selection procedure is used for both cases and controls, which can induce bias, as models might learn artifacts from the index time selection process rather than actual clinical features. Resources like (https://ohdsi.github.io/TheBookOfOhdsi/PatientLevelPrediction.html) provide guidance on ensuring consistent index time procedures for cases and controls. It would be beneficial for the authors to clarify the index time selection process for controls and address any potential biases introduced by differing procedures.

Lack of Transparency and Reproducibility: The model is trained and evaluated on a private dataset, and the code is not provided. Given the reliance on proprietary data, releasing the codebase would be essential for validation and further use of the model in research.

1. The paper has limited technology contributions to the computer science audience. The main contribution is about the numerical lab results tokenization. However, as a foundation model work, the generalizability of the method is unknown on other datasets, making it difficult to evaulate the methodology novelty
2. The main limitation lies in the generalizabiility of the model. The baseline models and evaluation tasks are limited compared to recent medical (EHR) foundation model works [1,2]. They are evaluated using tens of tasks or few-shot cross-dataset tasks. While the authors argue the reason not using MIMIC datasets, only predicting on five diseases is not convincing to show the generalizability of the model, which is crucial for foundation model works. Also the author fail to clarify why these five disesaes are selected.
3. The baseline models are also limited. The experiments should including recent foundation model works, and as well as traditional models such as LSTM and GRU to outline why using GPT-based architecture is preferred. [1] Steinberg, E., Fries, J. A., Xu, Y., & Shah, N. MOTOR: A Time-to-Event Foundation Model For Structured Medical Records. In The Twelfth International Conference on Learning Representations. [2] Wornow, M., Thapa, R., Steinberg, E., Fries, J., & Shah, N. (2023). Ehrshot: An ehr benchmark for few-shot evaluation of foundation models. Advances in Neural Information Processing Systems, 36, 67125-67137.

1. Limited Technical Innovation: The architecture largely leverages existing approaches, using GPT-2 as its foundation The tokenization strategy is based on Med-BERT because Med-BERT excludes lab results,The approach to incorporate lab results seems to be adapted from Golkar et al.'s work.
2. Potentially Misleading Claims: The abstract's characterization of a "new pre-training objective function" needs clarification What's presented as a novel objective function is essentially still a next-code prediction task? The main difference lies in preprocessing modifications rather than fundamental changes to the objective function itself.
3. Incomplete Evaluation: EHRSHOT is a closely related work and necessary to compare. More importantly, since authors consider CHIRon a foundation model, it's necessary to shows its adaptation capability. Lack of transfer learning experiments in the paper.
4. Reproducibility question: Absence of publicly available code

1. Missing Zero-shot classification Evaluation

Why did you decide to not investigate zero-shot binary classification performance in this work? This is one of the major applications in the field?

Several key related works provide frameworks or results evaluating autoregressive generative models on clinical data, and these works diminish the novelty of this paper:

1. ESGPT [1] is an autoregressively pretrained decoder-only transformer model that in the paper integrates age in tokens, and through it's user api can support user defined time specific features such as the place of service. The API allows zero-shot binary classification as well, something never touched upon in this work.

2. Ethos [2] is another model that performs decile binning and evaluates binary classification performance in the zero-shot setting. They actually run this on MIMICIV. It leaves out comparisons to baselines like XGBoost baselines, or supervised models though which you could improve upon.

3. Foresight [3] is another one of these autoregressive pretrained EHR data models you should cite.

4. Incomplete Analysis

• Generated Data Evaluation--tokenization method analysis: Why is there no comparison of the tokenization methods in the generated data evaluation setting?
• Custom percentile binning is not systematically assessed: I expected there to be a systematic assessment of well defined binning strategies for numeric values and specific takeaways on this, but there it no clear analysis/takeaway. Additionally I expected a comparison of no binning and encoding using only the continuous value of the lab, but also didn't find this.
• Sorting data - This paper seems to position itself as providing controlled experiments between tokenization methods and autoregressive pretraining tasks, developing an improved method from past works. One major limitation is that you always use random ordering of codes within visits. The affect of sorting codes within visits will have a significant affect on modeling. Next code prediction within a visit is significantly easier given a consistent ordering. This is precisely what ETHOS [1] does.

3. MIMICIV Experiments Should be Run

The authors do not run MIMIC IV experiments stating

While we agree MIMIC-IV is a well known public resource for high quality patient data from ICU stays, both CHIRon and related foundational models focus on more general longitudinal healthcare data,not specific to ICU stays.

In fact MIMIC-IV v3.0 includes

364,627 patients, 546,028 inpatient admissions, and 94,458 icustays. MIMIC-IV is generally a gold standard dataset for communicated ML results on in the ML literature, so the reasoning for not including it here is lackluster. The authors do use a much larger private dataset on the order of 25 million patients and do significant processing of this dataset, maybe a communication (in the limitations section) of the limitations and requirements of the extensive data preprocessing for your model and evaluation and how they make running experiments on MIMIC-IV out of scope for this paper is necessary.

4. Visit-based causal masking performing worse in the finetuning explanation is not a novel or interesting result.

Having access to previously generated tokens is not data leakage. An autoregressive generative model always is conditioned on past generated codes, so Visit-based causal modeling (where you mask out previous generated codes in the same visit) is obviously going to hurt model performance because model has to unconditionally generate codes without knowing which ones it already generated for the visit.

5. Insufficient Evaluation metrics for EHR data

The evaluation metrics used only assess similar marginal distributions of codes, which implies that ROUGE-1 is really not a clinically appropriate metric for assessing EHR generated data quality since it ignores sequential order. While BERTScore does implicitly capture sequential ordering in the contextual embeddings there should be additional metrics that explicitly evaluate sequential order in the medical context, such as metrics that consider the temporal relationships between generated medical events in different visits reflecting the real ordering across visits.

Presentation issues

Please make the legend in Figure 3 clearer, and make concise names for the tokenization methods, that you can use in the legend. You should clarify what the ptile binning strategy is. I had to check the appendix and the intro to understand that you did some custom percentile binning, which is great, but should be clearer in the main text and you should give a systematic discussion of that these percentil binning strategies are and why they were chosen (per weakness 5).

[1] McDermott, Matthew, et al. "Event Stream GPT: a data pre-processing and modeling library for generative, pre-trained transformers over continuous-time sequences of complex events." Advances in Neural Information Processing Systems 36 (2023): 24322-24334.

[2] Renc, Pawel, et al. "Zero shot health trajectory prediction using transformer." NPJ Digital Medicine 7.1 (2024): 256.

[3] Kraljevic, Zeljko, et al. "Foresight—a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study." The Lancet Digital Health 6.4 (2024): e281-e290.