Generating Multi-Table Time Series EHR from Latent Space with Minimal Preprocessing

Response to W1-1:
Thanks for the opportunity to clarify our "minimal preprocessing" claim. Our intent is to highlight how RawMed significantly reduces reliance on lossy preprocessing steps that are prevalent in existing EHR synthesis methods.

To clarify what constitutes this "lossy preprocessing", we refer to operations that fundamentally distort or reduce the original data. This includes techniques like binning continuous values or aggregating data into fixed time intervals, which sacrifice the original data's granularity. Similarly, term normalization can eliminate subtle distinctions present in the raw information by grouping or standardizing terms. We agree that one-hot encoding is a lossless transformation and will clarify this distinction in our paper.

Crucially, the first major limitation we pinpoint in existing works is their reliance on feature selection. This is a prime example of extensive, lossy preprocessing. It involves domain experts manually choosing only a subset of features they deem relevant for specific downstream tasks, thereby discarding vast amounts of original, potentially valuable information from the outset.

The core issue with all these lossy transformations—be it feature selection, binning, aggregation, or certain normalizations—is the inevitable reduction in data utility. For instance, data aggregated to hourly "averages" cannot accurately answer queries requiring precise "counts" within that hour, nor can it support analyses demanding finer temporal granularity. When the diverse future uses of synthetic data are unknown, losing original information inherently limits its overall versatility and applicability. RawMed directly addresses this by avoiding such transformations, thus preserving the richness of the raw data for broader utility.

Response to W1-2:
RawMed's preprocessing is not extensive. Unlike other methods requiring complex, domain-specific feature engineering, RawMed unifies all data into a text format. It then applies standard, general-purpose steps like text embedding and tokenization. This significantly reduces the overall preprocessing burden and complexity, making our approach comparatively minimal.

Furthermore, RawMed's preprocessing is near-lossless. Tokenization and text embedding primarily change the data's representation, not its content. Any potential information loss can occur during the compression stage within our generative modeling part, not during the initial preprocessing. The encoder’s ability to preserve information is empirically validated by token-level reconstruction accuracy on the test set, reaching 98.2% on MIMIC-IV and 99.8% on eICU, which indicates minimal information loss.

We hope this response fully addresses your questions regarding our stance on preprocessing. We will also incorporate these clarifications and discussions into the introduction and Section 2.1 (Related Work) of our paper to provide a more comprehensive explanation.

Response to W2:
We appreciate your valid concern regarding the potential for auto-regressive models to under-represent low-probability events, and the lack of a direct metric for this in our current evaluation suite. However, our metrics indirectly address this point. Specifically, our CDE (Column-wise Density Estimation) metric, when applied to item-related columns (e.g., specific lab tests, medications), measures the marginal distribution similarity across all possible events within that column. Additionally, our I-CDE (Item-wise CDE) metric extends this by averaging CDE across various item types, encompassing relatively rare drug classes and lab test varieties. In both CDE and I-CDE, RawMed consistently demonstrated superior fidelity to the real data compared to other baseline models.

Response to Q1:
Our post-processing pipeline involves two main stages: converting text to tabular format and enhancing data quality.

In the "converting text to tabular format" stage, we observed the following event-level modification and rejection rates: 0.42% modified and 0.07% rejected in eICU, and 0.73% modified and 0.17% rejected in MIMIC-IV. At the patient level, rejection rates were notably higher: 2.71% in eICU and 18.23% in MIMIC-IV.

For the "enhancing data quality" stage, stricter quality checks led to higher modification and rejection rates at the event level. In eICU, approximately 1.39% of generated events were modified and 0.49% were rejected. For MIMIC-IV, 1.07% of events were modified and 0.79% were rejected. These stringent checks resulted in significantly elevated patient-level rejection rates: 16.90% in eICU and 54.53% in MIMIC-IV.

The significantly higher patient-level rejection rates, compared to event-level rejections across both stages, are a direct result of our stringent data integrity policy. A single erroneous event leads to the rejection of an entire patient's data. For instance, MIMIC-IV patients can have up to 243 events, all of which must be accurate to ensure the patient's data is retained within the dataset.

Response to Q2:
We appreciate you raising this point. We applied an equivalent level of post-processing to the RealTabFormer baseline in Table 5. However, extending this same strict post-processing to all baseline models, particularly those in Table 2, 3, and 4, turned out to be challenging and largely impractical. For example, applying our full post-processing pipeline to baseline models trained on MIMIC-IV resulted in approximately 99% of their generated data being discarded. This high rejection rate indicates that most baseline models couldn't perfectly generate all patient events under our strict post-processing standards.

Nevertheless, to ensure a fair comparison of model performance alone, we applied a less stringent post-processing to RawMed, resulting in an output quality comparable to the baselines in Tables 2, 3, and 4.

The resulting performance is detailed in Tables A, B and C, which can be directly compared with the results in Tables 2, 3, and 4 of our main paper. We could not include those results here due to space limitations.

Generally, the performance for most metrics remains similar to our prior results, where full post-processing was applied to RawMed. However, specifically for MIMIC-IV, our model's PCC and SMAPE values are slightly higher than our prior results and also higher than ClavaDDPM, indicating slightly worse performance in these metrics. Despite this, across all other metrics and consistently compared to other baselines, our model demonstrates superior performance.

However, we emphasize that full post-processing is crucial for generating high-integrity data with RawMed. We do not recommend using partial post-processing, as it fundamentally compromises the quality of the synthetic data.

Table A: Results of single-table evaluation on MIMIC-IV and eICU datasets.

Table B: Results of clinical utility evaluation on MIMIC-IV and eICU datasets

Table C: Results of temporal fidelity evaluation on MIMIC-IV and eICU datasets

Response to Q3:
Thank you for your question about how we chose the k-value in top-k sampling and your valuable point about the lack of analysis on its varying impact on generated data. You're absolutely right that the choice of k significantly influences the data's characteristics.

Initially, we conducted preliminary, small-scale experiments using the MIMIC-IV and eICU dataset, drawing inspiration from autoregressive sampling in NLP. We started with small k-values like 5, 10, and 15. However, we realized these values didn't accurately reflect the event distribution of the real data. This led us to intentionally increase k, arriving at our chosen values through a more heuristic approach rather than a full, systematic analysis at the start. While we didn't perform a complete, systematic analysis in the beginning, your insightful question prompted us to look closer at neighboring k-values.

Due to time constraints, we've focused on two immediate surrounding values, but these preliminary results already highlight the impact on performance: For the eICU dataset, a k-value of 150 yielded the highest TSTR AUROC at 0.751, significantly outperforming k=100 (0.728) and k=200 (0.723). Similarly, for the MIMIC-IV dataset, a k-value of 250 demonstrated the best performance with a TSTR AUROC of 0.797, notably better than k=200 (0.773) and k=300 (0.776). We acknowledge the critical role of k and are committed to providing a more comprehensive analysis with a broader grid search during the discussion period, continually updating our findings to further solidify justification for these choices.

We sincerely appreciate your insightful feedback and constructive comments, which are invaluable in improving our work.

Response to Weakness1 & Question2:
You've rightly pointed out the importance of conditional generation for practical applications. We agree entirely. Our initial focus was to establish the model's fundamental generation capabilities. However, we've conducted preliminary experiments on conditional generation incorporating static features during this rebuttal period.

Specifically, we integrated age, gender, and admission diagnosis into the Temporal Modeling (Section 3.3). For training, we prepended these static features to the sequences, and the model learned the combined sequence auto-regressively. During generation, the model uses real static features as a condition to produce the complete sequence. To further explore conditional generation, we also experimented with providing initial medical context as a condition. This involved providing the model with real static features and approximately one-fourth of the initial real-data events from the sequence, with the model then generating the remaining three-fourths of the sequence.

To evaluate if the generated sequences effectively captured the relationship with static features, we used three classifiers to predict each static feature (age, gender, admission diagnosis) from the generated sequences, excluding the static feature portion itself. Age was binned for this classification. Model performance was evaluated using micro-AUROC for multiclass classifications (age and admission diagnosis) and AUROC for binary classification (gender).

These results show comparable performance for age and admission diagnoses between real and synthetic data, especially when leveraging the initial medical context. The increased AUROC values when conditioning on a portion of initial events (e.g., 0.69, 0.59, 0.88 for age, gender, and admission diagnosis, respectively, compared to 0.65, 0.52, 0.86 with static features alone) experimentally confirm our framework's flexibility in effectively supporting conditional generation using a richer set of features, leading to improved synthetic data quality. While we observed a slight drop in gender prediction compared to real data, we believe there's room for improvement with further methodological enhancements and hyperparameter tuning, given the short experimental timeframe. Due to these constraints, experiments were limited to the eICU dataset; we will update with MIMIC-IV results (for age, gender, and admission type) during the discussion period.

Regarding the extension to a hybrid generation mode with user preferences, this is indeed a very interesting direction. It would be feasible to fine-tune RawMed using techniques such as Direct Preference Optimization (DPO) [1] with a separate dataset that contains both "preferred synthetic data" and "un-preferred synthetic data." This aligns with our future research interests in developing more controllable and customizable synthetic data generation.

Response to Weakness2:
This process is not heuristic but rather a rule-based approach that adheres to the minimum requirements of time-series relational tables, including consistent table naming, column naming, preservation of column-value pairs, and adherence to the temporal ordering of events. By predefining the table names, column names, and data types (e.g., numeric or categorical) for EHR datasets, Algorithm1 can be applied to diverse datasets without additional tuning or modifications. Indeed, the same code was successfully applied to the MIMIC-IV, eICU, and MIMIC-IV-ED datasets.

Response to Question1:
The errors observed during the conversion of text to tabular format, as detailed in Algorithm 1, are primarily syntactic inconsistencies. These include incorrect table names, non-existent column-value pairs, mistyped column names, and mismatched data types. For example, a column that should be named "value" might appear as "value value", or "originalrate" might be truncated to "original." Additionally, a numeric column might contain non-numeric values like "1..23" or "8. 5. 6." instead of valid numerical entries. We posit that these errors originate from the auto-regressive modeling process within the compressed text space. These syntactic inconsistencies are not unique to RawMed. Other text-based autoregressive models for tabular data encounter similar errors. For example, the GReaT paper also discards syntactically invalid samples, highlighting a shared challenge in this domain.

Response to Question3:
When compressing raw EHR data with Residual Quantization (RQ), our method provides acceptable overall reconstruction accuracy on the test set. We achieved 99.8% token-level accuracy on eICU and 98.2% on MIMIC-IV, indicating that the data's general structure and core information are well-preserved across both datasets. Specifically, for eICU, table name reconstruction accuracy was 99.9%, column names 99.9%, categorical column values 99.8%, and numerical column values an estimated 99.4%. For MIMIC-IV, table name reconstruction accuracy was 99.9%, column names 99.9%, categorical column values 99.8%, and numerical column values an estimated 91.2%. While the reconstructed data might not perfectly match the original, the preserved information typically remains adequate for most data utilization purposes.

Reference

[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al.)

Response to Weakness1:
We appreciate the reviewer’s concern regarding the evaluation of RawMed on only two datasets (MIMIC-IV and eICU) and the suggestion to include more diverse datasets to assess generalizability. However, as noted in Section 4.1 (Setup: Dataset, lines 201–203 in the manuscript), RawMed was also evaluated on the MIMIC-IV-ED dataset, which represents emergency department settings and provides a distinct clinical context compared to the ICU-based MIMIC-IV and eICU datasets. The results, reported in Appendix F.3 and Table 16, show that RawMed’s performance closely matches real data utility and outperforms baseline methods, indicating its potential for generalization to other datasets despite different clinical environments.

Response to Weakness2:
Thank you for your insightful comments. To empirically quantify the deviation from original information that might occur during the compression of event embeddings, we measure the token-level reconstruction accuracy on the test set, reaching 98.2% for MIMIC-IV and 99.8% for eICU. This confirms our compression introduces only negligible distortion, barely affecting the subsequent autoregressive decoding stage, while considering the benefits of compression for text-based embeddings.

As detailed in 3.2 and 3.3, our method compresses individual event representations, not the entire time-series representation. Therefore, our model does not involve decoding a compressed time-series back to its original form. Instead, we focus on reconstructing each individual event representation.

Regarding your last question about querying patient height, we would greatly appreciate it if you could elaborate on your question. Our model is designed for synthetic EHR generation, not for question-answering tasks, and is not required to address such direct inquiries.

Thank you for your valuable review. We wanted to respectfully follow up on our rebuttal. As the discussion period is nearing its end, we would appreciate it if you could let us know if our responses have addressed your concerns. If there are any points that still need clarification, we would be glad to provide further details.

Response to Weakness1&2:

We sincerely appreciate your insightful feedback particularly in highlighting the current limitations regarding the scope of tables and the handling of static patient attributes. As we explicitly discussed in the Section 6 of our paper, these are the current limitations of RawMed. In this initial stage of our research, our primary focus was to establish the model's fundamental generation capabilities. We acknowledge these as key areas for future work, aiming to scale up to dozens of tables, including those with static features.

During the rebuttal period, we conducted preliminary experiments on conditional generation incorporating static features. Specifically, we integrated age, gender, and admission diagnosis into the Temporal Modeling (Section 3.3).

For training, we prepended these static features to the sequences, and the model learned the combined sequence auto-regressively. During generation, the model uses real static features as a condition to produce the complete sequence.

To further explore conditional generation, we also experimented with providing initial medical context as a condition. This involved providing the model with real static features and approximately one-fourth of the initial real-data events from the sequence, with the model then generating the remaining three-fourths of the sequence.

To evaluate if the generated sequences effectively captured the relationship with static features, we used three classifiers to predict each static feature (age, gender, admission diagnosis) from the generated sequences, excluding the static feature portion itself. Age was binned for this classification. Model performance was evaluated using micro-AUROC for multiclass classifications (age and admission diagnosis) and AUROC for binary classification (gender).

These results show comparable performance for age and admission diagnoses between real and synthetic data, especially when leveraging the initial medical context. The increased AUROC values when conditioning on a portion of initial events (e.g., 0.69, 0.59, 0.88 for age, gender, and admission diagnosis, respectively, compared to 0.65, 0.52, 0.86 with static features alone) experimentally confirm our framework's flexibility in effectively supporting conditional generation using a richer set of features, leading to improved synthetic data quality. While we observed a slight drop in gender prediction compared to real data, we believe there's room for improvement with further methodological enhancements and hyperparameter tuning, given the short experimental timeframe. Due to these constraints, experiments were limited to the eICU dataset; we will update with MIMIC-IV results (for age, gender, and admission type) during the discussion period.

Response to Weakness3:
The need for post-processing steps to ensure the structural and temporal integrity of generated data is a valid and widely accepted practice for quality assurance across various domains. These steps are (1) inevitable, as an autoregressive generation—predicting tokens sequentially—inherently accumulates errors over time, making it challenging to produce a 100% flawless sample in a single pass. For example, the text-based tabular synthesis model GReaT [1], which relies on autoregressive generation, discards samples that violate required formats. Moreover, they are (2) practical, as even renowned generative models in the image domain, such as VQGAN [2] and VQ‑VAE2 [3],use classifier-based rejection sampling to filter out low-quality samples and improve overall sample quality.

[1] Language Models are Realistic Tabular Data Generators (Borisov et al., ICLR 2023)
[2] Taming Transformers for High-Resolution Image Synthesis (Esser et al., CVPR 2021)
[3] Generating Diverse High-Fidelity Images with VQ-VAE-2 (Razavi et al., NeurIPS 2019)

We appreciate your valuable feedback. We'd like to clarify our post-processing approach, as there may have been a misunderstanding. Our method does not use rejection sampling to select superior samples from a pool of generated candidates. Instead, our post-processing is focused on guaranteeing the required data integrity for real-world applications.

To be specific, our post-processing serves two distinct purposes. First, it is a necessary step to convert the model's generated text into a tabular format. Second, it ensures the structural and temporal integrity of the data. For example, any sample that violates a specified min-max range for a column is discarded. We hope this clarification addresses your concerns and provides a clearer understanding of our methodology.