Detecting Data Deviations in Electronic Health Records

W1. We respectfully disagree with the concern regarding the relevance and generalizability of our perturbation strategy. In clinical practice, point-of-entry validation for EHR data is essential due to the risk of serious downstream consequences from erroneous inputs. However, existing validation practices rely largely on coarse plausibility checks—such as broad value ranges or thresholds based on standard deviations—which are insufficient for detecting subtle yet clinically significant deviations (see also our response to Reviewer opND's comment W1).

Our strategy, which perturbs feature values by their standard deviation $\sigma$, aligns with established clinical quality control practices and offers a clinically interpretable way to model abnormal laboratory readings [c, d, e]. Rather than an arbitrary choice of corruptions, what our work contributes is a principled, sensitive framework that addresses the limitations of current checks by enabling pre-hoc detection and alerting of meaningful deviations prior to downstream contamination.

This design reflects realistic data fidelity issues observed in operational EHR workflows. Our clinicians have confirmed that our experimental setup is not only meaningful but also directly aligned with the operational needs and safety requirements of healthcare data entry. We will revise the paper to articulate this clinical grounding and to discuss the broader applicability of our approach beyond the specific perturbations evaluated.

W2. We are not entirely certain if we have interpreted this comment correctly and would appreciate any clarification if our current understanding is inaccurate.

We would like to first clarify a potential misunderstanding on the experimental evaluation. Our paper does include the suggested “real vs corrupted” classification experiment to distinguish actual data samples from perturbed ones. In our controlled deviation injection setup, we (i) perturb data with the feature's standard deviation $\sigma$, (ii) generate paired samples—original (label = 0) and perturbed (label = 1), and (iii) construct a labeled benchmark dataset for fidelity evaluation. We then compute the expected decline in EHR data fidelity $\Delta \Psi$ and use its sign to indicate whether a deviation is detected.

As shown in Figure 3, our approach outperforms anomaly detection baselines, particularly in detecting small-magnitude deviations. We have also compared against domain-knowledge-based rules and presented a detailed ablation study assessing different constituent components. Please refer to our response to Reviewer opND’s comment W1 for related discussion. We remain open to additional suggestions on methods relevant to our setting and will incorporate such comparisons where feasible.

W3. We will revise the paper to introduce our core idea using clearer and more accessible descriptions before presenting the detailed design and terminology. We believe this restructuring will improve clarity and make our contributions more approachable to a broader audience.

W4. Our approach outputs a predicted EHR data fidelity score $\Psi$ for each data sample. To detect deviations, we compute the expected EHR data fidelity decline $\Delta \Psi$ and use its sign as an indicator.

In the controlled deviation injection experiment (Figure 3), we simulate physiologically plausible deviations by perturbing features with multiples of their standard deviation $\sigma$, forming paired samples ⟨original (label = 0), perturbed (label = 1)⟩. We apply our approach to compute $\Delta \Psi$ for each pair and use the sign to identify deviations. Results demonstrate the strong detection performance of our approach compared to baselines.

In the output sensitivity experiment (Figure 4), we use the same paired samples as input with varying magnitudes of the injected deviations. The resulting $\Delta \Psi$ values demonstrate that our approach reliably captures subtle deviations, underscoring its practical utility in early-stage deviation detection for clinical applications.

We acknowledge that the rationale and structure of these experiments may not be immediately clear on a first reading. To address this, we will revise the paper to introduce the experimental setups and key design choices earlier, improving clarity and readability.

Q1. We follow the established MIMIC-III benchmark for data processing, aggregating 17 physiological variables across seven predefined time ranges: the first and last 10%, 25%, 50% of the ICU stay, and the entire duration. For each variable within each range, six statistics are computed—minimum, maximum, mean, standard deviation, skewness, and count—yielding 714 features per sample. This preprocessing strategy has demonstrated strong performance on benchmark MIMIC-III tasks in prior work and is adopted here to ensure consistency. As our focus is on pre-hoc detection of data deviations at the point of entry rather than optimizing preprocessing, we do not explore alternative preprocessing strategies.

From a practical standpoint, incorporating more advanced time series models would exacerbate the already significant computational cost of estimating data Shapley values. Exact data Shapley computation is intractable for large-scale EHR data due to its exponential complexity [14]. Even with Monte Carlo approximation, the process requires repeated model retraining for each sample across multiple permutations. To address this, we adopt an amortized modeling approach, training a neural network to approximate per-sample data Shapley values [1, 6]. While this strategy reduces inference-time cost, it still requires computing ground-truth data Shapley values during training. Given these constraints, more complex time series models are currently infeasible. Improving the efficiency of Shapley-based data valuation continues to be an important and active area of research [46].

Q2. Our methodology is designed in a top-down, stepwise manner, guided by the central goal of detecting EHR data deviations. Below, we clarify the rationale behind each component.

To begin, detecting deviations requires a principled measure of data fidelity. Data valuation methods—particularly data Shapley values—are increasingly used to quantify sample-level contributions to learning models [14, 25, 46, 54, 57, a, b], making them a natural proxy for data fidelity. However, data Shapley values are inherently task-specific and thus insufficient for capturing a more comprehensive notion of fidelity across a patient’s EHR data.

To overcome this, we adopt a multi-task perspective that distills task-specific data Shapley values into a unified EHR data fidelity predictor. This integration enables a more comprehensive assessment by incorporating insights across multiple clinical tasks.

In summary, we propose two levels of knowledge distillation: (i) from computationally intensive data Shapley values to task-specific neural oracles, to address efficiency concerns (see Sections 3.1 and 3.2), and (ii) from the task-specific neural oracles to a unified EHR data fidelity predictor that captures cross-task insights.

The motivation for this design and the necessity of each knowledge distillation level are detailed in Section 2. We have also included an ablation study evaluating the impact of different components (see our response to Reviewer opND’s comment W1), and a sanity check validating the approximation of data Shapley values (see our response to Reviewer oTHf's comment W1).

We will incorporate the above discussion into the paper to further clarify the rationale behind our proposed methodology and facilitate readers' understanding of our technical design choices.

Q3. We would like to clarify that our objective is fundamentally different from enhancing the analytic performance of downstream models. Instead, our focus lies in pre-hoc detection of potential data deviations at the point of entry, with the explicit goal of preventing such records from contaminating the EHR system. Our approach raises alerts upon detecting deviations, enabling clinicians and healthcare staff to take timely actions, such as investigating root causes or refining data entry workflows, thereby ensuring overall data fidelity.

We understand that this comment likely arises from our use of data Shapley values as a basis for constructing a fidelity measure. Data Shapley values estimate a sample’s contribution to downstream tasks, which does not always equate to its alignment with true physiological measurements. This distinction is well noted, and addressing this gap forms the core innovation of our work. Specifically, we integrate task-specific data Shapley values derived from multiple clinically relevant tasks into a unified fidelity predictor, which captures the holistic quality of EHR data beyond any single downstream objective. By combining these multi-task contributions, our approach reduces the risk of over-reliance on task-specific correlations and produces a more robust indicator of data fidelity. Experimental results and expert evaluations from clinicians confirm that this integrated approach effectively identifies deviations and highlights clinically meaningful data quality issues observed in real-world cases.

We also acknowledge the complex relationship between misleading data and downstream performance. Some deviations degrade performance by introducing noise or systematic errors, while others may inadvertently boost performance via spurious correlations or confounding factors. Although exploring such task-dependent effects is beyond the primary scope of this study, we agree that this is an important perspective and will revise the paper to (i) restate our emphasis on data fidelity rather than downstream performance enhancement, and (ii) expand the discussion to reflect the broader implications and impact of misleading data on downstream performance.

W1. Ablation study results. We first clarify that the knowledge distillation process from $\mathcal{O} _ {ds}$ to $\mathcal{O} _ {nn}$—which enables the derivation of application-specific data valuation for downstream analysis—is an essential component of our approach and cannot be removed. In response to the suggested ablation study, we have compared our proposal against several weakened variants of the subsequent distillation step from $\mathcal{O} _ {nn}$ to $\Psi$. As shown in the table below, our approach, which incorporates an attention-based aggregation strategy, consistently achieves the lowest loss. These results underscore the effectiveness of the proposed teacher fusion mechanism and demonstrate the necessity of a principled aggregation strategy in this knowledge distillation process.

We have further conducted a fine-grained ablation analysis to assess the contribution of each component in the overall objective loss in Equation (12). Specifically, we examine the effects of removing individual loss terms, disabling the dynamic weight adjustment mechanism, and modifying the temperature setting. As shown in the table below, our full approach achieves the best performance. Performance degrades noticeably when any component is removed or altered, confirming the importance of each design choice. These results collectively underscore that the synergy among loss terms, dynamic weighting, and appropriate temperature calibration is critical for optimal knowledge distillation.

Rule-based approaches leveraging domain knowledge. Existing rule-based methods that leverage domain knowledge—particularly for laboratory test results—primarily focus on detecting "values outside normal physiological ranges" [c, d, e]. For this study, we have identified a publicly available list of clinically validated variable ranges associated with the MIMIC-III multi-task learning benchmark. This list was developed in consultation with clinical experts and reflects their domain knowledge of physiologically plausible measurement ranges [e]. Each variable in this list is associated with predefined upper and lower bounds that specify a physiologically acceptable range. Observed values falling outside these thresholds can be flagged as abnormal, thus forming a simple domain-informed rule-based detection method.

We have summarized key features from our dataset and reported their empirical means and standard deviations ($\sigma$) alongside the corresponding valid ranges in the table below. Notably, the clinically defined ranges typically exceed $10\sigma$, far beyond the scale of deviations considered in our experiments (which are limited to perturbations up to $5\sigma$). As a result, such rule-based methods are only capable of identifying overt anomalies and are inadequate for capturing more subtle yet meaningful deviations. In contrast, our proposed $\Psi$ is specifically designed to detect small-magnitude deviations often overlooked yet potentially impactful (see Figure 3). This capability supports timely clinical intervention and contributes to more robust data quality assurance in real-world healthcare settings.

[c] Shi, Xi, et al. "An automated data cleaning method for Electronic Health Records by incorporating clinical knowledge." BMC Medical Informatics and Decision Making 21.1 (2021): 267.

[d] Wang, Zhan, et al. "Rule-based data quality assessment and monitoring system in healthcare facilities." Improving Usability, Safety and Patient Outcomes with Health Information Technology. IOS Press, 2019. 460-467.

[e] Wang, Shirly, et al. "Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-iii." Proceedings of the ACM conference on health, inference, and learning. 2020.

W2. We clarify that the primary goal of our proposal is to enable pre-hoc detection of potential data deviations at the point of data entry into the EHR system during clinical practice. The aim is not data cleaning or post-hoc removal for downstream analytics, but rather early identification of anomalies at the time of data recording. By highlighting entries with possible deviations as they are introduced, our approach supports real-time quality control and reduces the likelihood of erroneous data entering the system. This leads to improved EHR data quality, which may in turn enhance the performance of downstream analytical tasks. However, this improvement is a secondary benefit; our central goal remains the reliable assessment of data fidelity in situ, to prevent flawed data from influencing clinical decision-making for real-world patients.

We also clarify that the presented case studies go beyond simple duplication scenarios. For instance, the Point-of-Care Testing (POCT) laboratory tests featured in our case studies are conducted at or near the site of patient care rather than in centralized laboratories, with the aim of enabling rapid clinical decisions. When multiple test results for the same feature are recorded within a short timeframe and exhibit inconsistencies or abnormalities, the underlying causes can be multifactorial. These include misattribution of results to the wrong patient, errors in sample collection, device malfunctions in uncontrolled environments, or testing conducted at inappropriate times relative to clinical interventions or medication schedules.

Our clinicians confirm the practical necessity and clinical value of detecting such potential data deviations in a pre-hoc manner. It is emphasized that alerting healthcare workers to possible issues at the time of testing can help prevent erroneous data from being integrated into subsequent clinical workflows, thereby improving the robustness of patient care and decision-making.

W3 & Questions. We would like to clarify that our primary goal is neither post-hoc data removal or cleaning, nor the development of high-performing EHR analytic models. Instead, our focus is on pre-hoc detection of potential data deviations prior to their integration into the EHR system in clinical practice. Such early detection enables clinicians and healthcare workers to intervene promptly, facilitating root cause investigation and the refinement of clinical workflows—for example, by improving device stability or updating standard operating procedures.

While we acknowledge the possibility of leveraging our deviation detection outputs as auxiliary information in EHR analytics—for example, using them as pseudo-labels to inform model training—we note that the impact of such inherent noise or deviations is nuanced. Depending on the context, these signals could either degrade performance or promote model robustness. Exploring this trade-off, however, goes beyond the primary scope of our current study. We will revise the paper to clarify these points and incorporate further discussion accordingly.

W1. While we acknowledge that computing exact data Shapley values would yield higher accuracy than permutation-based sampling, the exact computation incurs exponential complexity and is intractable for the large-scale, real-world datasets used in our experiments. Further, permutation sampling remains a widely adopted and theoretically grounded approximation strategy in this line of research [14, 46].

To evaluate the potential impact of this approximation, we have performed 100K independent Monte Carlo simulations to estimate data Shapley values, employing two complementary metrics to assess convergence with statistical rigor across four post-AKI progression tasks. Standard Error ($SE = \sigma/\sqrt{n}$) underpins the construction of confidence intervals: $\bar{x} \pm 1.96 \cdot SE$ provides 95% confidence bounds for the data Shapley values. The observed $O(n^{-1/2})$ decay rate aligns with the Central Limit Theorem, and the final SE reaches sub-percentage levels across all tasks. Rolling Variance ($RV$) quantifies temporal stability by measuring the variance of cumulative means over sliding windows of 100 samples. This serves as a practical indicator to assess when additional sampling yields diminishing returns. The observed monotonic decline in $RV$ indicates statistical stabilization of the estimates.

These analyses provide both theoretical justification and empirical support for the reliability of the permutation-based data Shapley value estimates used in our subsequent analyses.

W2. We thank the reviewer for highlighting the potential influence of domain shift on our amortized Shapley framework. We agree that domain shift—characterized by changes in the underlying data distribution—is a critical and challenging problem, particularly in healthcare settings where new diseases (e.g., COVID-19) or highly specialized sub-domains can result in out-of-distribution (OOD) samples. We acknowledge this as a potential limitation of our study, but emphasize that it is unlikely to affect our current setting for the following reasons.

First, our work focuses on EHR data within a single hospital. Although federated EHR scenarios involving multiple hospitals are valuable, they are beyond the scope of our current study, as we do not manage inter-hospital domain shifts.

Second, within our hospital, data heterogeneity across different departments and diseases is already addressed through our multi-task modeling framework. This approach effectively unifies diverse clinical contexts by incorporating task-specific information without requiring explicit domain adaptation.

Third, while domain shift research often investigates models trained on data from one domain and deployed in another, such a setting is not highly relevant to our routine clinical workflows. Except for emerging infectious diseases or other rare events, our clinical decision support system is typically developed and applied within the same institutional and departmental context.

Finally, our approach is primarily designed for pre-hoc detection of data deviations at the point of data entry, aiming to alert clinicians and healthcare staff before erroneous data affects downstream processes. From this perspective, the risk posed by domain shift is minimal because the features of interest—such as frequently recorded vital signs and laboratory results—are consistent across common clinical tasks. Domain shift in rarely used or highly specialized tests (e.g., biomarkers for rare diseases) has far less impact on data quality assurance, which is our main focus.

While we recognize domain shift as an important area for future extension, we believe it does not undermine the contributions or validity of our current work. We will revise the paper to explicitly discuss this potential limitation and outline possible directions for integrating domain adaptation techniques in broader, cross-institutional settings.

W3. Yes, the threshold $\delta$ in Definition 1 can be adapted to suit specific practical requirements and deployment scenarios. For instance, in our experiment evaluating the sensitivity of the proposed approach $\Psi$ to varying deviation magnitudes (results shown in Figure 4), we set $\delta = 0$ and interpret the sign of $\Delta \Psi$ as an indicator of whether a deviation is detected. As shown in Figure 4, when the detection objective emphasizes reliable performance within the interquartile range (P25–P75), this zero-threshold setting is sufficient. In particular, the sign of $\Delta \Psi$ consistently provides a stable and effective signal for identifying deviations, even for subtle perturbations near the decision boundary.

We acknowledge that in real-world applications, the threshold $\delta$ should be customized based on domain-specific considerations. We work closely with our frontline clinicians to calibrate this threshold for practical use, balancing detection accuracy with the need to minimize alert fatigue in clinical workflows.

W4. To assess the fidelity of EHR data, we leverage data valuation techniques to quantify the contribution of individual data samples. Among various approaches, the data Shapley value has emerged as a prominent method in the data valuation literature [14, 25, 46], and is increasingly recognized as a standard tool for assessing data quality in high-stakes domains such as healthcare [54, 57, a, b]. Accordingly, we adopt the data Shapley value as our data valuation method to derive EHR data fidelity. While the selection of data valuation methods is not the main focus of our work, our proposed approach is readily applicable to other, potentially more advanced, alternatives that may further enhance detection performance.

Given the high computational cost of exact data Shapley value calculation, we employ Monte Carlo sampling for approximation. The validity of this approximation has been discussed in detail in our response to your comment W1. Empirical results demonstrate that our approach achieves strong performance in detecting EHR data deviations and provides clinically meaningful insights into potential data quality issues of real-world cases, as validated by our clinicians.

Importantly, our approach is designed as a pre-hoc data fidelity assessment procedure that addresses critical gaps in current EHR workflows by (i) identifying and excluding erroneous records from clinical workflows, thereby mitigating the risk of inappropriate interventions and adverse outcomes; and (ii) enabling targeted data correction strategies to improve overall data quality and ensure more reliable downstream analyses. While it is possible that our approach may produce potential false positives or negatives, the deviations it identifies are sufficiently informative to warrant clinical investigation prior to data ingestion into the EHR system. In this regard, the proposed approach functions as a practical and effective data fidelity screening mechanism, reinforcing the robustness and reliability of the EHR data infrastructure.

[a] Pandl, Konstantin D., et al. "Trustworthy machine learning for health care: scalable data valuation with the shapley value." Proceedings of the Conference on Health, Inference, and Learning. 2021.

[b] Bloch, Louise, Christoph M. Friedrich, and Alzheimer’s Disease Neuroimaging Initiative. "Data analysis with Shapley values for automatic subject selection in Alzheimer’s disease data sets using interpretable machine learning." Alzheimer's Research & Therapy 13.1 (2021): 155.

Thank you to the authors for the detailed responses. While some of my concerns have been addressed, the main issues I raised still remain unresolved.

• First, the supplementary SE experiments verify only that permutation samplings produce consistent outputs; they do not quantify the deviation from the oracle. I am therefore still concerned that the error gap might widen after the two-stage distillation, given that earlier work employed permutation sampling in just one training stage with relatively simple ML models. In addition, RV was designed for time-series data, so its suitability here is unclear because the permutation samples are not time-series (please let me know if I have misunderstood).

• Second, I remain unconvinced that domain shift can be disregarded when the framework is targeted to be evaluated at only a single institution or department. Domain shifts, especially temporal ones, frequently arise in real-world healthcare data and may manifest within a single department, let alone across the wider institution.

• Third, the framework appears fragmented in real-world applications, as its performance seems highly sensitive to an ad-hoc threshold. It is unclear how a valid threshold can be predetermined before conducting any assessment, especially in the absence of a sensitivity analysis. Is there a general guideline for selecting this threshold, or must it always be determined/calibrated by clinicians on a case-by-case basis?

I am willing to adjust my score if the authors can adequately address the above concerns.