GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs

Thank you for recognizing the clarity and composition of our paper, the novelty of our LLM-built knowledge graph for medical concepts, and the comprehensiveness of our experimental work and interpretability analysis. We address your concerns and answer your questions below. We also uploaded a revision and used blue to mark the new changes.

---

W1: The discussion on related work could be more comprehensive.

(1) For recent works about KG extraction using LLM: We have taken your advice and added the related papers to our related work section, which are highlighted in $\textcolor{blue}{blue}$.

(2) For the paper related to the bi-attention method [1]:

Thanks for letting us know about this paper! We have reviewed the work by Lu et al. [1] and acknowledge its relevance to our bi-attention mechanism. We have accordingly updated our related work section to include this reference. While there are conceptual similarities between their multi-level attention mechanism and our bi-attention mechanism, several key distinctions set our methodology apart:

1. Simplicity and Efficiency: Our model diverges from Lu et al.’s approach by calculating node-level and visit-level attention scores through a multi-hot matrix representation. In this schema, each row corresponds to a distinct multi-hot vector that signifies the presence of nodes within a particular visit, with subsequent rows differentiating among various visits. This design streamlines the process, as it utilizes sparser representations that inherently require fewer dense matrix operations. Consequently, our method enhances practical computational efficiency and facilitates parallel processing.
2. Independence: Another advantage of our approach is the independent computation of node-level and visit-level attention. By decoupling these two, we prevent the potential propagation of errors from one level to another, enhancing the robustness of the model.

We have conducted an experiment to showcase the performance difference between these two attention computation methods applied to GraphCare on MIMIC-III mortality prediction:

The comparison clearly demonstrates that our bi-attention mechanism not only significantly enhances inference time efficiency but also achieves slightly higher prediction accuracy.

(3) Adding new baselines:

Thanks for introducing these new papers! We will include some of them in comparison. Due to the intricate process involved in accurately reproducing these methodologies, we anticipate needing a few days (before the author response deadline) to integrate them into our testing framework thoroughly We have shown the new results in "Update to Part 1 (New Baselines)" below.

---

W2: The knowledge graph building is somewhat unclear. A knowledge graph usually contains edges with multiple types. However, the edge type is not reflected in KG extraction, edge clustering, and the graph neural network.

Thank you for your observation regarding the construction of our knowledge graph (KG). We included our KG construction details in Appendix D. Our approach to KG assembly through LLM prompting leads to a variable number of edge types, which may differ from traditional KGs that typically feature a predefined edge taxonomy. The statistics regarding the number of edge types after clustering are revealed in Table 7 in Appendix D.3.

---

W3: One suggestion is to compare more clustering methods.

Thank you for the valuable suggestion to explore a broader range of clustering methods. We agree that a comprehensive comparison, including hyperparameter tuning, is crucial for a thorough evaluation of clustering strategies. Due to the extensive nature of these experiments, we will diligently work on this analysis and aim to present the results in our final revision.

(References are attached in Part 2)

Thank you for recognizing the breadth of our experiments, the innovative use of LLM in KG generation, and the interpretability provided by our knowledge graph in GraphCare. We address your concerns and answer your questions below. We also uploaded a revision and used blue to mark the new changes.

---

W1: The performance of mortality prediction is questionable. While previous literature has reported AUPRC values exceeding 20 on the MIMIC dataset, the results presented in Table 2 fall notably short of this benchmark [1]. What leads to such a gap?

Thank you for highlighting the performance discrepancy in mortality prediction as compared to previous literature. The observed gap can be attributed to several key differences:

1. Task Definition: Our study defines mortality prediction as forecasting the outcome of the next visit based on previous visits, aligning with studies [3, 4]. This contrasts with [1], which considers mortality within specific time windows and employs a different feature selection approach.
2. Input Features: Unlike [1], which selects features from multiple tables (chartevents, labevents, outputevents, icustays, diagnoses_icd) in the MIMIC-III datasets, our study uses features from the (diagnoses_icd, procedures_icd, prescriptions) tables, consistent with the setting in [3].
3. Medical Coding: We use the CCS and ATC (level-3) coding systems, as mapped from ICD-9 and NDC respectively, a well-recognized setting in literature [3, 6, 7]. This differs from the ICD-9 coding system used in [1].

These factors collectively contribute to the differences in performance results.

---

W2: The paper delves into various engineering details, such as Knowledge Graph (KG) construction and attention initialization, without offering sufficient clarity. The notation in sec 3.3 is too complicated to follow. Many of them are randomly presented without definition.

Thank you for pointing out the need for greater clarity in our explanation of the Knowledge Graph (KG) construction and attention initialization, as well as the complexity of notations in Section 3.3.

Engineering Details: To enhance clarity, we have comprehensively updated Appendix C. This appendix now contains detailed explanations of each step mentioned in Section 3, ensuring a clearer understanding of the engineering aspects of our work.

Notations Definition: We understand that the notations may have seemed complex. To address this, we have included a comprehensive notation-description table in Appendix H - Table 11 in the updated version of our paper. This table is designed to provide clear definitions and explanations for the notations used, aiding in easier comprehension of the paper.

We hope these updates make the technical aspects of our paper more accessible and easier to follow.

W3: The paper only compares with the baseline methods without leveraging external information/knowledge. However, it will be more important to compare with the knowledge graph generated by other methods [2].

Thank you for suggesting a comparison with the method described in [2], which utilizes the Google Health Knowledge Graph (GHKG). We acknowledge the value of such a comparison; however, there are practical constraints to consider:

1. Availability of GHKG: The GHKG, a key component of the method in [2], is not publicly available. Access to this resource is crucial for replicating or comparing with their approach. Unfortunately, obtaining permission from Google for access to the GHKG is not feasible within the limited time frame of this discussion window.

2. Comparison with Other Graph-Based Methods: While we were unable to compare with the GHKG-based method, we have conducted comparisons with other graph-based methods, GRAM [8] and GAMENet [9]. These comparisons are detailed in Table 2 and Figure 2 of our paper, providing insights into the performance of our approach relative to other graph-based methods that do not rely on proprietary datasets like the GHKG.

We believe these comparisons, although not with the specific method you suggested, still offer valuable insights into the efficacy of our approach in the context of available graph-based methods in the field.

(References are attached in Part 2)

Thank you for recognizing the strengths of GraphCare, especially its effectiveness with the MIMIC datasets and our comprehensive approach to limitations and future directions. We address your concerns and answer your questions below. We also uploaded a revision and used blue to mark the new changes.

---

W1: The complexity of the proposed framework presents challenges for implementation. The paper would benefit from a more comprehensive explanation of the implementation specifics of the framework.

Thank you for your feedback regarding the complexity of our framework and the need for clearer implementation details. We agree that a more detailed explanation would enhance the understanding of our framework.

1. In our original submission, we provided a summary of the implementation details in the README.md file within the supplementary material's code folder. This was intended to offer quick access to the necessary information for replicating our framework.

2. Following your suggestion, we have revised and expanded the Appendix C - Implementation Details section. This update aims to provide a more thorough and organized explanation of the implementation process, ensuring that readers can easily understand and apply our framework. We have also created a comprehensive notation table in Appendix H to enhance the clarity of our paper.

We believe these enhancements will greatly assist in comprehending and utilizing our proposed framework effectively.

W2: While the authors discuss the interpretability of GraphCare in Section 4.3, the depth of analysis is confined to observations on entity connections and their impact on model performance. It would be beneficial if the framework could uncover deeper insights in line with existing literature, or identify longer-distance relationships that might inspire new scientific discoveries.

Thank you for your valuable feedback regarding the depth of our interpretability study in GraphCare. We agree that exploring deeper insights and longer-distance relationships could significantly enhance its application in real-world scenarios and foster scientific discoveries.

To address your point, GraphCare could indeed be adapted to investigate critical factors in various scenarios. For instance, in identifying key causes of a disease, we can focus on the disease as the main prediction target, systematically examining and pruning long-distance relationships to pinpoint the most crucial ones. Similarly, we could study how these relationships might correlate with a patient's age or other demographic details. The potential applications are practically endless.

It's essential to recognize that conducting such in-depth analysis, especially in a clinical setting, demands comprehensive validation. This might involve integrating sophisticated fact-checking methods with advanced information retrieval techniques. While this level of detailed scrutiny is crucial, it extends beyond the primary scope of our current study.

Our main contribution lies in developing GraphCare as a versatile framework. This framework is instrumental in constructing personalized KGs and adeptly managing diverse prediction tasks. Nonetheless, we acknowledge the significance of your suggestion. It aligns well with our objectives, and we are eager to explore it as a central aspect of our future research.