6. Additional clarification on how to aggregate clinical events embeddings from the matrix “Z” to obtain a single visit representation is needed (lines 333-335).

For each visit $V_{t}$, we obtain a representation $\mathbf{v}_t \in \mathbb{R}^{d}$ by multiplying the final embedding matrix $\mathbf{Z}$ with a multi-hot vector $\mathbf{u}_t = \{0,1\}^{N^{(L)}}$, which represents clinical events existence in the visit. This operation simply sums up the representation vectors of all the codes inside a visit to get to the visit embedding.

7. In line 442, the authors likely mean “Table 3” instead of “Figure 3”. Similarly, in line 500, they might intend to refer to “Right” instead of “Left”.

Thanks for pointing this our. We fix it in the paper.

1. The authors should describe the elements in Fig. 6, such as the meaning of the nodes based on their color, to enhance understanding of the figure’s components.

Figure 6 presents an interpretative case study for OntoFAR, illustrating how it learns the representation of ICD-9 diagnosis code 428.0, which corresponds to "Congestive heart failure, unspecified" (CHF). The following points provide clarification:

• The figure depicts the extracted sub-knowledge graph (sub-KG) for ICD-9 code 428.0 within the Meta-KG framework.
• Each hierarchical level contains diverse medical codes, represented as follows: red circles for procedures, green circles for diagnoses, and blue circles for drugs.
• Larger circles indicate higher-level codes, representing more general concepts compared to lower-level codes.
• Within each hierarchical level, solid blue lines connect the codes, representing co-occurrence-driven edges for horizontal message passing.
• Dashed lines illustrate parent-child relationships for bottom-up vertical message passing (HGIP), with color coding (red for procedures, green for diagnoses, and blue for drugs). Parent-child edges for top-down GRAM propagation are depicted using solid black lines.
• The edge weights during horizontal and vertical propagation are learned through attention mechanisms. The thickness of the edges in the figure reflects the relative attention assigned to each edge.

2. The experiments are conducted on MIMIC-III and MIMIC-IV, which originate from the same healthcare provider. More datasets needed.

While MIMIC-III and MIMIC-IV share some patient overlap, they differ significantly in critical aspects, making their joint use valuable. MIMIC-III covers ICU stays from 2001 to 2012, while MIMIC-IV spans 2008 to 2019 and includes newer coding systems like ICD-10-CM/PCS. MIMIC-IV also adopts a modular structure with enhanced metadata. We also considered including the eICU Collaborative Research Database. However, many features, such as medications, are recorded using generic names rather than standardized systems like RxNorm or ATC. Since our study focuses on integrating different ontologies for diagnoses, drugs, and procedures, we decided not to include eICU, as it lacks an ontology system for each of these features.

3. The experiments are conducted on MIMIC-III and MIMIC-IV, which originate from the same healthcare provider. More datasets needed.

While MIMIC-III and MIMIC-IV share some patient overlap, they differ significantly in critical aspects, making their joint use valuable. MIMIC-III covers ICU stays from 2001 to 2012, while MIMIC-IV spans 2008 to 2019 and includes newer coding systems like ICD-10-CM/PCS. MIMIC-IV also adopts a modular structure with enhanced metadata. We also considered including the eICU Collaborative Research Database. However, many features, such as medications, are recorded using generic names rather than standardized systems like RxNorm or ATC. Since our study focuses on integrating different ontologies for diagnoses, drugs, and procedures, we decided not to include eICU, as it lacks an ontology system for each of these features.

4. In general, the writing requires significant improvements. For example, line 171: “work depicted Figure 1” should be corrected to “work depicted in Figure 1”; additionally, there are multiple typographical errors, such as the use of “massage” instead of “message” multiple times.

Thanks for pointing this our. We fix it in the paper.

5. Authors need to clarify the meaning of “Pr” in equation 1.

Thanks for pointing that our. By ‘Pr’ we simply mean the prompt. So Pr(c) denotes the generated prompt for code c. We will clarify this in the paper.

Dear Reviewer 8VUN,

Thank you for your thoughtful review and valuable feedback. Below, we address your questions and provide new experimental results to incorporate your recommendations and address your concerns.

1. The approach is evaluated solely on the task of sequential diagnosis prediction

We added two more clinical tasks: 1) mortality prediction and 2) hospital readmission prediction in additional to diagnosis prediction. Table A2 presents the results of these tasks alongside baseline comparisons, based on the MIMIC-III dataset. Experiments on the MIMIC-IV dataset are underway, and we will share the results once completed.

In addition, we also added more relevant baselines for mortality prediction and readmission prediction:

[1] Nguyen, Phuoc, et al. "Deepr: a convolutional net for medical records (2016)." ArXiv160707519 Cs Stat (2016).

[2] Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling." arXiv preprint arXiv:1803.01271 (2018).

[3] Song, Lihong, et al. "Medical Concept Embedding with Multiple Ontological Representations." IJCAI. Vol. 19. 2019.

[4] Choi, Edward, et al. "Learning the graphical structure of electronic health records with graph convolutional transformer." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 01. 2020.

[5] Gao, Junyi, et al. "Stagenet: Stage-aware neural networks for health risk prediction." Proceedings of The Web Conference 2020. 2020.

[6] Ma, Liantao, et al. "Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 01. 2020.

[7] Zhang, Chaohe, et al. "GRASP: generic framework for health status representation learning based on incorporating knowledge from similar patients." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 1. 2021.

[8] Xu, Ran, et al. "Hypergraph transformers for ehr-based clinical predictions." AMIA Summits on Translational Science Proceedings 2023 (2023): 582.

[9] Hadizadeh Moghaddam, Arya, et al. "Contrastive Learning on Medical Intents for Sequential Prescription Recommendation." Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024.

Table A2: Prediction Performance for Hospital Readmission and Mortality

Dear Reviewer 8VUN,

Thank you for your detailed and thoughtful initial review. We have carefully addressed your concerns in our rebuttal. If possible, could you kindly review our response and share any additional thoughts or feedback you may have?

Also, as we promised in our earlier response, we also completed the MIMIC-IV experimentation for the additional tasks of mortality prediction and readmission prediction (results are percentage %):

We sincerely appreciate your thoughtful feedback and have taken great care to thoroughly address your concerns. If you have any further comments or suggestions, we would be delighted to consider them to further improve the quality of our submission. Thank you once again for your time and invaluable insights!

2. There are different knowledge bases such as SNOMED-CT, CCS, and several others even for the diagnosis. Is there a reason why only one knowledge base is explored for diagnosis and/or medication (SNOMED-CT works for medication as well)?

Our model can be adopted on many ontology structures including, ICD9, ICD10, SNOMET, and CCS. We use ICD for diagnosis because it is currently the most widely used ontology in EHR datasets including MIMIC. Besides, ICD provides the rich hierarchies we need to fuse with other ontologies systems.

3. Given MIMIC-III and MIMIC-IV share the same dataset, it would be helpful to benchmark against something that is likely to have different patients. eICU is a good example of a potential open-source dataset.

While MIMIC-III and MIMIC-IV may share some patient overlap (we can’t verify it since all patients are deidentified), they differ significantly in critical aspects. MIMIC-III covers ICU stays from 2001 to 2012 (46,000 patients), while MIMIC-IV spans 2008 to 2019 (383,220 patients) and adopts a newer coding system like ICD-10-CM/PCS.

Although eICU Collaborative Research Database is a another widely used public EHR dataset, the key features of medications in eICU are recorded using generic names rather than standardized ontology concepts like RxNorm or ATC. Since our study focuses on integrating different ontologies for diagnoses, drugs, and procedures, eICU lacks an ontology system for each of these features.

4. The methodology section is quite dense and a bit hard to parse especially when trying to ascertain how information is shared across the different ontologies. It would be helpful to provide an example using ATC and ICD-9 hierarchy. Based on Figure 1, GAT or HAT are the mechanisms for sharing information across the same concept level across ontologies but this isn't made explicit.

We included an algorithm in appendix A.1 which demonstrates the step-by-step process of medical code representation learning in OntoFAR. Also, Figure 6 illustrates an example of how OntoFAR learns the representation of the ICD-9 code 428.0, which denotes "Congestive heart failure, unspecified" (CHF).

5. The citation style is incorrect, you should swap it to \citep as the default instead of \cite.

Thank you very much for pointing out that out. We fixed this error.

6. Why are OpenAI off-the-shelf LLMs used when there are many other open-source LLMs available? How sensitive is the performance to the quality of the embeddings from the LLM?

OpenAI’s LLMs, such as GPT, are renowned for their state-of-the-art performance and extensive pretraining on diverse datasets. Also, they provide a standardized API to retrieve embeddings. This ensures high-quality and efficient general knowledge embeddings, which are particularly valuable when initializing domain-specific representations.

Dear Reviewer xUeg,

Thank you for your thoughtful review and valuable feedback. Below, we address your questions and provide new experimental results to incorporate your recommendations and address your concerns.

1. Only a single downstream task is benchmarked.

We added two more clinical tasks: 1) mortality prediction and 2) hospital readmission prediction in additional to diagnosis prediction. Table A2 presents the results of these tasks alongside baseline comparisons, based on the MIMIC-III dataset. Experiments on the MIMIC-IV dataset are underway, and we will share the results once completed.

In addition, we also added more relevant baselines for mortality prediction and readmission prediction:

[1] Nguyen, Phuoc, et al. "Deepr: a convolutional net for medical records (2016)." ArXiv160707519 Cs Stat (2016).

[2] Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling." arXiv preprint arXiv:1803.01271 (2018).

[3] Song, Lihong, et al. "Medical Concept Embedding with Multiple Ontological Representations." IJCAI. Vol. 19. 2019.

[4] Choi, Edward, et al. "Learning the graphical structure of electronic health records with graph convolutional transformer." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 01. 2020.

[5] Gao, Junyi, et al. "Stagenet: Stage-aware neural networks for health risk prediction." Proceedings of The Web Conference 2020. 2020.

[6] Ma, Liantao, et al. "Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 01. 2020.

[7] Zhang, Chaohe, et al. "GRASP: generic framework for health status representation learning based on incorporating knowledge from similar patients." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 1. 2021.

[8] Xu, Ran, et al. "Hypergraph transformers for ehr-based clinical predictions." AMIA Summits on Translational Science Proceedings 2023 (2023): 582.

[9] Hadizadeh Moghaddam, Arya, et al. "Contrastive Learning on Medical Intents for Sequential Prescription Recommendation." Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024.

Table A2: Prediction Performance for Hospital Readmission and Mortality

7. How does your model compare against ADORE, which incorporates a multi-relational medical ontology, SNOMED-CT which combines medications and diagnoses into a single representation.

Our work and ADORE have key differences in method:

• In horizontal propagation, ADORE relies on the SNOMED ontology to capture interactions between diagnosis and drug codes (excluding procedure codes) using existed relational edges. However, this approach is based on an ontology structure (external source) and does not fully reflect interactions among medical codes from different sources as observed in EHRs reality. In contrast, OntoFAR captures these interactions more effectively by:

• • Connecting codes from different sources at all levels of hierarchical ontologies, using co-occurrence information inherent in the dataset to reflect EHR realities. (Connecting these multi-source ontologies using EHRs reality, which lead to Mining EHR patterns holistically, integrating them with ontology structure learning.) in other words, we captures these multisource interaction by taking into account a synergy of EHRs (real statistical information) and Ontology (predefined medical knowledge).
• • Deriving associations across varying granularities, integrating fine-grained and coarse-grained information for enhanced representation learning.

• In vertical code encoding, ADORE relies solely on the top-down propagation of GRAM, which does not account for the hierarchy's order. It combines parent codes with child nodes using attention to generate embeddings but does not distinguish between parents or incorporate information about children into parent embeddings. In contrast, OntoFAR introduces the Hierarchical Graph Information Propagation (HGIP) module. This module propagates information upwards through a bottom-up sequence of attention subgraphs across consecutive ontology levels. The final code representation is then generated using a GRAM module, ensuring a more comprehensive encoding of hierarchical relationships.

We were unable to implement the GitHub code for this baseline within the limited time available to include it in our experiments. However, we will acknowledge this valuable work in our related work section and include a discussion about it. To address this issue, we added 2 more recent baselines:

[8] Xu, Ran, et al. "Hypergraph transformers for ehr-based clinical predictions." AMIA Summits on Translational Science Proceedings 2023 (2023): 582.

[9] Hadizadeh Moghaddam, Arya, et al. "Contrastive Learning on Medical Intents for Sequential Prescription Recommendation." Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024.

Please refer to Table A1 below.

Table A1: Sequential Diagnosis Prediction

Dear Reviewer xUeg,

Thank you for your detailed and thoughtful initial review. We have carefully addressed your concerns in our rebuttal. If convenient, could you kindly review our response and share any additional thoughts or feedback? Your input would be greatly appreciated!

Also, as we promised in our earlier response, we also completed the MIMIC-IV experimentation for the additional tasks of mortality prediction and readmission prediction (results are percentage %):

We are truly grateful for your thoughtful feedback and have worked diligently to address your concerns. If you have any further comments or suggestions, we would gladly consider them to refine and improve our submission. Thank you again for your time and invaluable insights!

• The paper uses two experimental datasets with the same scope: While MIMIC-III and MIMIC-IV share some patient overlap, they differ significantly in critical aspects, making their joint use valuable. MIMIC-III covers ICU stays from 2001 to 2012 (46,000 patients), while MIMIC-IV spans 2008 to 2019 (383,220 patients) and includes newer coding systems like ICD-10-CM/PCS. Although eICU Collaborative Research Database is a another widely used public EHR dataset, the key features of medications in eICU are recorded using generic names rather than standardized ontology systems like RxNorm or ATC. Since our study focuses on integrating different ontologies for diagnoses, drugs, and procedures, eICU lacks an ontology system for each of these features.
• The performance gains are marginal. This is especially evident when LLM embedding initialization is excluded: Our results are based on the complex task of predicting ICD-9 diagnosis codes for the next visit, involving 4,283 unique codes in MIMIC-III and 8,818 in MIMIC-IV. The large prediction space makes the task challenging, where the gains in proposed model represent significant improvements in predictive performance. In addition, the performances are obtained through a cross-validation setting. Notably, our model excels in two key aspects: first, it performs significantly better at predicting less frequent (rare) diagnosis codes (Figure 3), and second, it demonstrates superior performance in data-limited settings (Figures 4 and 5). As shown in Table 3 (ablation study), initializing embeddings with LLM enhances overall performance. However, even without LLM, our model outperforms other baselines. Additionally, we have included new ablation experiments in Table A4 below, highlighting the contribution of each component across code frequency categories for your reference.

• The proposed methodology was evaluated on a limited variety of prediction tasks: We added two more clinical tasks: 1) mortality prediction and 2) hospital readmission prediction in additional to diagnosis prediction. Table A2 presents the results of these tasks alongside baseline comparisons, based on the MIMIC-III dataset. Experiments on the MIMIC-IV dataset are underway, and we will share the results once completed.

In addition, we also added more relevant baselines for mortality prediction and readmission prediction:

[1] Nguyen, Phuoc, et al. "Deepr: a convolutional net for medical records (2016)." ArXiv160707519 Cs Stat (2016).

[2] Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling." arXiv preprint arXiv:1803.01271 (2018).

[3] Song, Lihong, et al. "Medical Concept Embedding with Multiple Ontological Representations." IJCAI. Vol. 19. 2019.

[4] Choi, Edward, et al. "Learning the graphical structure of electronic health records with graph convolutional transformer." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 01. 2020.

[5] Gao, Junyi, et al. "Stagenet: Stage-aware neural networks for health risk prediction." Proceedings of The Web Conference 2020. 2020.

[6] Ma, Liantao, et al. "Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 01. 2020.

[7] Zhang, Chaohe, et al. "GRASP: generic framework for health status representation learning based on incorporating knowledge from similar patients." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 1. 2021.

[8] Xu, Ran, et al. "Hypergraph transformers for ehr-based clinical predictions." AMIA Summits on Translational Science Proceedings 2023 (2023): 582.

[9] Hadizadeh Moghaddam, Arya, et al. "Contrastive Learning on Medical Intents for Sequential Prescription Recommendation." Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024.

Table A2: Prediction Performance for Hospital Readmission and Mortality

2. The methodological approach lacks substantial novelty from a machine learning perspective. We believe our work is valuable to the field of EHR mining from the following perspectives.

• As existing approaches are limited by the isolation of different ontology systems (e.g., conditions vs. drugs), OntoFAR integrates diverse ontology graphs (diagnoses, prescriptions, drugs) and employs multifaceted graph learning across ontologies to enhance medical concept representation. To the best of our knowledge, this study is among the first to focus on fusing heterogeneous ontology systems.
• Our method jointly represents medical concepts across multiple ontology structures by performing message passing in two dimensions: • Horizontal propagation uses hierarchical parallel attention graphs to link ontologies, capturing co-occurrence across granularities and integrating fine- and coarse-grained information for richer representation learning and EHR pattern mining. • Vertical propagation within the ontology hierarchy facilitates intra-ontology concept association through the Hierarchical Graph Information Propagation. This two-round progressive encoding technique integrates information across all levels. It controls node update order to incorporate hierarchy and uses multi-head attention for multi-view representations, addressing EHR-ontology inconsistencies with a flexible, multi-sense approach. c. OntoFAR leverages the large language models (LLMs) to understand each target concept with its ontology relationships, providing enhanced embedding initialization for concepts. d. We conducted extensive experiments on two widely used EHR datasets, MIMIC-III and MIMIC-IV, focusing on sequential diagnosis prediction (plus the newly added tasks of readmission prediction and mortality prediction (in Table A2)). Our analysis includes performance enhancement when integrating EHR models (Figure 2), baseline comparisons (Table 2), ablation studies (Table 3), data insufficiency tests (Figures 4 and 5), and interpretative case studies (Figure 6).

3. Details and Clarification on the LLM prompt design strategy: This is comprehensively explained in section 3.3. The key steps are summarized as follows:

• We use LLM to initialize the embeddings of medical concept codes for all ontologies (diagnosis, prescription, drugs) and all levels (child, parent, grandparent).
• We design a specific prompting strategy for each medical code based on code information (the concept names) and its ancestors’ in the ontology.
• We only extract dense embedding from LLM, we do not get any prediction or written inference from LLM. This approach avoids LLM hallucinations while still leveraging its valuable general knowledge.
• The LLM embeddings are generated and stored prior to the training phase.
• These embeddings will be used as initialization of medical concept embeddings (they are not freeze and will be tuned).
• Based on Table 3 in the paper (ablation study), This LLM embedding initialization improves the overall performance. (However, without LLM, our model still outperforms other baselines)

4. A more explicit discussion is needed on whether combining various ontologies in this way can genuinely contribute to model performance. Integrating diverse ontologies lead to better medical concept representation and consequently better performance, for two reasons:

• Based on ablation study, Horizontal Massage Passing, which is the module connecting different ontologies, has the most contribution to the final results
• Compared to the baselines of GRAM, KAMME, MMORE, HAP, where ontology systems are separately treated, we perform significantly better. (Table 2 in the paper, and the Tables A1, A2 added above). We add more explicit discussion in the experimental section 5.2.

5. Related Work: Additional discussion on how the proposed model distinctly differs from related studies would be beneficial. Clarifying the relationship between prior work and the proposed model would enhance the understanding of OntoFAR's unique contributions. We will add more discussion in related work section to explain our work’s differentiation. Clarifications are summarized as follows:

Similarities:

• Both our work and similar works (GRAM, KAME, MMORE, HAP ADORE, etc.) utilize the vertical hierarchy of medical ontology specifically leveraging the parent-child relationship.
• All these methods utilize some kind of attention-based method to capture this parent-child connection inside ontology.

Distinction:

• Our work is the first to combine diverse ontologies (diagnosis (ICD), drug (ATC), procedures (ICD))
• We utilize the synergy of both vertical massage passaging (inside one ontology separately) and horizontal massage passaging (across multiple ontologies)
• Our vertical massage passaging is more advanced and expressive. Refer to Hierarchical Graph Information Propagation (HGIP) section 3.5, which leverages a bottom-up sequence of graph attentions, propagating information throughout the hierarchy.
• We design a specialized prompt to generate LLM embeddings that initialize each node (medical concept) in our Meta Knowledge Graph.

Dear reviewer ZVQp,

Thank you for your detailed and thoughtful review of our manuscript. We greatly appreciate your valuable feedback and the time you have taken to provide constructive suggestions. In the following response, we address your questions and present new experimental results to incorporate your recommendations and address your concerns.

1. Issues of experiments.

• More recent baselines are necessary: We added 2 more baselines to the task of sequential diagnosis prediction.

[8] Xu, Ran, et al. "Hypergraph transformers for ehr-based clinical predictions." AMIA Summits on Translational Science Proceedings 2023 (2023): 582.

[9] Hadizadeh Moghaddam, Arya, et al. "Contrastive Learning on Medical Intents for Sequential Prescription Recommendation." Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024. Please refer to Table A1 below.

Table A1: Sequential Diagnosis Prediction

Thank you so much for your thorough and diligent responses.

Many issues have been resolved thanks to your efforts, especially the extensive additional experiments you conducted, which I believe have significantly contributed to the robustness of the research.

However, there are still some unresolved issues on my end.

The aspect of leveraging heterogeneous ontology seems promising. It looks like applying HGIP in cases where various ontologies appear within the same visit in the EHR time-series dataset is a good approach.

However, the limitation seems to lie in the fact that this method ultimately only initializes the embeddings of the code vocab that serve as inputs to the prediction model. (Why your proposed model couldn't utilize heterogenous ontologies with predictive model? - I mean, is there any other way for end-to-end learning to fully leverage HGIP with predictive model?)

Therefore, I still do not fully understand what specific aspects of OntoFAR make it superior to other baselines, resulting in better performance. For instance, GRAM also leverages ontology, and its prediction model would learn whether codes appearing in different ontologies co-occur within the same visit.

Given that OntoFAR utilizes heterogeneous ontology only for code embedding initialization, I am curious why GRAM, which employs end-to-end learning, performs worse. (I assumed, for a fair comparison, that GRAM also utilized codes from various ontologies in this paper experiment, just like OntoFAR.)

Anyway, thank you for the good answer! I've raised the score.

As we promised in our earlier response, we also completed the MIMIC-IV experimentation for the additional tasks of mortality prediction and readmission prediction (results are percentage %):

We sincerely appreciate your thoughtful feedback and have made every effort to thoroughly address your concerns. If you have any additional comments or suggestions, we would be more than happy to incorporate them to further enhance the quality of our submission. Thank you once again for your time and valuable insights!

Dear Reviewer VcGN,

Thank you for your thoughtful review and valuable feedback. Below, we address your questions and provide new experimental results to incorporate your recommendations and address your concerns.

1. What is the difference between the proposed work and the KAMPNet?

Our work and KAMPNet have key differences in method:

1. In vertical (homogenous) code encoding, KAMPNet models the whole hierarchy structure using one graph, which is unable to account for the order of node calculation by hierarchies. In OntoFAR, we develop the Hierarchical Graph Information Propagation (HGIP) module, where a bottom-up sequence of attention subgraphs on each consecutive ontology levels propagate information upwards, and finally a GRAM module generates the final code representation. Although both models ensure that parent nodes distill information from their descendants, OntoFAR explicitly controls the order of node embedding updates across the main graph.

2. In horizontal heterogenous (multi-source) code encoding, KAMPNet only captures the interaction of muti-source codes (diagnosis, medication) in the leaves code level of ontologies (at only one level of granularity). However, OntoFAR connects multiple ontologies (diagnosis, medication, procedures) across all levels of hierarchies. This approach captures concept co-occurrence across varying granularities, effectively integrating both fine-grained and coarse-grained information for representation learning.

We add this valuable work in our related work section of our paper.

2. The last tested baseline is HAP (2020) and there are many following works during the last four years they are not included.

We made every effort to implement the three baselines you proposed. However, we encountered the following challenges:

KAMPNet: The GitHub repository only includes processed data, and the implementation code is unavailable.

KerPrint and MedPath: Both required additional input files that were missing in repo, preventing us from running the provided code in a short time.

To address this limitation to some extent, we have included two additional recent baselines for the sequential diagnosis prediction task. These additions aim to enhance the comprehensiveness of our validation.

[8] Xu, Ran, et al. "Hypergraph transformers for ehr-based clinical predictions." AMIA Summits on Translational Science Proceedings 2023 (2023): 582.

[9] Hadizadeh Moghaddam, Arya, et al. "Contrastive Learning on Medical Intents for Sequential Prescription Recommendation." Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024.

Please refer to Table A1 below.

Table A1: Sequential Diagnosis Prediction

Also to make the experimental section more comprehensive and extensive, we conduct additional experiment with two more clinical tasks of mortality prediction and readmission prediction. For each task we used all the baselines used in sequential diagnosis tasks plus relevant baselines in the following:

[1] Nguyen, Phuoc, et al. "Deepr: a convolutional net for medical records (2016)." ArXiv160707519 Cs Stat (2016).

[2] Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling." arXiv preprint arXiv:1803.01271 (2018).

[3] Song, Lihong, et al. "Medical Concept Embedding with Multiple Ontological Representations." IJCAI. Vol. 19. 2019.

[4] Choi, Edward, et al. "Learning the graphical structure of electronic health records with graph convolutional transformer." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 01. 2020.

[5] Gao, Junyi, et al. "Stagenet: Stage-aware neural networks for health risk prediction." Proceedings of The Web Conference 2020. 2020.

[6] Ma, Liantao, et al. "Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 01. 2020.

[7] Zhang, Chaohe, et al. "GRASP: generic framework for health status representation learning based on incorporating knowledge from similar patients." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 1. 2021.

[8] Xu, Ran, et al. "Hypergraph transformers for ehr-based clinical predictions." AMIA Summits on Translational Science Proceedings 2023 (2023): 582.

[9] Hadizadeh Moghaddam, Arya, et al. "Contrastive Learning on Medical Intents for Sequential Prescription Recommendation." Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024.

Table A2: Prediction Performance for Hospital Readmission and Mortality

Dear Reviewer HcPA,

Thank you for your thoughtful review and valuable feedback. Below, we address your questions and provide new experimental results to incorporate your recommendations and address your concerns.

1. The comparison with the baselines does not seem to be fair. The proposed framework OntoFAR utilizes the GPT text embedding model for the embeddings of medical concepts, which is a much more powerful model than those used by the baselines.

Based on Table 3 in the paper (ablation study), This LLM embedding initialization improves the overall performance. However, without LLM initialization, our model still outperforms other baselines.

2. The referencing style throughout the paper is not correct; it should use parentheses (i.e., \citep{} instead of \cite{}). "Figure 3" in line 442 should be "Table 3"

Thank you for pointing these two issues out. We fix this issue and upload a new version of the manuscript.

3. What's the base model in Figure 3?

In Figure 3, we aim to demonstrate the compatibility of OntoFAR as an advanced medical concept encoder. OntoFAR can be seamlessly integrated as an add-on to popular existing health predictive models, which can significantly enhance their performance. For example, Figure 3 presents the results of three predictive models (base model)—Transformer, TCN, and RETAIN—before and after incorporating OntoFAR, highlighting the performance improvements achieved with its integration.

4. The baselines used in the paper range from 2017 to 2020, which are kind of outdated. It's better for the authors to consider more recent baselines, such as SeqCare [1] GraphCare, MedPath, RAM-EHR.

We acknowledge that RAM-EHR requires multi-source clinical text data, which makes it incompatible with the scope of our model that exclusively uses discrete EHR codes. For GraphCare, we implemented it using the code provided in their repository. However, the results were suboptimal due to limited instructions on running the code, an issue also noted in the GitHub repository's "Issues" section.

Regarding SeqCare and MedPath, we attempted to implement them within the timeframe of the rebuttal period. Unfortunately, we could not get their GitHub code to function as required, primarily due to missing necessary files.

To address this question and provide additional comparisons, we have incorporated two recent baselines into the sequential diagnosis prediction task. We hope this enhances the comprehensiveness of our evaluation.

[8] Xu, Ran, et al. "Hypergraph transformers for ehr-based clinical predictions." AMIA Summits on Translational Science Proceedings 2023 (2023): 582.

[9] Hadizadeh Moghaddam, Arya, et al. "Contrastive Learning on Medical Intents for Sequential Prescription Recommendation." Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024.

Please refer to Table A1 below.

Table A1: Sequential Diagnosis Prediction

To make the experimental section more comprehensive and extensive, we experiment with two more tasks of mortality prediction and readmission prediction. For each task we used all the baselines used in sequential diagnosis tasks plus more relevant baselines. Table A2 presents the results of these tasks based on the MIMIC-III dataset. Experiments on the MIMIC-IV dataset are underway, and we will share the results once completed.

[1] Nguyen, Phuoc, et al. "Deepr: a convolutional net for medical records (2016)." ArXiv160707519 Cs Stat (2016).

[2] Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling." arXiv preprint arXiv:1803.01271 (2018).

[3] Song, Lihong, et al. "Medical Concept Embedding with Multiple Ontological Representations." IJCAI. Vol. 19. 2019.

[4] Choi, Edward, et al. "Learning the graphical structure of electronic health records with graph convolutional transformer." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 01. 2020.

[5] Gao, Junyi, et al. "Stagenet: Stage-aware neural networks for health risk prediction." Proceedings of The Web Conference 2020. 2020.

[6] Ma, Liantao, et al. "Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 01. 2020.

[7] Zhang, Chaohe, et al. "GRASP: generic framework for health status representation learning based on incorporating knowledge from similar patients." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 1. 2021.

[8] Xu, Ran, et al. "Hypergraph transformers for ehr-based clinical predictions." AMIA Summits on Translational Science Proceedings 2023 (2023): 582.

[9] Hadizadeh Moghaddam, Arya, et al. "Contrastive Learning on Medical Intents for Sequential Prescription Recommendation." Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024.

Table A2: Prediction Performance for Hospital Readmission and Mortality

Thank you for the authors' response. My main concern remains that OntoFAR may rely on powerful GPT embeddings, and its performance without LLM embeddings does not show a statistically significant improvement over the baselines. Nevertheless, I appreciate the authors' efforts in providing additional analyses, and I am willing to raise my score to 6.