Visual and Domain Knowledge for Professional-level Graph-of-Thought Medical Reasoning

We thank reviewer's constructive feedback; we have addressed your concerns below.

Q1. One potential concern is that the paper mentions the dataset is sourced from Massachusetts General Hospital, which might conflict with the anonymity requirements of a double-blind review process. This detail could inadvertently reveal the institution behind the dataset, thereby compromising the anonymity of the submission. Aside from this, the paper is strong in its methodological design and clinical relevance. The combination of long-term data collection, expert annotations, and integrated clinical reasoning within the model is highly commendable. Addressing the anonymity issue explicitly would help mitigate any ethical or procedural concerns.

Thanks for recognizing the value of our dataset and method. We further address the anonymity issue here. While the dataset used in this study originates from Massachusetts General Hospital (MGH) and was collected and shared under appropriate IRB approvals, this does not imply that the research was conducted by or on behalf of MGH, but just provide that we can collect the HIE specific data from one of the best hospitals in the world, with the most professional clinical knowledge of HIE on the world. The use of the MGH dataset was independent of any institutional affiliation and does not reveal the authors' identities.

Moreover, before submission, we have carefully reviewed the manuscript to ensure that any potentially identifying references to the authors' institutions have been removed or anonymized to comply fully with double-blind review requirements. We will restore the appropriate acknowledgments and dataset provenance upon acceptance, as per standard publishing practice.

Q2. Figure 3 could be improved as the current design allows for some overlap between images and text, which hampers clarity. A clearer layout that avoids any overlap will improve both readability and the overall presentation of the results. The authors might consider revising the figure with better spacing or annotations to ensure that all visual elements are distinct and easy to interpret. Enhancing the visual quality here would further support the clarity of the paper's contribution. Overall, this is a minor presentation issue that, once fixed, will polish an otherwise well-structured paper.

We thank the reviewer for the thoughtful suggestion regarding Figure 3. We agree that improving the layout and removing the overlap between images and text will enhance the clarity and readability of the figure. In the revised version, we will update Figure 3 to ensure that all visual elements are clearly separated, with improved spacing and annotations. We appreciate the reviewer’s feedback on this presentation.

Q3. The paper mentions that domain prompts are defined by doctors with varying years of experience, specifically distinguishing between low-experience and high-experience physicians. Could you elaborate on whether this difference in clinical expertise leads to significant performance variations or impacts the model’s outputs? How have you ensured that the variation in clinical experience among the doctors does not bias the dataset or the evaluation of the Clinical Graph of Thoughts (CGoT) model?

The initial annotations were performed by a junior yet experienced fellow and subsequently reviewed through consensus checks with senior experts. This process resulted in a single, unified annotation set. By consolidating annotations through expert agreement and consensus, we aimed to minimize inter-reader variability and reduce potential bias introduced by individual experience levels in both the dataset and the evaluation of the CGoT model. We will include more detailes of the annotation protocol in the revised manuscript to enhance transparency.

We thank reviewer's valuable suggestions and address concerns below within 5000 characters.

Q1. Fig. 1 lacks examples of "clinically irrelevant" questions.

"Clinically irrelevant questions" include general or superficial queries about images, such as modality ("What type of scan is this?") or organ identification ("What organ is shown?"). Although valid in general VQA, these questions lack direct clinical relevance or diagnostic utility.

Q2. How does CGoT avoid hallucination pitfalls like Med-Flemingo?

• Structured Reasoning Pathway: CGoT decomposes tasks into clinically meaningful subtasks aligned with expert workflows, increasing interpretability and traceability.
• Clinical Knowledge Grounding: Reasoning steps explicitly use specialist-curated clinical knowledge (e.g., ADC maps, injury scores), anchoring predictions in verifiable evidence.
• Modular Task and Output Verification: Each subtask undergoes ground-truth validation, reducing hallucinations via fine-grained feedback. Intermediate outputs (injury scores, consistency signals) allowing both users and the model itself to validate predictions and detect inconsistencies early in the pipeline.

Q3. Ablation studies on visual knowledge components (ADC vs. ZADC).

Table 3-1. Ablation study on visual knowledge components

Each visual knowledge type (ADC, ZADC, brain anatomy) has complementary roles. Raw ADC provides signal details, ZADC identifies abnormal regions, and brain anatomy supply anatomical priors. Removing any component degrades performance, but removing ZADC has the crucial effect, as it provides the probability of abnormal brain regions—crucial for MRI injury interpretation.

Q4. Justification and sensitivity of ZADC (–2).

• Justification (z < –2): The threshold of –2 aligns with clinical conventions and prior studies for HIE abnormal regions probabilities [A], indicating ADC values below two standard deviations from the normal atlas—often interpreted as abnormally reduced diffusion in neonatal HIE ADC maps—and serves as a marker for potential brain injury regions.

Table 3-2. Ablation study on varying ZADC threshold values

Across ZADC threshold variations, CGoT outperforms baselines, demonstrating robust and effective performance. The drop at z < −2.2 in MRI injury stems from an overly strict threshold that misses mild injuries. NRN includes mild cases (0 and 1), which are often excluded by this threshold, leading to missed low-grade injuries. z < −2 better captures these signals, yielding optimal performance.

Q5. Alternative reasoning structures.

Clinically Grounded. CGoT is predefined and grounded in well-established diagnostic and prognostic workflows used in neonatal care. Its structure captures a real clinical reasoning pipeline [A,B], where each edge and reasoning step reflects realworld clinical logic. Modifying or removing these edges would compromise both interpretability and clinical validity.

Quantitative Analysis. Table 4 (main paper) demonstrates omitting intermediate tasks significantly reduces neurocognitive outcome prediction accuracy, confirming all subtasks are essential. Alternative structures (e.g., bypassing injury scoring or directly predicting outcomes from lesions) resulted in lower accuracy and clinical implausibility (Table 4). Intermediate reasoning nodes and current reasoning structures are necessary; the model is robust and tolerant to minor errors in these steps, as shown in response to Reviewer #fHCe (Tables 2-1 and 2-2). Future explorations of adaptive structures must maintain clinical interpretability. Ablation studies on node features (Tables 3-1, 3-2) confirm that all types of knowledge inputs contribute meaningfully to performance, reinforcing the thoughtful and necessary design of CGoT.

[A] Mining multi-site clinical data to develop machine learning MRI biomarkers: application to neonatal hypoxic ischemic encephalopathy. 2019.

[B] NICHD Magnetic Resonance Brain Imaging Score in Term Infants With HIE: A Secondary Analysis of a Randomized Clinical Trial.

Q6. Add related evaluation work [1,2].

We will add and discuss the new related work in the later version.

Thanks for the constructive comments. We address your concern below within 5000 characters.

Q1. Computational Complexity

Table 1-1: Case-wise computational analysis on the Gemini-1.5-flash backbone.

CGoT introduces modest overhead compared to Gemini-1.5-Flash but achieves 15% accuracy improvement with clinical transparency. CGoT significantly reduces computational cost compared to SOTA agents (MedAgent, MdAgent) through structured reasoning that mirrors clinical workflows. We will add this new analysis in the later version.

Q2. Dependency on Prerequisite Tasks

Prerequisite tasks before final decisions align with standard clinical practice. Our approach enhances both transparency and effectiveness.

1. Intermediate steps are necessary and model is robust to minor errors of these intermediate steps:

   Intermediate step is necessary:

   Table 2-1: Removing MRI injury scoring from the reasoning chain leads to a sharp performance drop in outcome prediction.

   MRI Injury Score	2-year Outcome
   ✘	50.94
   ✔	71.73

   Robust to small errors of intermediate steps: We applied random ±1-level perturbations (across 4 severity levels) to MRI injury score in varying percentages of cases (0–30%). Results show graceful degradation, indicating robustness to small inaccuracies.

   Table 2-2. CGoT is robust to minor errors of intermediate steps.

   Perturbation Ratio (%)	2-year Outcome (%)
   0	71.73
   10	67.83
   20	66.22
   30	62.72

2. Clinically Grounded: CGoT mimic clinical diagnostic workflows by modeling the stepwise reasoning clinicians use. Its intermediate tasks are clinically meaningful and enhance interpretability. Step-by-step inference in chain-of-thought reasoning [A] and inference scaling [B] has been proved to enhance interpretability and improve performance on other complex reasoning tasks. Table 4 (main paper) shows incorporating such intermediate tasks improves the final outcome prediction compared to relying solely on end-to-end black-box models (Table 2, main paper).

Q3. Comparisons with MDAgent and MedAgent

Table 3-1: Performance comparisons.

CGoT outperforms both baselines by incorporating structured clinical reasoning with domain-specific visual and textual knowledge, enabling superior performance on complex clinical tasks.

Q4. Question Types

Table 4-1: Question distribution.

• Open-ended: Lesion percentage, lesion anatomy, rare lesion localization, MRI interpretation summaries.
• Multiple-choice: Lesion severity ratings, MRI injury scoring, outcome classification.

Q5. Evaluation Metric

(1) ROUGE-L, capturing content overlap and fluency by comparing generated answers to expert references, commonly in medical summarization tasks with LLM [C]. (2) F1 score for questions related to brain regions to assess correctness and completeness of region injured (Sec.5).

Q6. Identify Clinical Knowledge Relevance

Clinical experts (radiologists, neonatologists, and neurologists) curated, identified, and validated all relevant clinical knowledge during dataset construction to align with real-world clinical reasoning.

Q7. Analysis of Components in Table 3 (main paper):

Retaining only GoT or only clinical knowledge performs worse than removing both due to misaligned or incomplete reasoning. GoT without clinical context may propagate irrelevant patterns, while clinical knowledge without GoT ignores task dependencies. When both are removed, the model defaults to a purely end-to-end approach, which avoids the confusion caused by partially constrained reasoning. This highlights the importance of combining knowledge and clinical reasoning.

Q8. Paper Revision

We will carefully review and revise the paper.

[A]. W., J., et al. "Chain-of-thought prompting elicits reasoning in large language models." NeurIPS 2022.

[B]. G., D., et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv 2025.

[C]. T., L., et al. "Evaluating large language models on medical evidence summarization." NPJ digital medicine 6.1 (2023): 158.