Zodiac: A Cardiologist-Level LLM Framework for Multi-Agent Diagnostics

1. In section 4.1, it is mentioned that data from 2000+ patients from collaborating healthcare institutions are used, and 5% of the data is used for evaluation. Given that the evaluation set is in the same distribution with the data used for fine-tuning the LLM agents, I am wondering whether the framework could generalize to a broader patient population, e.g. patients in other hospitals/medical centers, or patients from very different geographic locations or demographics (e.g. different race or ethnicity)?
2. In section 5.1 and table 2, it seems that the numeric results of the eight metrics including accuracy, completeness etc., are aggregated across the ratings from different cardiologists. Could there be any calibration bias across the cardiologists, e.g., some people are much stricter raters and some are more lenient? If that is the case, I think ranking-based metrics are more suitable than directly averaging the ratings across cardiologists.

Text:

• typo in figure 7b signle
• line 375: remove "is": Instead of using public benchmarks, we adopt real patient data is to align with practical diagnostics

General:

• Representative Groups: race is not indicated as statistic.
• Human validation: how many people have provided a report? Was the data that Zodiac was pre-trained on from the same institution as the physicians? This might explain why clinical-specialist LLM's performed poorly (overfitted to a certain dataset).
• Line 375: Dataset: Instead of using public benchmarks, we adopt real patient data is to align with practical diagnostics. Is there a reason you could not do both? Your own benchmark might have some biases as you could have developed your model to perform especially well on the metrics that you defined.
• I am missing a limitations section in your work.
• While I understand that evaluation of these frameworks is far from trivial and physician evaluation is an important part of the solution, more details need to be provided for these experiments. Moreover, public benchmarks should be added. If current benchmarks are imperfect, as mentioned in line 247, a small public benchmark can be provided for future work to be able to compare to your work.
• I urge you to provide code/configurations for as much of your work as possible. This is important, especially to convince the non-health ML field of your work.

While the proposed method, ZODIAC, addresses a well-defined clinical problem of generating ECG reports and integrates valuable clinical insights through multi-agent collaboration, concerns remain regarding the rigor of its evaluation.

1. Evaluation Metric: The metrics defined in Table 1 are well-defined for qualitative evaluation, but appear subject to evaluator variability, raising concerns about the objectivity and rigor of its scale in Table 2. The following points are regarding the rigor of this evaluation method which would be expected to be improved in the following submission of the paper.

• Subjectivity of the Metric: The degree of subjectivity within the metrics may not be fully addressed. For example, what are the precise criteria for defining hallucination and bias within the FFH metric? Furthermore, the definition of bias is unclear regarding which 'characteristics' of the patient are considered. Does this refer to patient demographics, clinical features, or another criterion?

• Calibration of the Metric: There are also concerns about whether this metric can serve as a reliable, single quantitative standard for evaluating model outputs. A rigorous calibration process, such as Inter-Rater Reliability (IRR), would help substantiate the metric’s consistency and could address potential variability. This is particularly important because different evaluators may interpret the scale differently—for instance, what a score of 3 represents could vary among physicians.

• Details on Inter-Rater Reliability (IRR): The paper leaves IRR and confidence intervals unexplored, which could suggest inconsistencies in clinical outputs. Addressing IRR would help to ensure that ratings are stable and comparable across evaluators. There are standard deviations reported in Table 2, but it is better to separately report inter-rater confidence intervals as well.

2. Ablation Studies on $\theta_{M2F}$ only or $\theta_{T2F}$ only needed to see the effect of Triple agent vs. Double agents and the effect of leveraging metadata and ECG tracings.

3. There are more recent works on multi-agent collaboration or interaction, which could be included in the related works section of the paper.

[1] Kim, Y., Park, C., Jeong, H., Chan, Y. S., Xu, X., McDuff, D., ... & Park, H. W. (2024). Adaptive Collaboration Strategy for LLMs in Medical Decision Making. arXiv preprint arXiv:2404.15155.

[2] Jin, Q., Wang, Z., Yang, Y., Zhu, Q., Wright, D., Huang, T., … & Lu, Z. (2024). AgentMD: Empowering Language Agents for Risk Prediction with Large-Scale Clinical Tool Learning.

[3] Li, J., Wang, S., Zhang, M., Li, W., Lai, Y., Kang, X., ... & Liu, Y. (2024). Agent hospital: A simulacrum of hospital with evolvable medical agents.

[4] Fan, Z., Tang, J., Chen, W., Wang, S., Wei, Z., Xi, J., ... & Zhou, J. (2024). Ai hospital: Interactive evaluation and collaboration of llms as intern doctors for clinical diagnosis.

[5] Yan, W., Liu, H., Wu, T., Chen, Q., Wang, W., Chai, H., ... & Zhu, L. (2024). ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real World.

• While ZODIAC effectively uses a multi-agent system, the framework is somewhat derivative of existing LLM-based multi-agent approaches. A more comprehensive comparison of this framework against existing multi-agent approaches in healthcare [1,2] is needed to demonstrate the novelty and effectiveness of the proposed framework.

• The experiments are not convinced.

  • The paper relies on a small validation set (only 5% of the dataset, ~100 patients). It raises concerns about the robustness of the framework. The size is far from sufficient to draw solid conclusions about its effectiveness across a broader population or diverse clinical scenarios.
  • All experiments are conducted on single-sourced private data of a limited size (2000+ patients). Though the authors aimed to align real clinical applications by using their own data, there are a bunch of publicly available, widely adopted real-world ECG datasets (e.g., PTB-XL [3], MIMIC-IV-ECG [4], CODE [5]) that can be utilized as additional evaluation sets.
  • There is a lack of out-of-domain evaluation for demonstrating the generalizability of the framework. Both the training data and evaluation data are from the same data source.

• Clinical guidelines are used for fact-checking to generate final interpretations. However, there is insufficient information on how these guidelines are sourced or managed. There are no details on the selection criteria for clinical guidelines, their integration within the framework, or the mechanisms employed to handle potential conflicts arising from diverse guidelines.

• While the ZODIAC framework leverages a multi-agent setup to mirror cardiological diagnostic processes, the paper does not fully describe how they address potential redundancies or conflicts in findings across different modalities (e.g., ECG and tabular data).

• While the tracing-to-finding agent is intended to interpret ECG images, it remains unclear how crucial this agent is in identifying visual patterns independently, given the availability of comprehensive cardiologist-adjudicated text and clinical guidelines within the framework. This raises the question of whether the model is truly leveraging visual information from ECGs or primarily relying on textual data.

[1] Tang, Xiangru, et al. "Medagents: Large language models as collaborators for zero-shot medical reasoning." arXiv preprint arXiv:2311.10537 (2023).

[2] Bani-Harouni, David, Nassir Navab, and Matthias Keicher. "MAGDA: Multi-agent guideline-driven diagnostic assistance." International Workshop on Foundation Models for General Medical AI. Cham: Springer Nature Switzerland, 2024.

[3] Wagner, Patrick, et al. "PTB-XL, a large publicly available electrocardiography dataset." Scientific data 7.1 (2020): 1-15.

[4] Gow, B., et al. "MIMIC-IV-ECG: diagnostic electrocardiogram matched subset (version 1.0)." PhysioNet (2023).

[5] Ribeiro, Antônio H., et al. "Automatic diagnosis of the 12-lead ECG using a deep neural network." Nature communications 11.1 (2020): 1760.