Is classification all you need for radiology report generation?

R: We respectfully disagree with this comment.

Comparison from the perspective of motivation: The compared work "Medical Report Generation Is A Multi-label Classification Problem" investigates a specific classification-based approach, reaching conclusions similar to ours. However, this scenario represents only a subset of our study’s scope, as we also consider OOD scenarios. Additionally, we have observed a significant gap between diagnosis performance and report generation performance.

Comparison from the perspective of methodology: Our approach emphasizes high-level design paradigms and investigates how diagnosis influences report generation. We explore four different design paradigms from existing methods, while the compared approach essentially falls under one of our paradigms. Moreover, the compared work focuses more on algorithmic design details, whereas we focus on the broader impact of diagnosis on report generation.

Comparison from the perspective of experiments: We conducted extensive experiments across various classifiers, vision encoders, and LLMs. Our findings reveal that classification impacts report generation in different ways across in-domain, long-tailed, and OOD scenarios. In contrast, the compared work does not consider these experimental settings.

Finally, we are currently unable to conduct a direct comparison with this work because it does not provide reproducible source code. Furthermore, we were unable to locate this article in the IEEE MedAI repository.

We sincerely request the reviewer to re-evaluate our submission to ensure a fair and accurate assessment.

Thank you for your feedback. The method [1] you provided represents an important baseline. However, we believe that methods based on knowledge graphs (KGs) are fundamentally similar to one of our baselines, "Expanding." Both approaches rely on classification and predefined templates to generate the final report. In future work, we will explore further possibilities for improving the "Expanding" baseline, such as fine-tuning. Additionally, our method has been evaluated on a larger dataset compared to [1]. Moreover, our primary focus is on baselines centered on large language model (LLM)-based report generation. The conclusions of [1], while noteworthy, are significantly limited in the context of current technological advancements. Nevertheless, we highly respect these contributions and will consider refining our methodology to integrate these insights.

We respectfully disagree with your comment that Out-of-domain (OOD) case represents an uncommon case. OOD data is far from rare; instead, it is a critical challenge, particularly in deep learning-driven medical imaging analysis. This is because the distribution of medical imaging data can be influenced by various factors, including differences in patient populations, variations in imaging devices and protocols, the emergence of new diseases, and fluctuations in image quality[3-5].

As for the literature you provided [2], it closely aligns with our baseline (V+L) and demonstrates similar or slightly mismatched performance on the same dataset, MIMIC-CXR. For example, the metrics indicate comparable or marginally differing results: BLEU-4 (V+L: 14.6 vs. [2]: 13.4), METEOR (V+L: 32.3 vs. [2]: 16.0), and F1-14 (V+L: 35.7 vs. [2]: 35.8).

[1] Zhang, Yixiao, et al. "When radiology report generation meets knowledge graph." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 07. 2020.

[2] Wang, Zhanyu, et al. "R2gengpt: Radiology report generation with frozen llms." Meta-Radiology 1.3 (2023): 100033.

[3] Mehrtash, Alireza, et al. "Confidence calibration and predictive uncertainty estimation for deep medical image segmentation." IEEE transactions on medical imaging 39.12 (2020): 3868-3878.

[4] Narayanaswamy, Vivek, et al. "Exploring inlier and outlier specification for improved medical ood detection." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Yuan, Mingze, et al. "Devil is in the queries: advancing mask transformers for real-world medical image segmentation and out-of-distribution localization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Inadequate evaluations of classifier: The incorporation of the classifier is inadequately evaluated, making it difficult to understand the settings in which incorporating a classifier is useful, as detailed below. The paper could have benefitted from more nuanced insights on when including the classifier helps vs. detracts from the quality of generated reports.

• Q1a. Classifier Performance: Although the incorporation of the classifier is the key contribution of this paper, the authors do not provide any evaluations on the quality of the actual classifier.

Thank you very much for your suggestion. The prediction accuracy of the classifier varies across different observations. Below, we present the F1 scores for Swin Transformer-Large on various observations in the MIMIC dataset. We analyzed the accuracy of the generated reports for different observations on MIMIC-CXR and reported the results separately for the V+L and C+V+L settings. These results indicate that incorporating classification information can, to some extent, improve the quality of the reports. For out-of-distribution (OOD) scenarios, we argue that integrating classification information helps large language models (LLMs) improve the performance of generated reports on classification metrics. However, when the classifier's performance is poor, the improvement is limited. Nevertheless, this additional information does not degrade the quality of the original reports, which distinguishes this scenario from the in-domain setting. We believe that if training is conducted directly on PadChest or IU X-ray, the conclusions would align with those observed in the in-domain setting. We will focus on a detailed analysis of this aspect in the revision.

The F1-score of different observations on MIMIC dataset

• Q1b. Upper Bound: It would be useful to establish an upper bound on performance of the report generation model by utilizing an “oracle” classifier (i.e. a hypothetical classifier that predicts every condition correctly). This would establish whether classification, in the optimal setting, makes useful contributions to the report generation task. As of now, the evaluations are limited by the poor performance of the evaluated classifier, and the key findings/takeaways from the paper are based on this specific classifier.

Below, we present the results on the MIMIC-CXR dataset using ground-truth classification as input to the LLM.

The experimental results demonstrate that our trained classifier outperforms state-of-the-art classifiers on binary classification tasks. Furthermore, integrating classification results into our baseline yields consistent conclusions. However, a noticeable gap remains between the upper bounds of report generation and binary classification, even when classification information is incorporated.

We believe this gap arises from the inherent limitations of LLMs in handling long sequences. The classification tokens are relatively few, whereas the combined token count of the X-ray image and report far exceeds that of the classification tokens. Addressing this gap remains a challenging problem, but it falls outside the scope of our current paper.

• Q2: Inadequate evaluations of proposed metrics: Although several new metrics are presented as key contributions of this paper, the quality of these metrics is not evaluated. How accurate is the OpenAI GPT-4o model at extracting disease categories? Did the authors check for false negatives / positives in this extraction procedure? Does this metric align with ground-truth labels for the conditions where labels are provided? Evaluating the quality of the metrics via a dataset like ReXVal would have been useful [1].

We agree with your perspective that the validity of evaluation metrics has a significant impact on our conclusions. To address this, we have already validated the results of GPT-4o. Specifically, we randomly sampled 200 reports and conducted a detailed comparison. Our findings indicate that GPT-4o performs comparably to the ground truth in extracting long-tail disease observations. In future work, we plan to conduct more rigorous validation of all evaluation metrics to ensure their reliability and accuracy.

• Q1: The comparison between the four baseline models lacks analysis. The metrics give inconsistent results, how could the author get the results that C+V+LLM is better than V+LLM, and the comparison between Refining and C+V+LLM, giving statistical significance analysis of the results would be helpful.

R: Thanks for your suggestion. Our experimental results (Tables 1 and 2) demonstrate that the classifier contributes to improving the classification metrics of the generated reports. However, there is no evidence to suggest that it completely surpasses the V+LLM setting. In our Finding 1, we explicitly state that the classifier enhances diagnosis performance. As for the Refining baseline, it refines the report’s disease descriptions based on the classification results. This approach ensures the original textual quality is preserved while making minimal modifications to improve diagnosis performance. We will provide a more detailed analysis of these baselines in the revision.

• Q2: The poor performance of the 'Expanding' approach may be due to the prompt rather than inherent limitations. I do not find details about the prompts used (apologize if I overlooked them). The 'Expanding' prompt should include standard medical report templates to prove a fair comparison.

R: We completely agree with your viewpoint. To minimize the impact of prompts on our experimental conclusions, we adhered to the principle of simplicity in design and employed a comprehensive comparison approach. The principle of simplicity refers to designing all prompts with the simplest possible instructions. For example, for the Expanding baseline, we used the instruction: "Below are the classification results from a chest X-ray classifier. Your task is to expand these results into a segment of clinical findings." If in-context learning (ICL) is used, we provide the LLM with a few report templates as examples. Additionally, to ensure fairness, we applied the same prompt design for evaluating all LLMs in the study.

• Q3: The classifier only provides binary disease presence information without crucial details like location and severity. This limitation makes hallucinations in LLM-generated reports inevitable when using classification-only inputs. Instead of comparing with these inherently limited training-free approaches, the paper would be more valuable if it explored different variants of end-to-end Vision + LLM full training.

R: We fully agree with your perspective. In the Limitations part, we discuss how incorporating additional information might help LLMs reduce hallucinations—a point supported by findings in related work. However, the scope of this paper focuses specifically on examining the impact of classifiers on report generation. This focus is justified by the fact that classification information is more readily accessible compared to other types of information. Due to computational resource constraints, we were only able to explore common end-to-end baselines. In Figure 2, we illustrate the influence of different vision encoders on the diagnosis performance of report generation. We hope that our preliminary conclusions will inspire the community to further explore this topic.

• Q1: In addition to case studies, it would be helpful to include quantitative measures of the extent of hallucinations in the OOD experiments.

R: Thank you for your valuable feedback. In fact, the proposed LLM-Radjudge metric can partially reflect the degree of hallucinations in the generated reports. This metric is described on page 5, lines 228–230 of our submission. The four levels of the metric we reported are specifically designed to quantify the extent of hallucinations.

• Q2: To reduce overreliance on classification outputs in particular, it might be more helpful to experiment with controlling the weight of the classifier or including other types of information (e.g. location, severity). However, this was mentioned as part of the limitations.

R: Absolutely, the weight of the classifier, as well as more complex information (e.g., location and severity), significantly impacts the quality of the generated reports. We have mentioned these factors in the limitations. To address this issue, we conducted preliminary experiments by adjusting the value of classification tokens in the attention matrix. The following table summarizes the results:

Here, lambda represents the scaling factor applied to the original value of classification tokens in the attention matrix. These results demonstrate that the classifier can effectively contribute to report generation to a certain extent. Moreover, previous studies [1-3] have also shown that incorporating additional information can improve report quality and mitigate hallucinations to some degree.

[1] Chatcad: Interactive computer-aided diagnosis on medical image using large language models [2] Chatcad+: Towards a universal and reliable interactive cad using llms [3] MAIRA-2: Grounded Radiology Report Generation

• Q3: Optional suggestion: have you looked into also using the GREEN metric? https://arxiv.org/pdf/2405.03595.

R: Thank you for bringing this to our attention. After carefully reviewing the GREEN metric, we recognize its potential as a measure for quantifying errors in generated reports. Its objective aligns closely with LLM-Radjudge metric. We believe that incorporating GREEN could provide a more comprehensive evaluation in future work, and we plan to discuss this metric in our revision.

• Q4: Minor edit: "purple text" on page 9 appears to be red instead of purple - it might be helpful to use a different shade.

R: Thank you. We have corrected this issue in the revision.