Radiologist-like Progressive Radiology Report Generation and Benchmarking

Q3: Please further highlight the technical contribution of the proposed RLRG framework, such as differences compared to existing multi-stage generation or MLLMs.

A: As mentioned in Paragraph 3 of Related Works, there are two multi-stage generation frameworks, e.g., MedVersa and CheXagent. Both works do not leverage the indication as input during training. MedVersa takes the X-ray image as sole input and uses ChatGPT-generated template prompts to generate findings-only, impression-only, and complete reports (findings and impression). However, the authors of MedVersa have not granted us access to its pre-trained weight, which prevents us from evaluating their model. As for CheXagent, we have reported its results in Table 4.

Other MLLM-based methods, such as MAIRA-1 and MAIRA-2 (closed-source), use the indication as input but only generate the findings section, which leaves the clinical query in the indication unanswered.

To conclude, what differentiates our framework from previous works is that we redefine the paradigm of automated radiology report generation (RRG) from a clinical perspective, which narrows the gap between machine learning models and real-world bedside practice. Our training paradigm aims to verify the relationship among the indication, findings, and impression sections and how leveraging this relationship can benefit model performance.

Q4: From an application of MLLM’s perspective, there is not much difference between indication + image → finding & impression and indication + image → finding → impression, as both findings and impressions are important and should be generated as medical reports. Please further explain the need to split the findings and impressions.

A: Splitting the findings and the impression is necessary because generating the findings and generating the impression are inherently distinct tasks. From a clinical perspective, the findings section captures factual observations from the X-ray image, necessitating that the model focuses on learning accurate image-text alignment. In contrast, the impression section interprets the findings to address the clinical query posed in the indication, making it heavily reliant on the reasoning capabilities of the LLM. For instance, in Figure 1 (a), the impression states, Findings suggestive of heart failure which could be secondary to volume overload, directly addressing the clinical query Eval for volume overload in the indication. Specifically, the findings mention pulmonary edema and mild cardiomegaly, which are manifestations of heart failure. Thus, the impression is derived from reasoning over the findings.

Attempting to generate both sections in a single step risks mixing these two distinct tasks, potentially reducing the efficiency of the training process. Moreover, separating the generation of findings and impressions ensures a more straightforward, structured final report with well-defined sections. As shown in Table 4, our method outperforms other LLM-based methods by a large margin .

Dear Reviewer t2eU,

Thank you for your detailed and constructive feedback on our work. We appreciate your insights into both the strengths and weaknesses of our approach, and we address each of your points below:

Weaknesses

Q1: Originality and technical novelty: The proposed Radiologist-Like Progressive Generation (RLRG) is an MLLM-based 2-stage framework that feeds the output of the first stage as input of the second stage. Therefore, the overall design is not technically novel. Further, the neural network architecture is similar to existing radiology report generation methods such as R2Gen; the proposed dataset is modified from the existing publicly accessible dataset.

A: The main contribution of our work is to redefine the paradigm of automated radiology report generation by revealing the relationship among the three sections in a report and how leveraging this relationship can benefit model training.

Previous works like R2GenGPT treat radiology report generation as an image captioning task, taking X-ray images as the sole input to generate the findings and impression in a single step. However, this approach overlooks a critical aspect: the impression is an interpretation of the findings in the context of the clinical query provided in the indication, which is hard to derive from the image alone.

Other works, such as MAIRA-1 and MAIRA-2, incorporate the indication and image as inputs but generate only the findings section. This deviates from real-world clinical practice, as it leaves the clinical query in the indication unanswered. In contrast, our work addresses these limitations by offering a practical framework that generates both the findings and impression sections guided by the indication to provide meaningful clinical insights.

The second contribution of our work is the creation of a well-structured and cleaned dataset. As mentioned in the Introduction, the raw reports in MIMIC-CXR are in free-text form with sections that are not delineated. Our proposed version organizes the data into a clean and structured format, significantly enhancing its usability for developing radiology report generation methods with practical clinical applicability.

Q2: Clarity: The authors concentrated on describing the 2-stage report generation algorithm, while the details of curating the MIMIC-1V3 dataset are not clearly stated. As the contribution of this paper heavily relies on the dataset creation, it is better to clearly address questions such as “How to handle reports without impression/finding/induction?”, “what algorithm was used to separate the report?” and “How to evaluate the correctness of the split?”.

A: Section 4.1 explains the details of our three-fold effort to curate the MIMIC-1V3 dataset. The process involves heavy manual work to ensure the cleanness and structure of each report, such as rectifying misassigned sections.

For preliminary report parsing, in line 113 of the Introduction, we mentioned that we adopted the MIMIC-CXR official code base (https://github.com/MIT-LCP/mimic-cxr). After automated parsing, we only retain the reports with all three sections. Then, we manually acquire a list of patterned sentences misassigned as part of the findings (these sentences mainly belong to the Technique and Comparison sections). Lastly, we use a simple string match algorithm to remove these patterned sentences from all reports.

Regarding the split, as stated in paragraph 2 of Section 4.2, we use the official split file of MIMIC-CXR and make sure our training samples come only from the original training set and so on.

My concerns still remain after the authors' rebuttal.
First, I am not convinced that the proposed method is technically novel. Technically, the proposed framework is highly similar to previous works, except that the previous ones did not leverage/get access to the newly created dataset.
Second, I think the contribution of this study is insufficient for a top-tier conference like ICLR. The key modification is to re-divide the existing dataset with different section settings, which is below the acceptance threshold despite the fact that it comes with good motivation.
Therefore, I will maintain the original rating.

Dear Reviewer b8jm,

Thank you for your detailed and constructive feedback on our work. We appreciate your insights into both the strengths and weaknesses of our approach, and we address each of your points below:

Weaknesses

Q1: The key experiment is missing in Table 3: Given the IMG and IND, generate the FIN and IMPR. This is critical because this is the authors' main claim: Separating the generation of each section is superior to the joint generation. Especially since in Table 2, the FIN only generation is inferior to the joint generation of FIN and IMPR on the test set.

A: The table below shows the result of the experiment in which the model takes the image and the indication as input and generates the findings and the impression in one step (in the header, FIN indicates the evaluation results for findings,while IMPR for impression). Our approach still outperforms the one-step approach across all evaluation metrics on both findings and impression generation. This is because radiology report generation inherently consists of two distinct tasks: 1. findings generation (captioning): describe the factual observations of the radiograph that are relevant to the indication. 2. impression generation (reasoning): interpret the clinical implications of the findings and address the clinical query in the indication. Generating the findings and the impression in one step risks conflating these two tasks, which can lead to less accurate and coherent reports. By separating the generation process, our approach ensures that the factual observations are clearly and accurately detailed in the findings section, while the impression section provides a thoughtful and contextually relevant interpretation of these findings. This division not only enhances the clarity and reliability of the radiology reports but also aligns more closely with clinical workflows, where distinct attention is given to observations and their clinical implications. Consequently, our two-step method achieves superior performance across all evaluation metrics, demonstrating its effectiveness in producing high-quality, clinically useful radiology reports.

In Table 2, the FIN-only generation achieved better scores on Natural Language Generation (NLG) metrics than the joint generation approach on the test set, and scores of the Clinical Efficacy (CE) are comparable. However, on the validation set, separated generation outperforms joint generation in both sections across all evaluation metrics. One potential reason is that checkpoint saving is only based on the scores of NLG metrics on the validation set during training. As we state in Section 4.2, there is a notable distribution shift between the validation and test sets, which could result in unstable results on the CE metrics.

Q2: More fair comparisons to other methods with IND section provided.

A: We appreciate your concerns regarding the fairness of the comparison. The indication is not external prior knowledge but an inherent part of a radiology report[1][2], and previous works fail to explore the connection among three sections. While MAIRA-1&2 work with the indication, they are closed-source, and only generate the findings without fully utilizing the indication, leaving clinical questions unanswered. In our original Table 4, we compared their results strictly under their original settings.

To ensure a more fair comparison, we add the indication as input for other models in the following Table. Our approach still outperforms these LLM-based methods even adding the indication. For R2GenGPT, we add the indication to its original prompt and retrain the model. For the findings, we observe reduced scores for all evaluation metrics except the RE-NLI, which increases from 0.2363 to 0.2398. For the impression, adding indication can improve the model's performances on Bert-Score (0.3277 → 0.3678), RE-EM (0.1539 → 0.2010), and Rad-Complete (0.0869 → 0.1101), while negatively impact the performance on B-4 (0.0532 → 0.0441), CIDEr (0.4530 → 0.2895), F1-all (0.3875 → 0.3543), and RE-NLI (0.1657 → 0.1397).

Directly adding the indication without crafting R2GenGPT's prompt increased the complexity and provided no explicit instructions for the model on how to utilize the extra information, which could explain the performance drops. Nevertheless, the clinical information in the indication is more relevant to the impression section, and this information likely contributed to the improvements in certain scores.

For RadFM and CheXagent, we add the indication into their prompts and conduct inference on the MIMIC-1V3 test set. Adding the indication enhanced RadFM's performance in both sections, with increased scores on all evaluation metrics except for CIDEr and RE-NLI. RadFM's extensive pre-training enables the model to comprehend and utilize the information in the indication, highlighting the importance of the indication. However, since we performed inference without additional training, it is difficult to explain the observed decreases in the CIDEr and RE-NLI scores.

In contrast, for CheXagent, adding the indication led to a significant decline in finding generation performance. Conversely, including the indication improved impression generation performance, with all evaluation metrics showing enhancements. We cannot perform a detailed analysis since CheXagent has not open-sourced its code and training data. Therefore, we can only conclude that including the indication benefits impression generation. This finding aligns with our conclusion that the indication and the impression are closely related.

[1]. Evaluating the referring physician’s clinical history and indication as a means for communicating chronic conditions that are pertinent at the point of radiologic interpretation. J Digit Imaging 2015

[2]. Good practice for radiological reporting. Guidelines from the European Society of Radiology (ESR)

Q3: RLPG sounds very much like a Chain-of-Thought (CoT) multi-stage approach. Comparing the approach to single-stage CoT could reveal interesting insights in how once could better guide the model in a single stage manner.

A: We design the following prompt for one-stage CoT training.

Prompt: Given the patient's indication: <indication>, generate a findings section, which is the factual observation of the chest x-ray image. Then based on the indication and the generated findings, generate a diagnostic impression section which answers the clinical question in the indication.

The result is shown below. The drop in performance could be attributed to the increased complexity of the prompt, which introduces multiple sequential tasks that may overwhelm the model's processing capabilities. By requiring the generation of both a findings section and a diagnostic impression in a single prompt, the model faces a higher cognitive load, potentially leading to confusion or errors in task execution.

Q4: Will the dataset and training code be made publicly available? What is the reproducibility statement of the authors?

A: In section 5.1, we have provided adequate implementation details. We will release the source code and MIMIC-1V3 dataset upon acceptance.

Q3: Comparisons in Table 4 may not be entirely fair, as the proposed model leverages the indication as prior knowledge in report generation.

A: We appreciate your concerns regarding the fairness of the comparison. However, the indication is not external prior knowledge but an existing part of the MIMIC-CXR dataset. Moreover, the indication is a universal and critical component of the radiology report [1][2]. Previous works overlook the importance of indication and do not explore the connection among three sections during training. In our original Table 4, we compared their results strictly under their original settings.

To address the concern and ensure a more fair comparison, we add the indication as input for other models in the following Table (FIN indicates the evaluation results are for Findings, while IMPR for impression). Our approach still outperforms these LLM-based methods even adding the indication. For R2GenGPT, we add the indication to its original prompt and retrain the model. For the findings section, we observe reduced scores for all evaluation metrics except the RE-NLI, which increases from 0.2363 to 0.2398. For the impression section, adding indication as input can improve the model's performances on Bert-Score (0.3277 → 0.3678), RE-EM (0.1539 → 0.2010), and Rad-Complete (0.0869 → 0.1101), while negatively impact the performance on B-4 (0.0532 → 0.0441), CIDEr (0.4530 → 0.2895), F1-all (0.3875 → 0.3543), and RE-NLI (0.1657 → 0.1397). Directly adding the indication without crafting R2GenGPT's prompt increased the complexity and provided no explicit instructions for the model on how to utilize the extra information, which could explain the performance drops. Nevertheless, the clinical information in the indication is more relevant to the impression section, and this additional information likely contributed to the improvements in certain scores.

For RadFM and CheXagent, we added the indication into their prompts and conduct inference on the MIMIC-1V3 test set. Adding the indication enhanced RadFM's performance in generating both sections, with increased scores observed across all evaluation metrics except for CIDEr and RE-NLI in both sections. RadFM's extensive pre-training enables the model to comprehend and effectively utilize the information in the indication, highlighting the importance of the indication from a different perspective. However, since we performed inference without additional training, it is difficult to explain the decreases in the CIDEr and RE-NLI scores.

In contrast, for CheXagent, adding the indication led to a significant decline in finding generation performance across all evaluation metrics; conversely, including the indication improved impression generation performance, with all evaluation metrics showing enhancements. We cannot perform a detailed analysis since CheXagent has not open-sourced its code and training data. Therefore, we can only conclude that including the indication benefits impression generation. This finding aligns with our conclusion that the indication and the impression are closely related.

Q4: Does this approach assume a close relationship between indication and final diagnosis? Have the authors examined how indications relate to impressions or findings in the MIMIC-CXR dataset? Additionally, how does this approach perform if provided with an incorrect indication as input?

A: Our approach is rooted in the close relationship between the indication, the findings, and the impression. The relationship has been validated by [1,2,3].

The indication contains the patient's brief medical history and a clinical question. The findings are the factual observation of the image. The impression section contains radiologists' diagnoses followed by the most supportive findings as evidence for making such a conclusion. These diagnoses are inferred from the findings based on the information in the indication. An example from the MIMIC-CXR dataset was shown in Figure 1 (a), where the impression states, Findings suggestive of heart failure which could be secondary to volume overload, which directly addresses the clinical query Eval for volume overload in the indication. To be more specific, the findings mentioned pulmonary edema and mild cardiomegaly, which are considered the manifestations of heart failure. In other words, heart failure is inferred from the findings.

In real-world scenarios, providing an incorrect indication as input is considered malpractice, as mislabeling patient information could lead to serious diagnostic errors. Therefore, we do not see the necessity of evaluating the approach under such conditions, as this work focuses on improving performance with accurate and reliable inputs aligned with clinical practice.

[3]. How to create a great radiology report. RadioGraphics, 2020.

Q5: To ensure fair comparisons in Table 4, could the authors evaluate their model without using indication as input?

A: As described in Q3, we have discussed incorporating the indication as input for other compared models. In Table 2, we have reported the results of our method without the indication as input. Compared to other methods in Table 4, without the indication as input, the impression section generated by our approach achieves the best scores across most evaluation metrics except for CIDEr and RE-NLI, which are only lower than CheXagent (0.6060 vs 0.6147 for CIDEr and 0.1686 vs 0.1753 for RE-NLI), and the findings section generated by our model achieves the best scores on Natural Language Generation (NLG) metrics and comparable clinical efficacy (CE) scores.

Q6: Alternatively, if using indication as input, they could first generate the indication and report its intermediate results.

A: We believe it is unnecessary to generate the indication if it is already provided as input, as this would introduce redundancy rather than contribute to the overall process. Additionally, indications are typically supplied by the ordering physician and are not generated or written by radiologists. Therefore, generating indications would not align with the clinical workflow.

Dear Reviewer XTQR,

Thank you for your detailed and constructive feedback on our work. We appreciate your insights into both the strengths and weaknesses of our approach, and we address each of your points below:

Weaknesses

Q1: The proposed framework may not fully align with real-world medical practices. Although MIMIC-CXR reports follow the order of indication, findings, and impression, clinicians may not always document in this structured sequence, which raises questions about the practical motivation for this approach.

A: We appreciate your concerns about the generalizability of our proposed framework. However, the indication, the findings, and the impression are essential for a radiology report. [1] states that indications provided with a radiological examination are critical components of a quality interpretation by the radiologist. [2] lists clinical referrals (which have the same function as indications, with different naming conventions), findings, and impression sections as elements of a report. The importance of these sections underscores the need for a framework that captures their distinct roles.

Besides, although the raw reports in MIMIC-CXR have section keywords, without closely examining the functions of each section, the community tends to treat radiology report generation as an image captioning task. The free-text form of raw reports may exacerbate this misconception.

Realizing the limitations of previous work, our work's practical motivation is to redefine the paradigm of automated radiology report generation from one-step image captioning to a two-step captioning-then-reasoning process. To facilitate this paradigm transition, our proposed dataset lays the foundation for bringing the radiology report generation task back to a state that closely aligns with medical practice.

[1]. Evaluating the referring physician’s clinical history and indication as a means for communicating chronic conditions that are pertinent at the point of radiologic interpretation. J Digit Imaging 2015

[2]. Good practice for radiological reporting. Guidelines from the European Society of Radiology (ESR)

Q2: The study lacks a detailed pipeline error analysis to assess how intermediate stages impact final output quality. How does the clinical accuracy of findings influence the generation of the impression section?

A: As is shown in Table 3, after replacing generated findings with ground-truth findings as input for impression generation, we observed elevated scores across all evaluation metrics. To mitigate the impact, we specifically include X-ray images as part of the inputs for impression generation, as image features could compensate for the missing or inaccurate information in the intermediate stage.

We also conduct experiments that exclude X-ray images as input for impression generation. Since the inputs are pure text, we use Lora fine-tuning to add trainable parameters to the frozen LLM. The table below shows that replacing generated findings with ground-truth findings can improve the quality of impression generation for both methods. However, the magnitude of the improvement for different methods varies.

We calculate the relative improvements after using ground truth findings as input, and for easy comparison, we calculate the mean of five clinical efficacy metrics. Without image features, ground truth findings increase the B-4 score from 0.0383 to 0.1428, an improvement of 0.1045. This represents a 273% increase relative to the original score of 0.0383. The increases in CIDEr score and average clinical efficacy (CE) score relative to their original scores are 228% and 68%, respectively. When including images as input, ground truth findings increase the B-4 score from 0.0684 to 0.2074, an improvement of 0.1426. This represents a 220% increase relative to the original score of 0.0684. The increases in CIDEr score and average CE score relative to their original scores are 121% and 55%, respectively. These results prove that including images as input can increase the robustness of our model, making it less sensitive to the quality of the findings.

Dear Reviewer Y732,

Thank you for your detailed and constructive feedback on our work. We appreciate your insights into both the strengths and weaknesses of our approach, and we address each of your points below:

Weakness

Q1: The RLPG framework seems straightforward and intuitive, especially in its logical progression and handling of radiology reports. Moreover, aside from experimental metrics, there seems to be insufficient evidence demonstrating the innovation and superiority of this training paradigm.

A: Innovation: Although our framework seems straightforward and intuitive, our breakthrough is to redefine the paradigm of automated radiology report generation (RRG) from a clinical perspective. Based on the established medical literature[1][2][3], we point out that the creation of a radiology report consists of two distinct tasks, e.g., 1. findings generation (captioning): describe the factual observations of the radiograph that are relevant to the indication. 2. impression generation (reasoning): interpret the clinical implications of the findings and address the clinical query in the indication. In contrast, previous works mainly treat RRG as a simple image captioning task, with no involvement or insufficient usage of the indication. Therefore, our training paradigm is not only closely aligned with real-world clinical practice but also the first work to use the indication to generate the findings and impression sections.

Moreover, to better utilize the relationship among the three sections, we create a well-structured and cleaned dataset, MIMIC-1V3. As mentioned in the Introduction, the raw reports in MIMIC-CXR are in free-text form with sections that are not delineated. Our proposed version organizes the data into a clean and structured format, significantly enhancing its usability for developing radiology report generation methods with practical clinical applicability.

[1]. Evaluating the referring physician’s clinical history and indication as a means for communicating chronic conditions that are pertinent at the point of radiologic interpretation. J Digit Imaging 2015

[2]. Good practice for radiological reporting. Guidelines from the European Society of Radiology (ESR)

[3]. How to create a great radiology report. RadioGraphics, 2020.

Performance: To better demonstrate the superiority of our training paradigm, we have updated the Appendix section with generated reports of our framework and other LLM-based models. Please refer to section A.3 for more details.

Question

Q2: If the findings generated in the first stage contain inaccuracies, how does this affect the quality of the impression?

A: Inaccurate findings would negatively impact the impression quality. This trend has been revealed in our experiments. As shown in Table 3, after replacing generated findings with ground-truth findings as input for impression generation, we observed elevated scores across all evaluation metrics. Given the inaccuracies that occurred in the first stage, our results still outperform end-to-end training and other LLM-based methods.

Q3: Is there a mechanism to mitigate potential error propagation between stages?

A: Yes, we incorporate X-ray images as inputs for the second-stage impression generation to reduce potential error propagation between the two stages. Findings are the factual observation, or in simpler terms, the caption, of the X-ray image. Including X-ray images as input should compensate for the missing or inaccurate information in the findings. To demonstrate the quantitative effect of this mechanism, we also conduct experiments that exclude X-ray images as input for impression generation. Since the inputs are pure text, we use Lora fine-tuning to add trainable parameters to the frozen LLM. The table below shows that replacing generated findings with ground-truth findings can improve the quality of impression generation for both methods. However, the magnitude of the improvement for different methods varies.

We calculate the relative improvements after using ground truth findings as input, and for easy comparison, we calculate the mean of five clinical efficacy metrics. Without image features, ground truth findings can increase the B-4 score of the impression from 0.0383 to 0.1428, an improvement of 0.1045. This represents a 273% increase relative to the original score of 0.0383. The increases in CIDEr score and average clinical efficacy (CE) score of the impression relative to their original scores are 228% and 68%, respectively. When including images as input, ground truth findings increase the B-4 score of the impression from 0.0684 to 0.2074, an improvement of 0.1426. This represents a 220% increase relative to the original score of 0.0684. The increases in CIDEr score and average CE score of the impression relative to their original scores are 121% and 55%, respectively. These results prove that including images as input for impression generation can increase the robustness of the model, and image features can compensate for the missing or inaccurate information in the findings.

Q4: Is there often overlapping information between the findings and impression sections? If so, does the model tend to produce redundant content, and could this result in outputs similar to previous frameworks?

A: Yes, sometimes there is overlapping information between the findings and impression sections. But it is not redundant. From a clinical perspective, the impression section contains radiologists' diagnoses followed by the most supportive findings as evidence for making such a conclusion. These diagnoses are inferred from the findings based on the indication. For example, in Figure 1 (a), the impression states, Findings suggestive of heart failure which could be secondary to volume overload, directly addresses the indication's clinical query, Eval for volume overload. To be more specific, the findings mentioned pulmonary edema and mild cardiomegaly, which are considered the manifestations of heart failure. In other words, heart failure is inferred from the findings. To conclude, the overlapping part serves different purposes, i.e., the factual observation in the findings and the rationale for making diagnoses in the impression.

The outputs of our model are certainly not similar to previous works for two reasons. First, previous works mainly combine the findings and the impression as the training target, which makes distinguishing the findings from the impression in the outputs challenging. In contrast, stage one of our model only uses the findings as the training target, and stage two only uses the impression as the training target. Second, since we incorporate the indication as input for both stages, the outputs of our model are more closely aligned with the clinical information in the indication. In particular stage two is specifically designed to address the clinical query presented in the indication. This design is the defining feature of our model, one that all previous works have overlooked.