Retrieval Information Injection for Enhanced Medical Report Generation

Q2: Since medical images are highly similar as mentioned in the paper, is it possible for the workflow to retrieve images that are similar but have distinct symptoms, leading to inaccurate diagnosis?

A2: Thanks for your comment. The retrieved reports may include false-positive observations, which we address by employing an averaging mechanism during the decoding and report filtering stages. This approach mimics the voting process in expert consensus, aiming to mitigate the impact of such false positives. However, in extreme cases—when the majority of the retrieved reports contain the same false-positive observations—this mechanism may fail. For instance, as illustrated in Appendix A.2.1, most retrieved reports incorrectly identified false-positive observations of atelectasis, leading RIN to erroneous inclusion of atelectasis in the generated results.

[1]https://www.imageclef.org/2023/medical/caption

[2]Rückert, Johannes, et al. "Rocov2: Radiology objects in context version 2, an updated multimodal image dataset." Scientific Data 11.1 (2024): 688.

[3]Yim, Wen-wai, et al. "Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation." Scientific Data 10.1 (2023): 586.

[4]Van Veen, Dave, et al. "Clinical text summarization: adapting large language models can outperform human experts." Research Square (2023).

[5]Yim, Wen-wai, et al. "Dermavqa: A multilingual visual question answering dataset for dermatology." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2024.

[6]Soldaini, Luca, and Nazli Goharian. "Quickumls: a fast, unsupervised approach for medical concept extraction." MedIR workshop, sigir. 2016.

Thanks to the precious suggestions made by the Reviewer gLqw. These suggestions provide us with a lot of insights and help us improve the quality of our work. We are also highly grateful to the reviewer for dedicating her/his time and effort to help us improve the quality of our paper.

Q1: Has the method been tested on modalities other than chest X-rays, such as MRIs or CT scans, to assess its adaptability and effectiveness?

A1: Thank you for your suggestion. We attempted to evaluate the effectiveness of our method on the Caption Prediction Task of the ImageCLEFmedical Caption 2023 challenge[1]. The evaluation was conducted on the ROCO V2 dataset[2], which includes various types of medical images such as ultrasound, X-ray, PET, CT, MRI, and angiography. We incorporated our method into the pretrained MedICap model and present the results. To build the image retrieval database, we used BioMedCLIP instead of BiomedVLP, this is a contrastive learning model pretrained on various medical image types. During the decoding phase, our settings were as follows: $k=7$, $\alpha=1/3$ (the default settings in our paper), and beam search within to 4 (as reported by the authors). The results are shown in the table below.

*The results except CSIRO (MedICap) and RIN are from ImageCLEFmedical Caption[1]. CSIRO (MedICap) is reproduced by us and the result is slightly higher than the result reported in ImageCLEFmedical Caption[1].

We found that existing methods seem to be unable to effectively measure the subtle differences in the generated reports, which may be because these methods were not developed for medical text evaluation. In the absence of methods to evaluate clinical efficacy in the task, we employ the MEDCON metric[3] to assess the alignment between generated and referenced reports. MEDCON metric is currently widely used in different types of medical text evaluation[4,5]. Different terminological systems may employ varying names or codes to represent the same concept. Within the Unified Medical Language System (UMLS), each medical concept is assigned a distinct Concept Unique Identifier (CUI). MEDCON extracts each medical concept's unique identifier (CUI) in the surgical report through the QuickUMLS[6] and computes the F1-score to determine the similarity between the UMLS concept sets in predicted and referenced reports. Experiments show that our method can effectively improve the accuracy of medical concept description.

We sincerely appreciate the reviewer’s thoughtful time and effort in thoroughly evaluating our paper and offering invaluable feedback.

Q4: Please further explain the differences between the proposed retrieval module and the existing report retrieval methods.

A4: Thank you for your comment. Existing report retrieval methods can generally be divided into two main categories:

Methods fully dependent on retrieval. This approach typically populates a predefined template with the retrieved key information[2]. While this ensures consistency, it limits flexibility and adaptability by producing fixed sentence structures. Recent advancements have use of retrieved information as input to large language models (LLMs)[4] to guide report generation. This enables more natural and diverse outputs but LLMs may struggle to accurately perceive the multiple retrieved reports, leading to biases or omissions [5] in the generated reports.

Methods integrating retrieval information with report generation models. These methods incorporate retrieval information into models through mechanisms like attention[2]. This facilitates more dynamic and context-aware report generation but comes with the drawback of significant training costs.

Compared to the first method, our method effectively ensures linguistic fluency in report generation. This is achieved by balancing the inherent knowledge of the vanilla report generation model with the retrieved information. In contrast to the second method, our method is training-free. By leveraging the contrastive decoding strategy that adjusts the probability distribution during the decoding stage, our method eliminates the need for additional training. As a result, our method addresses the challenges faced by the other two approaches, namely the inability to generate coherent reports or the need for additional training costs.

Q5: Report generation needs to retrieve $k$ highly relevant reports, how to determine the value of $k$, and what is the specific value of $k$ used in this paper.

A5: Thanks for your comment. We will address your two questions in order. The first question pertains to the issue of determining the value of $k$, and the second question relates to the parameters of specific value of $k$ used in our manuscript.

There are several ways to determine the value of $k$. Here, we introduce two feasible approaches.

We need to experiments on the validation set to identify the optimal $k$ for retrieving similar reports. For each validation sample, we retrieved the top-k most similar reports ($k$=1 to 10).

Evaluate generated reports in validation set to determine the value of $k$. For different candidate $k$ values, we generate corresponding reports on the validation set and calculate the CE metrics between the generated report and the ground truth report to quantify the generation quality. By comparing the CE performance corresponding to each $k$ value, we finally select the $k$ value with the highest F1 score (best performance) as the determined k value to ensure that the model generation performance is optimal.

Evaluate retrieved reports to determine the value of $k$. We calculate the CE metrics between the different retrieved reports of $k$ values and the ground truth reports on the validation set to quantify the generation quality. By comparing the average F1 score performance corresponding to different retrieval reports of $k$ values, we finally select the $k$ value with the highest F1 score (best performance) as the determined k value to ensure that the model generation performance is optimal.

We next address the concern about the specific value of $k$ used in this manuscript. We kindly remind the reviewer, in fact, we have introduced the relevant parameter of the $k$ in the original manuscript. In our manuscript we set $k=4$.

[1]Tanida, Tim, et al. "Interactive and explainable region-guided radiology report generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2]Syeda-Mahmood, Tanveer, et al. "Chest x-ray report generation through fine-grained label learning." Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part II 23. Springer International Publishing, 2020.

[3]Jin, Haibo, et al. "Promptmrg: Diagnosis-driven prompts for medical report generation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 3. 2024.

[4]Ranjit, Mercy, et al. "Retrieval augmented chest x-ray report generation using openai gpt models." Machine Learning for Healthcare Conference. PMLR, 2023.

[5]Zhou, Yujia, et al. "Trustworthiness in retrieval-augmented generation systems: A survey." arXiv preprint arXiv:2409.10102 (2024).

Thanks to the precious suggestions made by the Reviewer WUqZ. These suggestions provide us with a lot of insights and help us improve the quality of our work. We are also highly grateful to the reviewer for dedicating her/his time and effort to help us improve the quality of our paper.

Q1: In Section REPORTS RETRIEVAL, ... This work does not further explain the design and effectiveness of the retrieval method. The authors are advised to further validate the effectiveness of the retrieval model.

A1: Thanks for your comment. To further validate the effectiveness of the retrieval process, we designed an ablation study to compare the performance of different models and distance metrics on the final results. We conducted the following experiments in the same settings: $k=4$, $\alpha=1/3$, and beam search within to 4. The outcomes are summarized in the table below. The experimental results demonstrate that employing BiomedVLP, a model pretrained on biomedical data, outperforms directly using CLIP for encoding. Additionally, the choice of distance metric has little effect on the results.

Q2: In the ablation study, as shown in Table 4, the model using Pre-trained Model Prediction Distribution, Retrieved-reports Average Distribution, and Report Filter did not achieve the best results in BLEU-2, BLEU-3, and BLEU-4. The authors are advised to further analyze the reasons for the poor performance of the model

A2: Thanks for your suggestion. When evaluating BLEU scores, it is essential to simultaneously consider additional text metrics such as METEOR and ROUGE. BLEU primarily measures exact n-gram matches, whereas METEOR and ROUGE emphasize semantic relevance and content coverage. As illustrated in Table 4, our approach enhances METEOR and ROUGE scores while exhibiting a decrease in BLEU. This may be because the report generated by our method will cover more semantic information, but the vocabulary in the generated report may have morphological changes.

Moreover, natural language generation (NLG) scores are of limited importance in medical report generation tasks. NLG scores heavily depend on the specific preprocessing applied to reference reports[1]. For instance, converting text to lowercase has been shown to substantially improve BLEU scores when compared to uppercase references[1]. In contrast, clinical efficacy (CE) metrics are invariant to such preprocessing because they compare the presence or absence of diseases between reference and generated reports[1].

Q3: The authors are advised to supplement the setting details of hyperparameters, as well as a discussion of model effects using different hyperparame

A3: Thank you for your suggestion. We would like to kindly point out that the relevant hyperparameters, $\alpha$ and $k$, were introduced in Sections 3.2 and 4.5 of the original manuscript. To enhance clarity, we have now included these hyperparameters in the EXPERIMENTAL SETTINGS section. Additionally, we have analyzed the effects of different hyperparameters in Section 4.5 PERFORMANCE ANALYSIS of the original manuscript. For further details, please refer to Figure 4 and Figure 5.

Q4:Line 296 - Average F-1 score should be based on match to ground truth. Is that what is meant in line 296 or F-1 score is computed relative to which report?

A4: Thank you for your comment. For each test sample, we calculated the F1 scores between the top-k retrieved reports and the reports generated by the vanilla model as well as the reports generated with RIN. Among vanilla model generated report and RIN generated report, we selected the report with the highest average F1 score as the final report.

Q5: Steps 1-6 described in Figure 1 are not very clear. Is image information used only in step 3 or also in step 5?

A5: Thanks for your comment. Image information is only used in steps 1 and 2. Step 1 is used as the image input for the vanilla report generation model, and step 2 is used for image feature extraction for the retrieval report.

Q6:Instead of using Bio-VLP for image-to-image matching why not use it directly to retrieve radiology reports as done in earlier papers with CLIP-based retrieval (https://proceedings.mlr.press/v158/endo21a/endo21a.pdf) since Bio-VLP is a multimodal model?

A6: Thanks for your suggestion, we found that it seems that image-to-text matching is still difficult, which may be due to the diversity of radioactive reports, so we only use the image encoder for image-to-image matching. In order to verify the effectiveness of image-to-image matching. We conducted the following experiments in the same settings: $k=4$, $\alpha=1/3$, and beam search within to 4. Experiments show that injecting the image-to-image retrieved reports into the vanilla model can generate higher quality report:

Q7: The use of the term 'distribution' to refer to the generated output from report generator is confusing. Are multiple reports coming out in one step from the report generator?

A7: Thank you for your comment. The report generator models a probability distribution over the next token, aligning with the interpretation of "distribution" frequently discussed in the provided paper [5]. Notably, the generator's ability to produce multiple distinct reports is directly influenced by the configuration of the batch size, which governs the diversity and volume of generated outputs.

Q8: Were the results from CheXBert freshly generated for the datasets by the authors or a reuse of numbers quoted from previous work since the ChexBert using the Allen NLP has some dependencies on older CUDA libraries.

A8: Thank you for your comment. I apologize if I misunderstood your point. I’ll make an effort to understand it better. In our process, we use Chexbert twice: once for filtering and once for evaluating the final effect, and both instances require recalculations.

[1]Syeda-Mahmood, Tanveer, et al. "Chest x-ray report generation through fine-grained label learning." Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part II 23. Springer International Publishing, 2020.

[2]Jin, Haibo, et al. "Promptmrg: Diagnosis-driven prompts for medical report generation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 3. 2024.

[3]Ranjit, Mercy, et al. "Retrieval augmented chest x-ray report generation using openai gpt models." Machine Learning for Healthcare Conference. PMLR, 2023.

[4]Kim, Youna, et al. "Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts." arXiv preprint arXiv:2408.01084 (2024).

[5]Qiu, Zexuan, et al. "Entropy-based decoding for retrieval-augmented large language models." arXiv preprint arXiv:2406.17519 (2024).

[6]Zhou, Yujia, et al. "Trustworthiness in retrieval-augmented generation systems: A survey." arXiv preprint arXiv:2409.10102 (2024).

Thanks to the precious suggestions made by the Reviewer KHTY. These suggestions provide us with a lot of insights and help us improve the quality of our work. We are also highly grateful to the reviewer for dedicating her/his time and effort to help us improve the quality of our paper.

Q1:There are multiple methods that have taken the RAG... In general, the relation of retrieval injection to RAG will have to be explained.

A1: Thank you for your suggestion. We have incorporated the mentioned papers into the related work section to ensure a comprehensive contextualization of our study. Below, we provide a detailed explanation of the distinctions between our approach and these referenced methods:

Report retrieval methods

Existing report retrieval methods can generally be divided into two main categories:

1. Methods fully dependent on retrieval. This approach typically populates a predefined template with the retrieved key information[2]. While this ensures consistency, it limits flexibility and adaptability by producing fixed sentence structures. Recent advancements have introduced the use of retrieved information as input to large language models (LLMs)[4] to guide report generation. This enables more natural and diverse outputs but LLMs may struggle to accurately perceive the multiple retrieved reports, leading to biases or omissions[6] in the generated reports.

2. Methods integrating retrieval information with report generation models. These methods incorporate retrieval information into models through mechanisms like attention[2]. This facilitates more dynamic and context-aware report generation but comes with the drawback of significant training costs.

Compared to the first method, our method effectively ensures linguistic fluency in report generation. This is achieved by balancing the inherent knowledge of the vanilla report generation model with the retrieved information. In contrast to the second method, our method is training-free. By leveraging the contrastive decoding strategy that adjusts the probability distribution during the decoding stage, our method eliminates the need for additional training. As a result, our method addresses the challenges faced by the other two approaches, namely the inability to generate coherent reports or the need for additional training costs.

Contrastive decoding in RAG

Some recent works[4,5] have introduced RAG into contrastive decoding methods, aiming to improve the open-domain question answering capabilities of LLM. These work focuses on mitigating the distractibility issue from both external retrieved documents and parametric knowledge. And these tasks are basically applied to short-form QA tasks. Our job is to generate long reports with clinical efficacy.

Q2: The terminology used to explain Figure 1 is confusing. You mention text decoders and report generators. Are there referring to the same module or two different modules? If different, this is not reflected in Figure 1.

A2: Thanks for your comment. They are different. The output of the report generator is a probability distribution, and the text decoder (Beam Search is used in our work) selects the next token based on these probability distributions.

Q3: The use of image encoding features to retrieve similar images needs to be evaluated to see the type of reports retrieved. What is the ratio of overlap of findings of such retrieved reports with the ground truth reports associated with these chest X-rays. Since MIMIC dataset is used, all the chest X-ray images (train-test-validate) should have ground truth reports.

A3: Thanks for your suggestion. We calculated the clinical efficacy coverage of the retrieval report and the groundtruth of the test set, and the specific results are as follows. We found that the performance of the simple retrieval result is lower than the result of the final generated report, which also reflects that our method is that balances the vanilla model knowledge and the external retrieval knowledge obtained to generate the report.

Q5: In Table 3, it appears that the proposed retrieved-reports average distribution...could further strengthen this method.

A5: Thank you for your suggestion. We will consider introducing a more suitable denoising module in the next version.

Q6: Does this decoding strategy heavily rely on retrieval accuracy?

A6: Thanks for your comment. Our decoding strategy depends on, but is not entirely dependent on, retrieval accuracy. RIN generates reports based on the hyperparameter $\alpha$-balanced retrieval information and vanilla model prediction results. We tried experimenting with different external image encoders and distance metrics, and the results showed that even using a simple clip as an external encoder for retrieval can improve CE metrics’ performance. We conducted the following experiments in the same settings: $k=4$, $\alpha=1/3$, and beam search within to 4. However, more accurate retrieval information obviously helps generate more effective results.

[1]Jin, Haibo, et al. "Promptmrg: Diagnosis-driven prompts for medical report generation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 3. 2024.

Thanks to the precious suggestions made by the Reviewer Fskn. These suggestions provide us with a lot of insights and help us improve the quality of our work. We are also highly grateful to the reviewer for dedicating her/his time and effort to help us improve the quality of our paper.

Q1: Although the improvement in results is significant, there is a lack of intuitive explanation or insight into the source of this improvement.

A1: Thanks for your comment. As mentioned in the abstract (line 017-019), "The essence of this method lies in fully utilizing similar reports of target images to enhance the performance of pre-trained medical report generation models."

Q2: Additionally, it remains unclear whether this decode strategy can be applied to other report generation methods.

A2: In our original manuscript, we integrated our RINmodule in CvT2DistilGPT2. CvT2DistilGPT2 uses GPT2 as Report Generator. In A4, we integrated our modules to the latest SOTA model PromptMRG[1]. PromptMRG uses Bert as Report Generator, proving that our method is Model-Agnostic and generally applicable to various autoregressive generation methods.

Q3: If space permits, I suggest moving the details of the INFORMATION INJECTION (currently at the end of the supplementary materials) into the Methods section. Additionally, the current pseudocode is not detailed enough and should be elaborated further.

A3: Thanks for your suggestion, we will move it to the Methods section later. Besides, We have rewritten the pseudocode, please refer to Appendix A1.1 in our manuscript.

Q4: The experimental results in Table 1 do not reach the current state-of-the-art (SOTA) level. The authors could try to combine more advanced methods to verify the stability of the proposed decoding strategy.

A4: Thank you for providing us with the latest SOTA baseline PromptMRG[1]. We have supplemented the results of adding our method to the pre-trained PromptMRG model. The detailed parameters are as follows: $k$=3, $\alpha=1/3$ (this is the default setting in our paper), beam search within to 3 (this is the default setting in the author's paper[1]), and the results are shown in the following table. The experimental results show that our method has achieved 2.8%, 4.1% and 3.8% improvements in the three CE metrics of precision, recall and F1, respectively.

*Since we do not have access to the MIMIC-CXR Database preprocessed by R2Gen, our experiments are conducted directly on the original MIMIC-CXR Database provided by physionet, which results in lower baseline results than the performance in the author's paper.