MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models

Q7: Following the logic above, I feel like a better evaluation scheme is to (1) Use original image (2) Feed it to retriever (3) Selecting data with bad retrieved context (i.e., retriever fail). Then see how MMMed-RAG can rescue these examples. Any comment for this?

A7: For retrieved information, we minimize noise by optimizing the number of retrieved contexts $k$ (e.g., Adaptive Retrieved Context Selection in Section 3.2). Following this, we introduce RAG-PT to specifically address the misalignment issues that arise after incorporating RAG, thereby strengthening Med-LVLM to balance its internal knowledge and external retrieval information. Inspired by your suggestion, we employ a rationale-guided approach [6] that uses LLM to explicitly learn denoising of retrieved content through self-synthesized rationales. First, given a question, the retrieved documents, and the ground truth from the training set, we prompt a powerful Med-LLM (i.e., LLaMA3-Med42-70B [7]) to generate a rationale. This rationale explains how to derive the answer from potentially noisy inputs. Next, we use the synthesized rationale from the previous step to guide another smaller Med-LLM (i.e., LLaMA3-Med42-7B [7]) to explicitly learn denoising of the retrieved documents through in-context learning and supervised learning. By employing this rationale-guided Med-LLM to filter noisy retrieval information, the reliability of our retrieved data improves. As shown in Table R4, experimental results show that after rationale-guided RAG, the model's performance further improved. Thank you very much for your valuable suggestions. We have revised the paper and put these details in Appendix A.6.12.

Table R4: Performance comparison on IU-Xray dataset, including RAG and Rationale-Guided RAG variants.

Q8: Do authors have any rational judgment on not adding more basic baseline so that we can have a better reference method in the paper?

A8: We have already compared several state-of-the-art baselines (post-2023), including MedDr (released in April 2024), FactMM-RAG (released in July 2024), and RULE (EMNLP 2024), which are highly relevant baseline methods. If you have any additional specific suggestions, we would be happy to include more baselines.

Reference

[1] Zhang J, Qu X, Zhu T, et al. CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling[J]. arXiv preprint arXiv:2409.19291, 2024.

[2] Wang X, Zhou Y, Liu X, et al. Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences. ACL, 2024.

[3] Meng F, Wang J, Li C, et al. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models[J]. arXiv preprint arXiv:2408.02718, 2024.

[4] Gravel P, Beaudoin G, De Guise J A. A method for modeling noise in medical images[J]. IEEE Transactions on medical imaging, 2004, 23(10): 1221-1232.

[5] Sanchez M G, Sánchez M G, Vidal V, et al. Medical image restoration with different types of noise[C]//2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2012: 4382-4385.

[6] Wei Z, Chen W L, Meng Y. InstructRAG: Instructing Retrieval-Augmented Generation with Explicit Denoising[J]. arXiv preprint arXiv:2406.13629, 2024.

[7] Christophe C, Kanithi P K, Raha T, et al. Med42-v2: A suite of clinical llms[J]. arXiv preprint arXiv:2408.06142, 2024.

Thank you for your valuable feedback to help us improve our paper. We have revised our paper based on your feedback. We detail our response below and please kindly let us know if our response addresses your concerns.

Q1: A bit over-claimed novelty: The author claimed MMed-RAG is a general approach to address images from different sources. While it is technically true, the way author implement is merely put 3 independent piece together without certain level creativity and claimed it is general (i.e. still relying on training "individual" retriever model on each image source). It is more like engineering trick than methodological novelty.

A1: We have tried training a general retriever by mixing images from all modalities together, instead of using a domain-specific retriever. We conduct experiments based on BiomedCLIP and MedCLIP, but the results are unsatisfactory (please see Table R1). Then we adopt an MoE (Mixture of Experts) architecture. Based on CLIP-MoE, we fine-tune CLIP-MoE [1] with mixing images from all medical imaging modalities, but the performance is still suboptimal. This might be because CLIP-MoE is not pretrained on large-scale biomedical data. All the results are reported in Table R1. Considering model performance, we ultimately adopt a domain-specific retriever architecture. In fact, our approach is also flexible and scalable. When encountering a completely new modality, both the general retriever and our method may still require fine-tuning to achieve optimal retrieval performance. In the future, we plan to explore solutions for pretraining a general retriever on large-scale biomedical data using a Mixture of Experts (MoE) architecture, aiming to determine whether it is possible to develop a truly universal retriever. We have revised the paper and put these details in Appendix A.6.9.

Table R1: Performance comparison based on different retrievers.

Q2: When comparing to Med-Flamingo, which setting did you use? (6-shots or zero-shot?)

A2: All our experiments are conducted under a zero-shot setting.

Q3: I would suggest including LLaVA-Med 1.5 few shot as baseline. Based on the github record [1], LLaVA-Med 1.5 was published on 05/13/2024. while Med-Flamingo, RadFM,MedVInT and miniGPT-Med were published on 2023 based on the reference. So, I suppose LLaVA-Med 1.5 is a more powerful VLM than those. I think it is important to see how far this new more powerful model can go with simple trick (i.e., few shot in context learning) like what Med-Flamingo author did (they have 0-shot and few-shot results)

A3: Thank you very much for your suggestion. We conduct experiments on LLaVA-Med-1.5 using the same few-shot strategy as in Med-Flamingo. As shown in Table R2, the results show that compared to the zero-shot setting, the model's performance significantly decreases, even with RAG applied. Our analysis of this phenomenon reveals that, unlike Med-Flamingo, LLaVA-Med does not use interleaved multimodal data for pretraining. As a result, it lacks the capability for few-shot learning. This point has been mentioned in some discussion forums and GitHub issues. In addition, LLaVA-1.5's unsatisfactory performance on multi-image understanding benchmarks also supports this observation [2,3]. We have revised the paper and put these details in Appendix A.6.10.

Table R2: Performance comparison under zero-shot and few-shot settings.

Q4: Concerns about the alignment analysis: First of all, what data did you use to evaluate in Figure 3?

A4: As mentioned in Line 195-202, Section 3.3 Alignment Analysis, we conduct two tests on LLaVA-Med-1.5 using the Harvard-FairVLMed dataset to analyze the misalignment issues caused by RAG in the Med-LVLM.

Q5: Copy reference rate: If I understand correctly, this refers to line 8 y_{l,o1} in Algorithm 1 , which model overly relies on retrieved text. I find the construction of this idea confusing in practice. Since X_{r} is based on context retrieved by the original image then you pair this context with the noise image. However, in practice, you would never get this clean X_{r} from ground truth. What you will have is a noisy image + noisy context, then the model takes this as input. I am a bit unsure what the point of this evaluation is.

A5: In medical imaging, noise refers to random variations in image signals caused by hardware limitations or environmental factors [4,5]. However, the noise we refer to here pertains to images unrelated to the original image, generated through a two-step process: 1. We use a retriever to select images with the lowest similarity to the target image. 2. We introduce strong diffusion noise to these images. As a result, the noisy images in our case are almost entirely random noise and are not merely examples of domain shifts, such as changes in lighting conditions. Refer to the third section of Figure 1 for examples, and additional examples are included in the Appendix A.9 for reference.

The motivation behind our design is that replacing the original image with a highly noisy image while adding retrieved information corresponding to the original image reveals a significant issue of cross-modal misalignment in the Med-LVLM—namely, it ignores the image information and directly copies the retrieved contexts. To mitigate this issue, we construct such preference pairs to specifically strengthen the model's cross-modal alignment capability.

Q6: Over reliance rate: If I understand correctly, this refers to line 16 y_{l,03} in Algorithm 1m which model is interfered by noisy retrieved context. My question is? Does it just mean the retriever is bad or not well-trained so that the retrieved context is nosy and hurt the model alignment? Referring to Figure 3 OR, it shows over 0.4. With such number, I wonder how well the retriever is trained in the paper, do authors have some results or any observation?

A6: The overall alignment issue arises from the conflict between retrieved information and the model's internal knowledge. For retrieved information, we cannot guarantee 100% accuracy, so some noise is inevitable. The Over-Reliance (OR) rate shown in Figure 3 refers to the proportion of initially correct responses that become incorrect after adding the retrieved context, calculated relative to the total number of incorrect samples, not the total number of all samples. This rate represents the proportion of errors caused by over-reliance, rather than indicating poor performance of the retriever. Through RAG-PT, we can effectively mitigate this issue, significantly reducing the OR rate. Regarding the retriever's performance, as shown in Table R3, we compared the performance of our retriever with several CLIP-based models on radiology datasets for image-to-text retrieval. The results demonstrate that our retriever significantly outperforms the other models in retrieval performance. We have revised the paper and put these details in Appendix A.10.

Table R3: Performance comparison of different retrievers on Recall@1 (R@1) and Recall@5 (R@5) metrics.

Dear reviewer mHiX,

We would like to follow up to see if the response addresses your concerns or if you have any further questions. We would really appreciate the opportunity to discuss this further if our response has not already addressed your concerns. Thank you again!

Q3: In the "Comparison with Other Med-LVLMs" section, it’s not clear which metrics are averaged for the medical VQA and report generation tasks. Additional clarification on this point would help interpret the results.

A3: Thanks for your advice. As shown in Table R3, we have revised this part and put these details in Appendix A.6.2.

Table R3: Accuracy (%) of different Med-LVLMs based on LLaVA-Med-1.5 on medical VQA task.

Q4: In the section on Cross-Modality Alignment, more intuitive explanations are needed to clarify how constructing preference pairs (by introducing noisy images) addresses the issue of cross-modality alignment.

A4: To construct preference pairs for cross-modality alignment, we first select a preferred response by having the model generate an answer using the correct medical image, clinical query, and retrieved knowledge, ensuring the response matches the ground-truth answer. Then, we select a dispreferred response by introducing an unrelated input image. This unrelated image is selected by finding the one with the lowest similarity to the target image and adding noise to distort it further. The dispreferred response is generated when the model uses this noisy, unrelated image along with the query and retrieved knowledge to still produce the correct answer. By comparing these pairs during training, the model learns to prioritize relevant and accurate inputs (e.g., the correct medical image) over noisy or irrelevant ones, improving cross-modality alignment. We have revised this part and put these details in Appendix A.9.

Q2: In Table 2, it is unclear which specific BLEU score is being reported—is it BLEU-1, BLEU-2, BLEU-3, BLEU-4, or an average of these?

A2: Apologies for the confusion. Table R2 shows the average BLEU score. We provide detailed results in Appendix A.6.8.

Table R2: BLEU Score (%) of different methods based on LLaVA-Med-1.5 on report generation task.

Thank you for your constructive comments and suggestions. We have revised our paper according to your comments. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.

Q1: In the analysis of RAG-PT, the authors evaluate the effectiveness of each individual RAG-PT component (1, 2, and 3). To better understand how RAG-PT mitigates misalignment and improves performance, it would be helpful to include experiments with different combinations, such as RAG-PT 1+2, RAG-PT 1+3, and RAG-PT 2+3.

A1: Thank you very much for your suggestion. Preference data designed for different alignment objectives can indeed produce varying effects. Therefore, conducting ablation experiments on combinations of different types of preference data is necessary. We perform comprehensive ablation experiments on RAG-PT1/2/3 as well as their combinations (1+2, 2+3, 1+3) to analyze the effectiveness of each type of data and their combinations. As shown in Table R1, we find that the combination of 1+3 produced the most significant results, indicating that the two misalignment issues (i.e., cross-modality and over-reliance issues) are the most prominent. Targeted mitigation of these two issues yielded the greatest improvement. However, incorporating data for all three alignment objectives yields the best performance, demonstrating the importance of each alignment component. We have revised the paper and put these details in Appendix A.6.6.

Table R1: Ablation results using RAG-PT based on subsets of preference data. VQA: Visual Question-Answering; RG: Report Generation.

Thank you for reviewing our paper and for your valuable feedback. Below, we address your concerns point by point and we’ve revised our paper according to your suggestions. We would appreciate it if you could let us know whether your concerns are addressed by our response.

Q1: The manuscript lacks a comparison with other general methods, such as GPT-4, Gemini, and QwenVL. An intuitive comparison to these general-purpose models would provide valuable context for understanding the advantages and limitations of MMed-RAG.

A1: Thank you for your valuable suggestion. We have conducted a comparison of general models, including GPT-4, Gemini, QwenVL, LLaVA-1.6, and InternVL-2. As shown in Table R1, our findings show that MMed-RAG consistently outperforms these models, further demonstrating its effectiveness. We have revised the paper to include these details in Appendix A.6.2.

Table R1: Accuracy (%) of different Med-LVLMs based on LLaVA-Med-1.5 on medical VQA task.

Q2: A major concern is the use of preference datasets for fine-tuning, which may cause the model to focus excessively on hard samples, potentially leading to challenges such as catastrophic forgetting and overfitting. It would be beneficial to use an external validation dataset from the same domain to assess the model's generalizability.

A2: Considering the risk of overfitting, and based on your suggestion, we use external validation datasets from the same domain to evaluate the generalizability of MMed-RAG. We select two domain-specific subsets from PubMedVision [1], i.e., fundus digital photography and microscopy image, for ophthalmology and pathology, respectively. As shown in Table R2, the results show that our method still significantly outperforms other Med-LVLMs on the external validation datasets, indicating MMed-RAG performs well when generalized to external datasets, demonstrating its strong generalization capability. We have revised the paper and put these details in Appendix A.6.7.

Table R2: Performance comparison of models on external validation datasets.

Reference

[1] Chen J, Gui C, Ouyang R, et al. Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale. EMNLP 2024.

Reference

[1] Wu C, Zhang X, Zhang Y, et al. Towards generalist foundation model for radiology[J]. arXiv preprint arXiv:2308.02463, 2023.

[2] Seyfioglu M S, Ikezogwo W O, Ghezloo F, et al. Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos. CVPR 2024.

[3] Holland R, Taylor T R P, Holmes C, et al. Specialist vision-language models for clinical ophthalmology[J]. arXiv preprint arXiv:2407.08410, 2024.

[4] Dong, G., Zhu, Y., Zhang, C., Wang, Z., Dou, Z. and Wen, J.R., 2024. Understand what llm needs: Dual preference alignment for retrieval-augmented generation. arXiv preprint arXiv:2406.18676.

[5] Li, X., Mei, S., Liu, Z., Yan, Y., Wang, S., Yu, S., Zeng, Z., Chen, H., Yu, G., Liu, Z. and Sun, M., 2024. RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards. arXiv preprint arXiv:2410.13509.

[6] Li H, Liu J, Wang Z, et al. LITE: Modeling Environmental Ecosystems with Multimodal Large Language Models, COLM 2024.

Q2: Though the idea of RAG-PT is novel in the application to medical imaging, there exist several related works in exploring DPO for RAG fine-tuning in LLMs. The authors should discuss these works in the related work section and how MMed-RAG adds to the literature [4,5].

A2: Thank you for your suggestion. DPA-RAG [4] addresses the alignment issues between the external reranker and the internal LLM through supervised fine-tuning. Later, RAG-DDR [5] leverages a rolling method to generate perturbed responses, further mitigating conflicts between parameter memory and external knowledge. We have already discussed these references in the related work section in the revised paper. In addition, we would like to point out RAG-DDR is concurrent work with ours and is released after the ICLR 2025 submission deadline.

Q3: It is unclear if RAG-PT is specific to only the medical imaging domains, or if it can be applied broadly to other domains. I think it would be valuable to assess the contribution in RAG-PT in problems outside of medical imaging, especially where image-text prompts are likely to come from heterogeneous domains.

A3: We have applied RAG-PT to one additional domain (i.e., environmental ecosystems modeling) to further validate the effectiveness of RAG-PT. We conduct experiments on two environmental system modeling datasets [6]. The CRW-Temp dataset is a river water temperature prediction dataset aimed at forecasting the daily average water temperature of a specific day based on observed physical variables. The CRW-Flow dataset focuses on predicting river segment flow based on observed physical variables. The model used is LITE [6], an environmental system large model based on LLaMA2. We train a semantic time-series encoder using time-series information-text pairs, which works in conjunction with a text encoder as the retriever. Then we retrieve the most similar environmental descriptions based on the current environmental descriptions. As shown in Table R2, our approach also demonstrates significant performance improvements on tasks in this domain. We have revised the paper and put these details in Appendix A.6.4.

Table R2: Performance comparison of different models on CRW-Temp and CRW-Flow datasets.

Q4: Is the problem of misalignment in introducing RAG context only in LLAVA-Med 1.5 7B, or does this exist in other open-source (etc., Med-Flamingo, miniGPT-Med) and commercial LVLMs (like GPT-4o)?

A4: Following the alignment analysis method we apply to LLaVA-Med-1.5 in Section 3.3, we conduct two alignment analysis tests on multiple open-source Med-LVLMs and commercial LVLMs using the Harvard-FairVLMed dataset with the incorporation of retrieved information. These tests respectively evaluate (1) cross-modality alignment and (2) overall alignment with the ground truth. As shown in Table R3, the results indicate that both existing open-source Med-LVLMs and commercial LVLMs exhibit misalignment issues with retrieved information. In addition, it is worthwhile to mention that GPT-4o demonstrates the best alignment performance compared with other models when incorporating RAG, especially in cross-modal alignment. This is likely because GPT-4o has been well-trained in visual perception and may also have utilized some post-training methods (like preference optimization) to optimize modal alignment. We have revised the paper and put these details in Appendix A.6.5.

Table R3: Comparison of Copy-Reference Rate and Over-Reliance Rate across different models.

Thank you for your valuable feedback to help us improve our paper. We have revised our paper based on your feedback. We detail our response below and please kindly let us know if our response addresses your concerns.

Q1: The contribution of domain-specific retrievers for RAG context in MMed-RAG is not clear to me. Practically, it is simpler to build domain-specific LVLMs that solves a targeted clinical problem well than generalist biomedical LVLMs. If we are evaluating an off-the-shelf for VQA in radiology, why would we also include pathology and ophthalmology as possible documents for RAG? To evaluate this contribution, we would need to compare performance of MMed-RAG against domain-specific LVLMs (radiology, pathology, ophthalmology). All comparisons in this experiment here should ideally also adopt RAG-PT.

A1: We design a domain-specific retriever leveraging a generalist Med-LVLM to retrieve information from a dedicated database based on the identified modality of the input medical image. Here, the domain identification models used are capable of reliably recognizing modalities with high accuracy (~99.83% accuracy in our experiments). For radiology VQA tasks, input radiology images are classified as “radiology” by the model, enabling the retrieval of knowledge exclusively from the radiology database to enhance generation. All retrieved documents are specific to radiology and exclude other modalities.

In addition, we conduct additional experiments to compare our method with domain-specific Med-LVLMs as follows: Radiology: RadFM [1], Pathology: Quilt-LLaVA [2], Ophthalmology: RetinaVLM [3]. For radiology, we use the IU-Xray dataset to evaluate VQA. For pathology, we use the PMC-OA pathology subset to evaluate VQA. For ophthalmology, since the domain-specific Med-LVLM, i.e., RetinaVLM, is only trained on report-generation tasks, we use the Harvard-FairVLMed dataset to evaluate report generation. As shown in Table R1, our method significantly outperforms each domain-specific Med-LVLM. Additionally, following your advice, we apply RAG-PT to each domain-specific Med-LVLM. As shown in Table R1, after incorporating RAG-PT, the performance of these models improve significantly, demonstrating the compatibility of our method. Furthermore, domain-specific Med-LVLMs could outperform generalist Med-LVLMs in their specialized domains, as they are fine-tuned using specialized medical domain data. While this significantly enhances their medical understanding in specific domains, it may reduce their generalization ability, such as their capacity to comprehend retrieved information. Consequently, even after incorporating RAG-PT, the performance of several domain-specific Med-LVLMs (e.g., RetinaVLM and RadFM) is inferior to MMed-RAG. We have revised the paper and put these details in Appendix A.6.3 and A.7.

Table R1: Model performance comparison with domain-specific Med-LVLMs.

Dear reviewer 3Jou,

We would like to follow up to see if the response addresses your concerns or if you have any further questions. We would really appreciate the opportunity to discuss this further if our response has not already addressed your concerns. Thank you again!

We sincerely appreciate the thoughtful discussion and your valuable feedback. We are delighted that our rebuttal has addressed all your concerns. We kindly notice that your rating appears unchanged and would greatly appreciate it if you could take a moment to double-check your review. Thank you again!