MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models

Thank you for reviewing our paper and for the thoughtful comments. We are glad to hear that the reviewer likes MediConfusion and thinks it “hits on a crucial gap in medical AI that’s often overlooked” and that our paper “brings a novel perspective by focusing on systematic failure modes of medical multimodal models”. Please find below our response to the reviewer's concerns.

Re weakness 1: This is a great suggestion. As we summarized in the top-level comment (and Appendix D), we applied the reviewer’s idea and fine-tuned LLaVA-Med on MediConfusion. The detailed findings are in Appendix D, but we give an overview of the results here. We follow standard procedure to fine-tune LLaVA-Med, consisting of fine-tuning the language model weights and the multimodal adapter, while the image encoder is frozen.

Our results show that even after 1000 epochs, the model cannot achieve 100% set accuracy on the training data, indicating that the inputs of some pairs are too similar for the model to distinguish between them. Note, that typical fine-tuning consists of between 1-50 epochs to avoid overfitting. Furthermore, the best test accuracy is around 20%, which is still well below random guessing. These results show that the model likely learned the multiple-choice format due to fine-tuning (improved accuracy from 0% to 20% in multiple-choice evaluation mode), but otherwise the knowledge learned on the train set does not generalize. This strongly implies that vision embeddings of existing models are unable to capture details fine-grained enough for medical reasoning. Thus, in order to improve model robustness and reliability, and to eventually attain a near-perfect score on MediConfusion, we need to improve the vision encoder through improved training data quality (more detailed captions, better alignment, higher quality images), pretraining strategies tailored to the medical domain, or architectural innovations, all of which are exciting directions for future work.

Re weakness 2: Thank you for this insightful suggestion. We agree that the visual encoder plays an important role, and the lack of a specialized vision encoder can limit the models. Also, the models need to have the medical knowledge necessary to answer these questions. However, it is worth noting that RadFM has its own vision encoder trained on medical images, including 3D radiology images, but it still fails on MediConfusion. Furthermore, following the reviewer’s suggestion, we perform additional experiments on the vision features of common models. The detailed results are included in Appendix F, and we have a summary in the top-level comment (point 4).

In particular, we analyze the similarity of MediConfusion pairs in the feature space of vision encoders used by open-source models in our experiments: CLIP and the visual encoder of RadFM. We average pool across the token dimension in the output of vision encoders to produce the visual embedding. The histograms of confusing pair similarities depicted in Figure 8 in our updated manuscript show that the image pairs have high similarity in not only BiomedCLIP’s, but other vision encoders’ embedding spaces, both trained on general-domain data and medical data. We would also like to emphasize that producing different enough embeddings for confusing pairs is a necessary but not sufficient condition to tackle MediConfusion. Specifically, Gemini tends to select different answers for questions within a pair (implying it is able to differentiate between them), however, its overall performance is still below random guessing, highlighting the importance of medical knowledge as well.

We thank the reviewer for their valuable feedback on our manuscript. We are glad to hear that the reviewer finds the paper to be a “novel” and “unique contribution to medical AI evaluation”, and that our work has “broad implications on the deployment of MLLMs in healthcare. Please find below our response.

Re weakness: Thank you for this great suggestion. MediConfusion is an ongoing project with further additions and expansions yet to come with more categories and modalities. However, due to the need for rigorous radiologist review, the curation process is expensive and time-consuming. We have many more automatically generated samples that have not been reviewed by radiologists yet, which can be useful for training medical MLLMs in future work.

Re question 1: Thank you for raising this point. Our primary focus so far for MediConfusion is on radiology due to its enormous clinical impact. The ROCO dataset has a very high number of radiology images with detailed captions, making it a perfect fit for our cause. We plan to expand Mediconfusion with more pairs and add more samples from tasks other than radiology. MediConfusion is our first step towards a collection of challenging medical benchmarks covering diverse modalities.

Re question 2: This is a good suggestion. In our experiments, we set up the prompt following recommendations for the specific model as closely as possible. Following the reviewer’s suggestion, we performed additional ablation studies on the effect of prompting on MediConfusion performance. We recap our experiments from the top-level comment here. We create 10 variants of the multiple-choice input prompt and record the performance of GPT-4o, InstructBLIP, and LLaVA-Med. The table below shows the mean and standard deviation of the performance:

We observe that GPT-4o performance is robust to prompt format, but the performance is still below random guessing. LLaVA-Med struggles with the multiple-choice format, and using different prompts does not resolve this issue. For InstructBLIP, the sensitivity to prompting is higher than GPT-4o, but its performance does not change drastically. Overall, we find that prompt format and style are likely not the root cause for poor performance on MediConfusion. For more details on the specific prompts we used please find the experimental details in Appendix C. Thanks a lot again for this great suggestion.

Re question 3: This is a very important question. Our new fine-tuning experiment (Appendix D, or please see top-level comment for a summary) indicates that the vision embeddings likely don’t capture enough detail to effectively differentiate between images in confusing pairs, as evidenced by the fact that LLaVA-Med cannot even overfit to MediConfusion when trained on the dataset (vision encoder kept frozen). This leads us to believe that the pretraining data quality (caption density, image-caption alignment, image resolution and diversity) used for medical vision encoder training needs to be greatly improved (as the reviewer has also suggested). We believe that incorporating sets of similar images (“confusing sets”) and corresponding anatomically detailed captions into the pre-training data would have the largest impact on improving the vision encoder. In this scenario, the model would be forced to learn more refined and distinguishing image representations for medical images, in order to minimize the contrastive pre-training loss. In tandem with improvements in data quality, the vision encoder architecture also has to be improved in its capacity to represent nuanced medical image features in order to fully utilize the improved dataset. This can be done through spatial attention mechanisms, as the reviewer suggested, an increased (or adaptive) number of visual tokens, and incorporating multi-scale image representations. Fundamental architectural innovations for medical MLLMs is an interesting direction for future research.

We thank the reviewer for their invaluable insight. We are glad that the reviewer finds that our work “thoroughly examines why models struggle to differentiate between visually similar medical images” and that the evaluation metrics are “comprehensive”. Please find below our response to the reviewer's concerns.

Re weakness 1: Great question. Most medical benchmarks for evaluating MLLMs test models on general medical VQA, where the specific shortcomings of visual representations and limitations to discern nuanced but medically relevant details (the scenarios that our work is targetting) may be outnumbered by simple questions that can be easily solved by more superficial medical image understanding capabilities, and in some cases even without looking at the image. For instance, LLaMa, which is a language-only model, achieves non-trivial accuracy on PMC-VQA (as it was reported by their paper) without seeing the images. This may result in high reported accuracies in such benchmarks, as the failure cases are averaged into a large number of standard (and potentially easy to solve) problems. However, in a safety-critical setting, such as healthcare, it is of utmost importance to understand the limitations of AI solutions. To overcome this limitation of other benchmarks, we specifically design MediConfusion to probe a particular type of failure mode originating in the ambiguity of visual embeddings, resulting in a challenging benchmark that forces the model to “look at” the image (language-only accuracy is random guessing by design). We hope that this discussion clarifies our motivation.

Re weakness 2: Thank you for raising this great question. One obstacle to simply varying the benchmark generation thresholds is that each final pair included in MediConfusion, takes 15 minutes of radiologist time on average, including the amortized time to filter through a large pool of candidates, review the pair for correctness and relevance and to improve the language if needed. Overall, it takes around 40 hours of direct radiologist involvement to generate a dataset with approximately 350 samples. As a result, it is extremely costly to perform extensive ablations on dataset curation parameters. That said, we find the reviewer’s question very relevant and interesting, therefore we create an “easy” dataset while skipping radiologist feedback on the included pairs, which we call MediConfusion-easy. We add details in Appendix E. MediConfusion-easy consists of approx. 1000 pairs with BiomedCLIP similarity below 0.4, indicating that they should be clearly distinct for medical models. We tested GPT-4o and LLaVA-Med on MediConfusion-easy, and the results are below.

Our experiment demonstrates that models can indeed differentiate between image pairs in our confusing pair-based VQA format, if the images are different enough, as evidenced by the nearly 100% accuracy of GPT-4o on MediConfusion-easy and the significant improvement in LLaVA-Med’s set accuracy. Thus, this experiment further supports the need for a strict selection criteria for challenging confusing pairs, as easier variants of our VQA may be unable to identify the shortcomings of state-of-the-art models. To ablate the effect of removing radiologist feedback from dataset curation, we create an unfiltered MediConfusion variant using an identical methodology to MediConfusion but skipping radiologist filtering. As shown in the table, removing radiologist feedback does not necessarily improve performance (GPT-4o has slightly lower set accuracy, and LLaVA-Med performed somewhat better on unfiltered data).

Re weakness 3: We thank the reviewer for pointing this out. We added more clarification to this discussion to remove any ambiguity. We highlighted the relevant section in the Introduction of the updated manuscript. We hope that the reviewer finds the language to be more precise in the updated version.

Thanks for getting back to us.

Re question on modality: We followed an identical procedure when curating the easy variant as the original benchmark, and therefore we didn’t apply any hard constraints on the modality of image pairs. That said, due to the relaxed conditions on confusing pairs, the images in the pair are often of different modality or anatomical region. Please find an example image pair in Appendix E (Figure 7.) and a more detailed discussion on MediConfusion-easy. We believe that in this relaxed setting, the MLLMs can leverage less nuanced medical features to differentiate between the images, thus the improved accuracy.

Re question on accuracies: For MediConfusion-Easy, the 34% accuracy reported for LLaVA-Med is “set accuracy” which is 25% for random guessing (only counts as “hit” if both answers are correct within a pair of 2-choice questions). The standard notion of (individual) accuracy of this model on MediConfusion-Easy is 66.9%, which is also higher than random guessing (50%). We believe that both LLaVA-Med’s image encodings (confusion is still 65.6%) and medical knowledge are lacking the necessary detail to achieve better performance on this dataset. We have evidence that the easy benchmark can be solved to near-perfect accuracy, as GPT-4o achieves 98.38% set accuracy (99.04% individual accuracy) without any fine-tuning. We have added individual accuracy and confusion along with the set accuracy in the corresponding table in the updated manuscript to avoid future misunderstanding.

Please let us know if you have any further questions.

We thank the reviewer for their effort in reviewing our work. We are glad that the reviewer finds the problem we target is “well-motivated and underexplored”, that our “benchmark creation process is robust” and believes that our “experiments are comprehensive”. Please find below our response to the concerns and questions.

Re weakness: Thank you for raising this point. Our novelty originates in focusing on failure modes in the medical domain, where bringing attention to limitations and shortcomings is crucial for safe deployment. Before our work, it had been unclear whether general domain failure modes even appear in medical MLLMs, due to significant differences in the data distribution. Specifically, most ambiguities in general-domain vision-language models, such as CLIP, have been traced back to poor pre-training data quality: images are typically not similar enough within a training batch to force the model to learn fine-grained representations in order to differentiate between them and thus the model can rely on “shortcuts” to minimize the contrastive loss. It is unclear if the same phenomenon appears when pre-training medical MLLMs, as both the image and caption distribution are vastly different from general-domain data. Our work reveals the novel insight that in fact medical MLLMs are affected by similar shortcomings, and therefore we provide a challenging benchmark to the community to assess and eventually improve the reliability of MLLMs in the healthcare domain.

Re question 1: This is a very interesting question. We find such pairs by searching for images with high DINO similarity, but low BiomedCLIP similarity. We included sample pairs in Appendix G in Figure 9. As the reviewer can see, these pairs have similar overall structures (e.g., sonography scans) that can be confusing for a general domain purely vision model, such as DINO, but a medical vision encoder can distinguish between them. We note that in MediConfusion, all pairs have low DINO similarities and high BiomedCLIP similarities, which is the opposite of the condition we used to find these specific samples.

Re question 2: Thank you for this great suggestion. Our new experiments (please see top comment 3 for an overview and Appendix D for details) demonstrate that after fine-tuning the language model and multimodal adapter of LLaVA-Med on MediConfusion, the model cannot achieve perfect score on the training data even after a very high number of fine-tuning epochs. This indicates that the frozen visual embeddings are not detailed enough to distinguish between images in some confusing pairs. Novel approaches to improve the vision inputs of medical MLLMs are required to improve performance on MediConfusion, and thus improve their reliability in healthcare. We believe that improved training data quality (more detailed captions, better alignment, higher quality images) and pre-training strategies, architectural innovations, and potentially combining vision features from various models, as the reviewer has suggested, can potentially address the shortcomings, which are all interesting directions for future work.

Re question 3: Thank you for raising this point, this is an important aspect of benchmarking. As depicted in Figure 4 in our paper, some categories, like vascular images, have higher representation in MediConfusion. This is due to their more frequent occurrence in medical datasets in general, (due to their importance in clinical practice) resulting in an overall higher number of valid VQAs after radiologist filtering. We expect models to perform better in such categories (as they have likely seen more samples of these in their training set), and indeed, this is what we observe with GPT-4o (see categorical breakdown of results in Table 2).

4. Input feature similarity for additional image encoders: As suggested by Reviewer 772B, in Appendix F we performed additional experiments on the vision embeddings of various models. In particular, we analyze the similarity of MediConfusion pairs in the feature space of vision encoders used by open-source models in our experiments: CLIP and the visual encoder of RadFM. RadFM has its own vision encoder trained on medical images, including 3D radiology images. Our experiments highlight that, even though we constructed our benchmark based on BiomedCLIP features, the confusing pairs have highly similar features in other models’ representation space as well. Interestingly, we don’t find major differences between general-domain (CLIP) and medical vision-language models (BiomedCLIP, RadFM encoder) in this regard, potentially indicating the need for improved data and training practices.