Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA

Thank you for your constructive feedback and thoughtful suggestions. Below, we address your concerns and questions in detail.

Clarifications on Adversarial Questions and Evaluation Metrics

(Addresses Weakness 1 and Questions 1 & 2: Inclusion of adversarial questions and explanation of evaluation metrics)

W1: My main complaint with the paper is that I find the presentation of the results difficult to follow at times. For example, it is not perfectly clear to me whether the "adversarial" questions are included in the tally of 57k QA pairs, and are also included as part of the evaluation benchmark by default - or if the 57k QA pairs represent the default.

Q1: But in either case, the tally of the original questions that do not have an adversarial pair remains consistent? Wouldn't this obfuscate the true impact of the adversarial examples since it's diluted by the original, non-adversarial "no" questions?

Q2: Are adversarial questions included in the tally of 57k QA pairs, and used in evaluation by default? e.g. in table 4, figure 4, 5, etc.

Adversarial questions are included by default in the 57k VQA pairs and are incorporated into all accuracy metrics reported in the study, except for the "averaged accuracy" column in Table 3. Our evaluation ensures fairness by requiring models to answer both the ground-truth and adversarial questions correctly. For instance, if a model answers the ground-truth question correctly but the adversarial question incorrectly, its "averaged accuracy" in Table 3 would be inflated to 0.5, but its score in our main evaluation framework would remain 0. This design ensures that adversarial questions robustly test model reliability and resilience, as illustrated in Figure 2.

The reported "accuracy with adversarial pairs" in our study is the default metric, ensuring results are not diluted by non-adversarial questions. Table 3 highlights the significant accuracy drop across various models, emphasizing their vulnerability to adversarial challenges and providing insights into model robustness. We have added additional clarifications in Section 4.1.2.

Why Multiple-Choice Questions Were Not Used

W2: Additionally, a limitation of the study is that the benchmark / adversarial questions are only limited to binary yes / no settings, when I can imagine the methodology easily extending to multiple choice questions, which seems like a missed opportunity.

As explained in Section 2.2, recent studies [1, 2] highlight key limitations of multiple-choice questions (MCQs), such as selection bias and permutation sensitivity. Specifically:

• A model might select the correct ground-truth attribute (e.g., A from A, B, C) because it finds A relatively more probable than the alternatives, rather than explicitly denying B and C.
• This behavior complicates the evaluation of a model's reasoning capabilities.

By using binary (yes/no) questions, we adopt a deterministic and straightforward evaluation strategy that ensures models are rigorously tested on their ability to confirm the presence of ground-truth attributes while rejecting hallucinated adversarial attributes.

Prompting Techniques for Different Model Types

Q3: Besides the question + image itself, is there any additional formatting / parsing used to evaluate the various models? especially for open-source MLLMs that may not be perfect at instruction following, what happens if a model outputs a full sentence response instead of a simple yes / no answer.

For specialized medical-domain models, we followed the prompting formats specified in their respective papers. For general-purpose models, we appended "(please answer yes or no)" to the end of each question. This minor modification aligns with the instruction-following capabilities of general-purpose models, improving response consistency.

When models produced full-sentence responses instead of simple binary answers, we post-processed these outputs to extract the yes/no component for evaluation.

Thank you for your constructive feedback and thoughtful suggestions. Below, we address your concerns and questions in detail.

Clarifying Dataset Focus and Analysis

(Addresses W1: Concern about trivial questions and focus on specialized questions)

W1: Some questions are too trivial in the dataset. The ultimate goal of multimodal models is deployment in real clinical practice when the accuracy is good enough. However, no clinician will ask questions on basic modalities (e.g., CT, MR) or organs because they are too trivial. I would suggest that the authors focus their analysis and discussion on the more clinically relevant and challenging aspects of the dataset, such as identifying specific abnormalities or conditions.

We appreciate your feedback regarding the clinical significance of the questions in our dataset. Our benchmark provides a comprehensive evaluation with 57,000 VQA pairs derived from over 6,000 medical images. Each image is associated with questions ranging from basic attributes, such as organ and modality identification, to more specialized inquiries involving abnormality detection, specific conditions, and their corresponding locations.

While questions about basic modalities or organs may seem trivial, they serve as an essential foundation for assessing a model’s readiness for clinical deployment. Accurate identification of organs and imaging modalities is a critical prerequisite for more advanced diagnostic tasks. Ensuring that models can handle these basic elements is important for their overall reliability in real-life clinical assistance.

In response to your suggestion, we have updated our analysis in Section 4.2.1 to emphasize specialized questions of higher clinical relevance. Sections 4.2.2 and 4.3.2 include in-depth error analyses and procedural diagnostics, focusing on these specialized questions to provide more clinically meaningful insights as well.

Incorporating Open-Source Multimodal Models

(Addresses Q1: Performance improvement with open-source multimodal models in ablation study)

Q1: It is interesting that augmentation with visual descriptions generated by GPT-4o can improve the performance. Can open-sourced multi-modal models (e.g., Qwen2-VL-72b-Instruct, LLaMa 3.2 Vision) improve the performance as well?

We acknowledge the importance of augmenting performance with open-source multimodal models. Due to computational resource constraints, we have extended our experiments to include LLaMA 3.2 Vision (90B) on a subset of 13,132 VQA pairs from the 57,132 VQA pairs in the dataset. The table below compares the performance of four models on the 13k samples under three settings: (i) vanilla inference, (ii) augmentation with external visual descriptions generated by GPT-4o, (iii) augmentation with external visual descriptions generated by LLaMA 3.2 Vision. Similar to the observed improvements with proprietary models like GPT-4o, augmentation with stronger visual descriptions boosts performance.

These findings suggest that the enhancement is not exclusive to proprietary models and that open-source models can also benefit from this approach. Once we complete the evaluation of the full dataset, we will update the results in the paper.

Thank you for your thoughtful and constructive feedback. Below, we address your comments and questions in detail.

Addressing Concerns About Data Contamination and Evaluation Fairness

(Addresses W1 and W2: Potential data contamination and model fairness)

W1: My main concern is the possible data contamination in larger closed or open-source models. When some models have been trained on the data and some not, evaluation using this dataset loses its fairness.

W2: Having a model that has been previously trained on the data, can we ensure that it does not cheat?

In the era of large multimodal models (LMMs), avoiding data contamination is inherently challenging, especially when datasets are curated from publicly available real-world data. To mitigate this, we deliberately introduced adversarial question-answer pairs alongside ground-truth pairs to enhance robustness and fairness.

For ground-truth QA pairs, we transformed the data into formats that differ from their original representations. For example:

• ChestX-ray14: Bounding boxes were converted into textual positional descriptors, and classification labels were integrated into our VQA curation process.
• MedICaT: Captions were processed into structured metadata containing extracted ground-truth information rather than being directly used.

Each ground-truth QA pair was paired with an adversarial counterpart using hallucinated attributes. These adversarial attributes were carefully curated to be unseen by models, even those potentially trained on the original data. Additionally, our evaluation framework calculates accuracy by requiring models to correctly answer both the ground-truth and adversarial pairs. As illustrated in Figure 2, if a model answers the ground-truth question correctly but the adversarial one incorrectly, it receives no credit. This mechanism ensures robustness and prevents inflated scores due to data contamination.

The drop in accuracy in Table 3 highlights the vulnerability of models to adversarial perturbations. The scores reported elsewhere in the paper are “accuracy with adversarial pairs” by default.

Ensuring Robustness and Quality of Adversarial QA Pairs

(Addresses W4: Concerns about adversarial question quality and expert validation)

W4: The adversarial question design is creative, but it has issues as well. Within clinical data. there are always cases that have co-occurrence of multiple forms of the disease, but in the original caption, we only have one of them as according to a clinician it is the important one. In this regard, when we create adversarial questions, this important fact has been ignored. So, each question actually needs to be validated by a medical expert. I have seen that 100 samples were examined by experts, but that number is significantly small compared to the size of the dataset. In fact, the paper lacks a thorough and careful expert study to ensure correctness.

The metadata underlying our dataset comes from expert-verified sources such as ChestX-ray14 and MedICaT, which inherently account for the co-occurrence of multiple diseases in a single medical image. After preprocessing, 40% of the images marked as abnormal in our dataset feature multiple conditions. This information was derived from:

• Multiple labels in ChestXray14.
• The presence of several disease names in MedICaT captions.

To comprehensively evaluate models, we generated two QA pairs for each condition in a single image. Adversarial QA pairs were constructed by selecting random attributes within the same organ-modality category (e.g., conditions for a Chest X-ray were sourced from conditions identified in other Chest X-rays). We ensured that adversarial attributes did not overlap with the existing condition(s) in the image. This careful design guarantees that adversarial QA pairs are logically consistent and robust.

If a condition was omitted in the original data source, the likelihood of selecting it as a hallucinated attribute was minimal - estimated at 0.53% across all [organ, modality] combinations (Table 2).

To further validate the correctness of metadata and QA pairs, we engaged two medical experts - a postdoctoral researcher and a graduate student from a U.S. medical school - to review 100 randomly sampled metadata entries and the corresponding 1,090 curated QA pairs (including adversarial QA pairs). The expert reviewers verified two aspects for each sample: (i) whether the “condition: location” field in the curated metadata is valid given the original caption and (ii) if the question-answer pairs are valid given the original [image, caption]. Their evaluations yielded an average metadata accuracy of 94.00% and QA pair accuracy of 97.79%, underscoring the reliability of our dataset.

3. Proposed Categorization of Models by Training Data Transparency

To address your concerns further, we propose a framework to enhance the transparency of model evaluations on ProbMed by categorizing models based on the disclosure and nature of their training data. This approach aims to provide clearer insights into the potential influence of data exposure on model performance and promote fairer comparisons. For example, a potential categorization could be:

a. Proprietary/Undisclosed Training Data

Models in this category rely on proprietary datasets with undisclosed training sources. This lack of transparency introduces the highest uncertainty regarding data contamination. Examples include:

• OpenAI's GPT models
• Google's Gemini models

b. Partially Disclosed Training Data

These models provide partial transparency regarding their training data, listing some sources but not the full scope. Examples include:

• Meta's LLaMA models trained on hybrid datasets that combine publicly available and proprietary sources

c. Fully Open Source with Known Data Sources

This category includes models that are fully open source, with complete disclosure of training datasets and pipelines. Examples include:

• Academic models published with detailed training protocols, such as BiomedGPT

d. Fine-Tuned Models (Domain-Specific)

These models are fine-tuned on specialized datasets after their general pretraining phase, often targeting specific domains such as medicine. Examples include:

• LLaVA-Med and Med-Flamingo, fine-tuned LMMs on medical QA datasets

Implementation on the Leaderboard

To ensure fairness and transparency, we propose implementing this categorization on the ProbMed leaderboard:

1. Label each model with its training data category alongside its performance metrics, especially for proprietary models that lack of disclosure of training resources.
2. Indicate any fine-tuning on medical datasets or domain-specific datasets to differentiate performance improvements due to further customization.

How This Addresses Data Contamination Concerns

This categorization will not eliminate the possibility of data contamination (which is nearly impossible for all publicly available datasets and benchmarks) but will:

• Provide transparency to users of the benchmark about the degree of training data overlap risk.
• Facilitate fairer comparisons by contextualizing performance metrics based on data exposure.
• Encourage future discussions and advancements in benchmarking practices to minimize bias and contamination.

We believe this proposal strikes a balance between acknowledging inherent challenges in evaluating LLMs and ensuring robust and transparent comparisons. If additional details or modifications are needed, we would be happy to refine this approach further.

Thank you for your follow-up comments and for engaging with our work. We appreciate the opportunity to address your concerns further and clarify the new plots.

1. Addressing Data Contamination Concerns

We acknowledge the difficulty in fully eliminating the possibility of data contamination, particularly for proprietary API-based models whose pretraining data is not disclosed. However, the design of ProbMed mitigates this issue through our rigorous adversarial QA framework.

Specifically:

• Each ground-truth QA pair is paired with an adversarial counterpart constructed from random selection, requiring models to correctly answer both the ground-truth and adversarial questions to earn credit. This approach penalizes reliance on memorized patterns and ensures robustness against potential contamination.

• As shown in Table 3, all models experience a significant accuracy drop (on average 29.97%) across all models when adversarial pairs are introduced, indicating that performance is not solely reliant on prior exposure to the data but rather on their reasoning and generalization abilities. If a model were simply memorizing ground-truth infomation, it would not have exhibited the worse-than-random performance we observed for specialized questions.

It is also important to note that the use of publicly available sources as data, even for test sets, is not unique to ProbMed. Many well-cited benchmarks face similar challenges:

• [1] sources data from publicly available materials, including practice questions for exams like the Graduate Record Examination (GRE) and the United States Medical Licensing Examination (USMLE).
• [2] uses data drawn from online resources, textbooks, and lecture materials.
• Another highly regarded medical benchmark, [3], derives its test sets from pathology textbooks and online digital libraries.

In this era, any publicly available source could potentially serve as training material for LMMs, particularly proprietary API-based models. While this reality poses inherent limitations, it reinforces the importance of rigorous evaluation frameworks like ProbMed to ensure fairness, minimize bias, and accurately measure models' reasoning capabilities.

2. Clarifying the Total Number of Responses in Distribution Plots

The discrepancy in the total number of responses across different models arises from invalid answers, such as when models refuse to answer or generate irrelevant content. These invalid responses are considered incorrect in our accuracy calculation and excluded from valid responses' distribution plots.

This phenomenon is most common among API-based proprietary models. For example, in Section 4.2.2, we analyze errors for models such as GPT-4V and Gemini Pro. Table 5 in the paper provides a breakdown of errors into categories, including:

• Deny Ground Truth: Incorrectly answering "no" to ground-truth questions.
• Accept Hallucination: Incorrectly answering "yes" to adversarial questions.
• Reject to Answer: Cases where the model either refuses to answer or generates irrelevant content.

The distribution plots focus solely on direct valid responses to better illustrate the models' performance characteristics. When models produced full-sentence responses instead of simple binary answers, we post-processed these outputs by letting GPT-4 decide the correctness of the answer. We hope this explanation clarifies the observed differences in the total number of responses across models.

We appreciate your thoughtful feedback and the opportunity to clarify these points. If there are additional questions or further clarifications needed, we are happy to provide them. Thank you again for your engagement and support.

[1] Hendrycks, Dan et al. “Measuring Massive Multitask Language Understanding.” ArXiv abs/2009.03300 (2020): n. pag.

[2] Yue, Xiang et al. “MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI.” 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023): 9556-9567.

[3] He, X., Zhang, Y., Mou, L., Xing, E.P., & Xie, P. (2020). PathVQA: 30000+ Questions for Medical Visual Question Answering. ArXiv, abs/2003.10286.

Thank you for your constructive feedback and for continuing to engage with our work. We value your thoughtful suggestions and your acknowledgment of the dataset's contributions to the field. Below, we provide additional clarifications and new result analysis (as suggested) to address your concerns about dataset bias and contamination risks.

1. Empirical Experimentation with Bias Analysis

In response to your suggestion and within the constraints of time and computational resources, we conducted an additional experiment to empirically examine the potential bias introduced by dataset overlap. Specifically, we fine-tuned Phi-3.5-Vision, a lightweight 4.2B open-source multimodal model with strong instruction-following capability, using data from the MedICaT dataset and conducted the following comparative analysis:

Training Settings:

(i) Clean data: 20K clean examples from MedICaT, ensuring no overlap with the 4.5K MedICaT images included in ProbMed.

(ii) Contaminated Data: 20K examples that include all the 4.5K images used in constructing ProbMed (22.5% overlap).

Evaluation Data:

(i) All models were evaluated on the 41.8K VQA pairs in ProbMed curated from MedICaT.

(ii) To show the effectiveness of the adversarial evaluation, we also evaluated the models without adversarial pairs for comparison.

Findings:

• The comparison of zero-shot vs. in-domain fine-tuning on unseen MedICaT data reflects a typical and expected outcome. In-domain fine-tuning inherently helps the model align better with the task-specific distributions, as demonstrated by the jump from 43.97% (zero-shot) to 46.21% (Fine-tuned on clean, unseen data).

• However, compared with fine-tuning on the clean, unseen MediCaT data, fine-tuning directly on the contaminated data only brings 1.65% improvement on ProbMed, which shows the robustness of our adversarial evaluation benchmark ProbMed in evaluating the reliability of LMMs in MedVQA. ProbMed’s rigorous adversarial QA framework is designed to stress-test reasoning and generalization of LMMs (e.g., the LMM has to answer all positive and negative questions about the same image correctly to count as a hit), reducing reliance on memorized patterns. To further verify this, we evaluated the “contaminated” model on normal medical VQA data in ProbMed without adversarial pairs, the accuracy is as high as 72.24% (with an 11.40% absolute improvement vs. 1.65% on ProbMed).

• In summary, based on the quick evaluation of Phi-3.5-Vision model, fine-tuning a LMM within the same domain provides some advantages; while data contamination certainly leads to better performance, its impact is effectively mitigated by the adversarial evaluation mechanism in ProbMed.

Phi-3.5-vision-instruct fine-tuning parameters: Epoch = 1, lr = 4.0e-5, bs = 16, wd = 0.01, LoRA rank = 64

2. Potential Bias Discussion and Model Categorization

The categorization of LMMs based on training data sources is also very important based on the results. To further improve transparency, we propose categorizing models based on their transparency in training data. We will include a Discussion section in the revision to delve deeper into these topics, including the influence of dataset contamination and the role of benchmarks like ProbMed in advancing reliable evaluations.

ProbMed’s rigorous adversarial QA framework is designed to stress-test reasoning and generalization, reducing reliance on memorized patterns. However, we acknowledge that adversarial designs cannot fully eliminate data contamination risks, especially for proprietary models with undisclosed training datasets. While ProbMed is not a definitive measure of clinical applicability, it serves as a robust benchmark to evaluate LMM reasoning under adversarial conditions.

We appreciate your recommendation for such an analysis and recognize its value in contextualizing the strengths and limitations of ProbMed. We will aim to extend this analysis further in future work to quantify dataset bias comprehensively.

Structured Probing vs. Open-Ended Evaluation

(Addresses W3 and Q3: Why closed-ended tasks were chosen and their scalability)

W3: ProbMed uses closed-ended questions and lacks open-ended tasks, limiting the dataset's comprehensiveness for evaluation.

Q3: What is the impact of adding open-ended tasks on the model's performance? How can report generation or open-ended question-answering be incorporated into ProbMed?

Open-ended tasks, such as medical report generation, pose significant challenges for LMMs due to hallucinations and the complexity of evaluating free-text outputs. These tasks often require labor-intensive, subjective human evaluation, making them difficult to scale [3]. In contrast, our approach emphasizes structured binary (yes/no) questions, which offer a scalable, deterministic framework for fine-grained probing. This structured approach ensures consistency and stability while effectively evaluating LMMs in clinically relevant scenarios. Additionally, our dataset includes inline captions for each image, enabling future research to extend evaluation to open-ended tasks if desired.

Rationale Behind the Ablation Study

(Addresses Q4: Insights from augmenting open-source models with GPT-4 visual reasoning)

Q4: Could GPT-4-generated visual descriptions introduce bias or hallucinations?

The ablation study in Section 4.3 investigates how proprietary models like GPT-4 can improve the performance of open-source, domain-specialized models. While GPT-4's performance remains imperfect, with limitations like hallucinations, it consistently outperformed other models (Table 4, Appendix A). Our findings reveal an average accuracy improvement of 9.44% across all question categories when open-source models are augmented with GPT-4's visual reasoning capabilities. This highlights the critical role of enhancing visual understanding to address bottlenecks in the performance of existing medical LMMs.

We hope this addresses your concerns. Please let us know if further clarification is needed.

[1] Yue, Xiang et al. “MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI.” 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023): 9556-9567.

[2] Hendrycks, Dan et al. “Measuring Massive Multitask Language Understanding.” ArXiv abs/2009.03300 (2020): n. pag.

[3] Chaves, Juan Manuel Zambrano, et al. "Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation." arXiv preprint arXiv:2403.08002 (2024).

Thank you for your thoughtful and constructive feedback. Below, we address your comments and questions in detail.

Dataset Balance and Generalization Across Modalities

(Addresses W1 and Q2: Concerns about dataset balance and verification across modalities)

Q2: Does class imbalance in the dataset affect the model's generalization ability?

Our benchmark provides a comprehensive and rigorous evaluation of 57k VQA pairs derived from over 6k medical images spanning twelve [organ, modality] combinations. While public Chest X-ray data is more abundant, the dataset features balanced binary answers. Similar to widely cited benchmark papers in the field [1,2], we focused on pure evaluation without model fine-tuning, adhering to the standard research practice of evaluating pre-trained models. Hence, our approach does not compromise the models' generalization ability.

W1: Has the model's performance across different modalities been experimentally verified?

Q2: Validation across multiple modalities is recommended to assess the impact of imbalanced data distribution on each modality.

Breakdown results for each of the twelve [organ, modality] combinations are provided in Appendix A. Consistent with these subsets, our analysis in Section 4 demonstrates robust findings across categories. Furthermore, the ablation study (4.3.1 and Appendix C) reveals universal performance improvements across all combinations, further highlighting the robustness of our evaluation design.

Metadata Extraction and Adversarial QA Construction

(Addresses W2, W4, and Q1: Hallucination prevention, adversarial question quality, and expert validation)

W2: How does the study prevent hallucinations when using GPT-4 for caption analysis, abnormality identification, and positional descriptions through few-shot prompting?

The metadata used to construct our dataset originates from expert-verified ChestX-ray14 and MedICaT datasets. GPT-4's role was limited to extracting condition names and their corresponding positions from MedICaT captions through few-shot prompting (Appendix D). This textual extraction task is significantly simpler than diagnosing from medical images and poses minimal risks of hallucination.

W4: Constructing adversarial samples by selecting random entities (such as alternative organs, modalities, or hallucinated conditions with "no" answers) may increase testing difficulty but could also introduce mistakes, resulting in samples that aren’t truly adversarial.

Q1: Is the construction of adversarial negative samples reasonable? Could additional strategies be introduced to ensure the validity of these adversarial samples?

To construct adversarial QA pairs, we employed a rigorous process:

1. General Questions: Adversarial attributes (e.g., organs or modalities) were logically invalidated by filtering the original [image, caption] pairs to ensure the image contained a single organ of a single modality.
2. Specialized Questions: Adversarial attributes were selected within the same [organ, modality] category to enhance challenge and robustness while ensuring no overlap with existing condition(s) in the image.

By design, adversarial QA pairs in our dataset are logically consistent and challenging yet robust. That being said, if a disease was indeed omitted in the original data source, the likelihood of the omitted disease being selected as a hallucinated attribute is minimal - estimated at 0.53% on average across all [organ, modality] combinations, according to statistics in Table 2.

An expert verification process further confirmed metadata and QA pair quality (Section 3.3). Two annotators - a postdoctoral researcher and a graduate student from a U.S. medical school - reviewed 100 randomly sampled metadata and 1,090 corresponding QA pairs (including adversarial QA pairs). The annotators were instructed to verify two aspects for each sample: (i) whether the “condition: location” field in the curated metadata is valid given the original caption and (ii) if the question-answer pairs are valid given the original [image, caption]. Their evaluations yielded an average metadata accuracy of 94.00% and QA pair accuracy of 97.79%, underscoring the reliability of our dataset.

Dear Reviewer 6yHr,

Thank you for your valuable feedback. We have clarified the dataset balance, adversarial QA construction process and validation, and highlighted the structured evaluation framework's scalability.

We kindly invite you to revisit our paper in light of these updates. We hope these improvements address your concerns and encourage a reevaluation of your rating.

Best,

The Authors

Thank you for your continued engagement. We would like to clarify the scope and contributions of our work since there seems to be some confusion about that. ProbMed is not intended to replace existing benchmarks for MedVQA but to address a critical gap in reliably evaluating LMMs within MedVQA. Its unique incorporation of adversarial negation pairs for each question-answer pair ensures diagnostic specificity and reliability, setting ProbMed apart from the existing closed-ended MedVQA framework. For instance, Table 3 highlights significant performance drops when our adversarial pairs are introduced to the VQA-RAD test set, demonstrating the limitations of current models that reported near-saturated results in earlier studies [1, 2]. Additionally, it enables categorical accuracy assessments across various procedural diagnostic dimensions for each image and allows fine-grained analysis.

Our structured approach, using closed-ended questions, ensures quantifiable and consistent evaluations, revealing insights that less controlled settings might miss. While ProbMed does not currently encompass the complexities of open-ended real-world clinical scenarios, it lays a critical foundation for future research that can extend to include such elements. Our goal is to enhance the understanding of LMM capabilities and limitations in medical imaging, complementing - not replacing - other benchmarks and paving the way for more clinically realistic tasks as technologies evolve. We hope this clarifies the scope and contributions of our work.

[1] Li, Chunyuan, et al. "Llava-med: Training a large language-and-vision assistant for biomedicine in one day." Advances in Neural Information Processing Systems 36 (2024).

[2] Moor, Michael, et al. "Med-flamingo: a multimodal medical few-shot learner." Machine Learning for Health (ML4H). PMLR, 2023.