MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine

Question 4 & Weakness 4 (Evaluation on Multigranular-Specific Tasks)

do you have plans to test your dataset on tasks specifically designed for multigranular information? How can you demonstrate the effectiveness of your dataset's detailed annotations in improving model performance on such tasks?

To the best of our knowledge, there is no existing benchmark specifically designed to evaluate multigranular information generation. We have provided both qualitative and quantitative results on the report generation task using the MIMIC-CXR dataset, which is challenging and requires the ability to generate multi-granular answers for chest radiology. Specifically, we finetune the baseline models of LLaVA-Tri and LLaVA-pp [3] on a small set of 10k samples in our proposed dataset due to limited time. The results are summarized in the table below:

Here is a qualitative comparison of a sample with study ID 67 from the MIMIC-CXR dataset:

• Result (trained on multigranular annotations): The lungs are well expanded and clear. The cardiomediastinal silhouette, hilar contours, and pleural surfaces are normal. No pleural effusion or pneumothorax is present.
• Result (not trained on multigranular annotations): The cardiomediastinal silhouette is normal. There is no pleural effusion or pneumothorax. There is no focal lung consolidation. There is no acute osseous abnormality.
• Ground Truth: The lungs are well inflated and clear. The cardiomediastinal silhouette, hilar contours, and pleural surfaces are normal. There is no pleural effusion or pneumothorax.

The quantitative results show that fine-tuning with a small set of our dataset already significantly improves generation performance across the tested models. The qualitative comparison further illustrates that training on multigranular annotations helps the model generate more fine-grained and professional text, aligning more closely with human reports.

Question 5 & Weakness 5 (Addressing Unique Medical Image Presentations in RAG )

How does your RAG approach handle the specificity and variability of individual medical image descriptions?

Our RAG approach can handle the diversity and specificity inherent in individual medical cases by employing a comprehensive knowledge base from extensive medical literature, including 23.9 million biomedical articles from PubMed, covering various diseases and their known manifestations, such as those related to atelectasis. For diseases with diverse descriptions available in these articles, our knowledge base is designed to preserve and incorporate all relevant descriptions in their entirety.

For instance, we are able to retrieve articles like "Atelectasis: mechanisms, diagnosis and management [4]" detail that atelectasis may occur in three ways: (i) airway obstruction; (ii) compression of parenchyma by extrathoracic, intrathoracic, or chest wall processes; and (iii) increased surface tension in alveoli and bronchioles. Subsequently, we utilize the LLM to align the retrieved texts describing the three manifestations of atelectasis with the visual features extracted from the medical images. Through this image-text alignment, the LLM generates annotations that accurately reflect each medical case's unique presentation, effectively capturing the specific disease manifestations.

[3] Rasheed, H., et al. "Llava++: Extending visual capabilities with llama-3 and phi-3 (2024)."

[4] Peroni, D. G., and A. L. Boner. "Atelectasis: mechanisms, diagnosis and management." Paediatric respiratory reviews 1.3 (2000): 274-278.

Weakness 1 & Question 1 (Handling of Visual Data Without ROIs)

For medical images that lack explicit ROI annotations, how does your pipeline construct the corresponding multimodal data pairs?

For medical images without explicit ROI annotations, our pipeline leverages expert models to generate ROIs. Details about the expert models used for ROI generation can be found in Table 6 in Appendix D of original manuscript.

Weakness 2 & Question 2 (Assessing the Impact of Multigranular Information on Med-MLLMs)

Considering that ROIs and multigranular annotations are central to your dataset, have you conducted experiments to evaluate how these features specifically affect the performance of medical MLLMs?

Thank you for your insightful question. We have conducted detailed experiments to assess the impact of incorporating multigranular information on the performance of medical MLLMs. The results, presented in Table 4 and discussed in Section 4.2 of the original manuscript, demonstrate the advantages of integrating multigranular information into the training process.

To provide a detailed illustration of how incorporating metadata, ROI, and RAG modules affects generation quality, we have included comprehensive examples in Figures 3, 4, and 5 of the original paper. These examples show that incorporating these features can significantly improve the quality of the generated annotations.

Weakness 3 & Question 3 (Comparison with Existing Multigranular Benchmarks)

Could you elaborate on how MedTrinity-25M compares with existing benchmarks that involve multigranular information?

Thank you for highlighting the need for a more thorough discussion. We appreciate your suggestion to include works such as GMAI-MMBench [1] and Asclepius [2]. These are indeed valuable contributions to the field, and we will incorporate a discussion of these works along with proper citations in the updated version of our manuscript.

In addition, while GMAI-MMBench and Asclepius provide multi-granularity annotations, they do not explicitly offer detailed descriptions of lesion characteristics, nor is there explicit evidence that they capture or annotate relationships or correlations between different regions. We provide a comparison with these benchmarks that involve multigranular information in the table below:

[1] Chen, P., Ye, J., Wang, G., Li, Y., Deng, Z., Li, W., ... & Qiao, Y. (2024). Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. arXiv preprint arXiv:2408.03361.

[2] Wang, W., Su, Y., Huan, J., Liu, J., Chen, W., Zhang, Y., ... & Lyu, M. R. (2024). Asclepius: A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models. arXiv preprint arXiv:2402.11217.

Weakness 1 (Data Quality)

... MedTrinity-25M are generated by the proposed automated pipeline, raising concerns about the accuracy of the generated text ... an expert-based evaluation .. resulted in an accuracy score of 85%.

We appreciate your concerns regarding the accuracy of the generated text, as reflected in the expert evaluation score of 85%.

This score represents the average across five evaluation aspects: modality, organ detection, region of interest (ROI) analysis, lesion texture, and region-wise correlations. While near-perfect scores were achieved in modality, organ detection, and ROI analysis, the accuracy in lesion texture and region-wise correlations may be less satisfactory.

Upon examining cases with relatively lower scores, we found that inaccuracies primarily stemmed from the omission of certain terminologies. We hypothesize that this issue arises from gaps in our medical knowledge base. Since the knowledge base is constructed from public resources (e.g., PubMed), it inherently reflects biases—common diseases are often described in greater detail, whereas rare diseases in specific domains may have incomplete or coarse descriptions. These gaps can result in generated descriptions that omit crucial terminologies. To address this, we plan to expand our knowledge base with a more comprehensive corpus that covers a broader range of diseases, which we believe will enhance the accuracy of the generated descriptions.

Although the results for lesion texture and region-wise correlations are slightly less satisfactory, our dataset still demonstrated significant contributions to the performance of multimodal learning models. As shown in Table 3 of the original manuscript, it improved performance by 10.8%, 6.1%, and 8.3% on the VQA-RAD, SLAKE, and PathVQA datasets, respectively, despite the presence of some annotation noise.

Weakness 3 (Result on Report Generation)

... overlooking its potential application to ... such as visual report generation

Following your suggestion, we conducted ablation studies to evaluate the effectiveness of multigranular alignment in report generation on the MIMIC-CXR dataset. Specifically, we finetune the baseline models of LLaVA-Tri and LLaVA-pp [1] on a small set of 10k samples in our proposed dataset due to limited time. The results are summarized in the table below:

As shown in the table above, finetuning with a small set of our dataset already significantly improves performance across the tested models. For LLaVA-Tri, BLEU-1 increased from 22.2 to 28.2, BLEU-4 from 1.0 to 6.9, and BERT Score from 20.1 to 32.4. For LLaVA-pp, BLEU-1 rose from 16.8 to 19.3 and BERT Score from 19.5 to 23.6. These results demonstrate the potential of multigranular alignment in enhancing report generation.

[1] Rasheed, H., et al. "Llava++: Extending visual capabilities with llama-3 and phi-3 (2024)."

Question 1 (Metadata)

...what approach is taken if the metadata lacks organ or disease labels?

To ensure that necessary information is available, we source metadata from various and easily accessible resources, including publications and websites related to the dataset. This metadata serves as the basis for retrieving detailed knowledge.

In rare cases where the disease type is missing in metadata, we utilize the available information to retrieve relevant knowledge. For example, if the metadata does not specify a disease like a brain tumor but indicates that the image is of the brain, we use this information to retrieve knowledge from our knowledge base. The retrieved knowledge will still provide sufficient detail about various brain diseases, not limited to brain tumors.

Dear Reviewer d16k,

We sincerely appreciate your review and have carefully considered each of your questions. Detailed responses are provided in the rebuttal, and we welcome any further questions or concerns you may have.

Question 2 & Question 3 (Revision of Table 3)

...I suggested distinguishing the results presented

Thanks for your suggestion. We have revised the presentation of results in Table 3 of the original manuscript by the following modifications:

1. distinguishing between models that are fine-tuned on the corresponding training sets and those evaluated directly without fine-tuning.
2. categorizing the result into two groups for CLIP-based and non-CLIP-based models, respectively.
3. explicitly indicating the number of fine-tuning epochs of LLaVA-Med and LLaVA-Tri (3 epoches for all fine-tuned methods).

Method Fine-tuned on the Training Set of the VQA Benchmark

CLIP-based

non-CLIP-based

Method Not Fine-tuned on the Training Set of the VQA Benchmark

Question 4 (Explanation of the "open" and "close" terms)

What do "open" and "close" specifically refer to?

Thank you for your question. In the context of the Visual Question Answering (VQA) tasks in our tables, the terms "open" and "close" refer to different question-and-answer formats:

• Open: Open-ended questions where the model generates free-form text responses without predefined answer options.
• Close: Closed-ended questions, such as multiple-choice or yes/no questions, where the model selects from a predefined set of possible answers.

We have updated the table captions and the related text in the results section of our manuscript to clearly define these terms.

Weakness 1 (Data Quality)

"The quality of the generated image-text data may not be sufficiently high..."

Thank you for your detailed and insightful feedback. We have thoroughly reviewed the prompts and metadata provided to the MLLM for annotation generation and are confident that they are correct. For your reference, we include our metadata for the ct_rate and quilt_1m datasets below:

• ct_rate: This is a chest image from CT volume with {disease} in green bounding boxes.
• quilt_1m: This is an image of histological samples of various human cancers. Each type of cell nuclei is color-coded and outlined with a bounding box. Larger bounding boxes are used to group clusters of the same type of cell nuclei.

We further provide an anonymous code link for illustrating the correctness of the metadata provided to the MLLM.

Therefore, we believe that the observed misidentification of modalities likely originates from the MLLM itself, which may have rewritten or misinterpreted the provided metadata. This suggests a critical limitation of MLLMs that warrants further investigation, though it is beyond the scope of this paper. Now, we will manually correct all the wrong modality in our data to ensure the correctness. And we will address this misinterpretation in future work and clarify it in the paper.

While we acknowledge that annotations generated by MLLMs may contain some noise compared to human annotations, we strongly assert that MedTrinity-25M is a highly valuable resource for advancing medical MLLMs. For instance, as shown in Table 3 of the original manuscript, our dataset led to performance improvements of 10.8%, 6.1%, and 8.3% on VQA-RAD, SLAKE, and PathVQA, respectively, even in the presence of some annotation noise.

Weakness 2 (Result on MMMU)

A broader range of specialized benchmarks, such as the health subset of MMMU, could provide a more robust comparison of the multimodal large models’ performance.

Thank you for your suggestion. Accordingly, we compared the performance of LLaVA-Med and the proposed LLaVA-Tri on the health and medicine subsets of MMMU. We report the micro-average accuracy scores in the following table.

As shown in the table, LLaVA-Tri significantly outperforms LLaVA-Med in all evaluated areas, indicating the effectiveness of training the model with MedTrinity-25M.

The details of this experiment are as follows:

Question 1 (Choice of GPT-4V)

Why did you choose to use GPT-4V to generate the supervised fine-tuning data for the fine-tuning process?

The choice of MLLMs is flexible and not the main focus of our paper. Our primary contribution lies in developing an automated pipeline to scale up multimodal data by generating multigranular annotations from unpaired image inputs. In our pipeline, MLLMs are tasked with producing multigranular annotations based on complex instructions. We selected GPT-4V for this purpose due to its strong ability to follow detailed instructions and generate comprehensive outputs [1]. However, it is important to note that while GPT-4V was chosen, the pipeline is designed to be flexible and can readily incorporate other MLLMs as needed.

[1] Peng, Baolin, et al. "Instruction tuning with gpt-4." arXiv preprint arXiv:2304.03277 (2023).

Dear Reviewer FQpc,

We sincerely appreciate your review. We have carefully considered each of your questions and provide detailed responses in the rebuttal. Please let us know if you have any further questions or concerns.

Dear Reviewer FQpc,

We sincerely appreciate your thoughtful review and valuable feedback. We have carefully addressed each of your questions and provided detailed responses in the rebuttal. If our responses address your concerns, could you please consider raising your score?

I am a researcher in the field of medical image analysis, but I am not a clinical expert. However, medical data must be rigorous and accurate.

For Weakness 1: So far, there are many methods for generating medical multimodal data based on MLLMs，such as，BioMedGPT[1]，Huatuo-Vision[2], MedDr[3]. I think this paper present insufficitent novelty. Furthermore, I believe the key challenge lies in addressing/avoiding the errors present in the data generated by MLLMs.

For Weakness 2: The Health & Medical track of MMMU consists of 5 parts, as referenced in Huatuo-Vision. However, the LLaVA-Med and LLaMA-Tri models demonstrate very low accuracy in two of them. The same metrics in Huatuo-Vision are more significant than yours. I believe this does not sufficiently prove the high quality and effectiveness of the data.

For Question 1: Given the current lack of MLLMs (Multimodal Large Language Models) that meet sufficiently high standards in the medical field, selecting an appropriate MLLM is a critical step in the entire pipeline. If GPT-4 is not used and another model, such as LLaMA 3, is chosen, its knowledge base in the medical domain is insufficient (as demonstrated by our previous experiences). Avoiding hallucinations in large models is a key aspect throughout the pipeline. As noted in Weakness 1, solely constraining the pipeline through prompts is inadequate, which is a common issue in all GPT-generated multimodal medical datasets.

Therefore, I maintain my score, acknowledging the efforts made by the authors in cleaning large-scale data.

Reference:

[1] A generalist vision--language foundation model for diverse biomedical tasks.

[2] HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale.

[3] MedDr: Diagnosis-Guided Bootstrapping for Large-Scale Medical Vision-Language Learning.

Weakness 1 & Question 4 (Scalability)

...domain-specific models may limit scalability when handling new modalities or emerging diseases.

Thank you for your insightful comment. Our pipeline is designed to be inherently scalable for generating Regions of Interest (ROIs) for medical images by leveraging freely available expert grounding models. For images from novel modalities or emerging diseases, we can adopt universal grounding models, such as the Segment Anything Model (SAM), to annotate ROI in a zero-shot manner. If needed, these models can be fine-tuned efficiently using parameter-efficient fine-tuning (PEFT) methods with only a small number of samples from new domains in a parallel manner, ensuring scalability of the grounding module to underrepresented diseases or modalities.

Currently, we have opted for domain-specific models over universal models to enhance the accuracy of ROI generation. However, it is important to emphasize that this is not the central focus of our paper. Instead, the core contribution of our work lies in the development of an automated pipeline for generating multigranular visual and textual annotations from images. The central emphasis is on the scalability and versatility of the pipeline, which can seamlessly integrate either specialized or universal models as required.

Weakness 2 & Question 1 (Data Quality)

How can a high standard of automated labeling be ensured? ... has it been validated by human experts?

We ensure the quality of the generated data through our pipeline design and quantitative validation.

In our pipeline design, we decompose the generation of multigranular annotations into 2 steps: 1) We collect trustworthy metadata as a query to retrieve information from a comprehensive medical knowledge base (e.g., from professional resources such as PubMed), obtaining high-quality expert knowledge. 2) We use ROIs extracted from expert domain-specific models as constraints, prompting the caption model to match the texture of lesion areas with the corresponding descriptions from the retrieved medical knowledge to generate high-quality annotations.

In quantitative validation, we conduct human expert evaluation and LLM evaluation. As shown in Table 2 of the original manuscript, our dataset achieves scores of 0.85 and 0.86 out of 1.00 in expert and LLM evaluations, respectively, with modality, organ detection, and ROI analysis nearing perfect scores. These quantitative metrics indicate the high quality of our dataset.

Question 2 (Comparison with Expert-Labeled Datasets)

Has a comparison with expert-labeled datasets been considered to further quantify the quality of automated labeling?

As shown in Table 2 of the original manuscript, our dataset achieves scores of 0.85 and 0.86 out of 1.00 in expert and LLM evaluations, respectively. This suggests that its quality is comparable to that of expert-labeled datasets. Furthermore, in Figure 1, we provide qualitative examples that directly compare our dataset with expert-labeled datasets. These examples demonstrate that our dataset covers more aspects and provides more comprehensive information than the expert-labeled datasets MIMIC-CXR and SLAKE, highlighting its increased level of detail. Additionally, Figure 8(d) compares the average word count of text descriptions between our dataset and several expert-labeled datasets. Our dataset exhibits a significantly higher word count, indicating greater richness.

Weakness 3 & Question 3 (Potential Bias)

How does the dataset address potential biases in source data?

We appreciate your thorough concerns regarding the potential biases in data distribution. Since our method aims to construct large-scale multimodal datasets by assembling existing public medical image datasets, our dataset inevitably inherits any potential biases present in the original public data.

For public medical image datasets with biases in demographics and disease distribution, we plan to implement two strategies to address these biases. First, we will uniformly sample a high-quality subset from MedTrinity-25M to reduce existing biases. Second, we will utilize our automated pipeline to generate additional data for rare diseases and underrepresented demographics, aiming to achieve a more balanced and comprehensive dataset.

Weakness 4 & Question 5 (Test Set Leakage)

How to ensure that MedTrinity-25M does not contain data from test set datasets?

Thank you for bringing up this important concern about potential test set leakage. As shown in Table 5 of Appendix A, we verify that MedTrinity-25M includes only the training sets of VQA-RAD, PathVQA, and SLAKE, and does not contain any data from the validation or test sets of these datasets.

Dear Reviewer jxrU,

Thank you for your review. We have carefully addressed each of your questions in detail within the rebuttal and would appreciate any further feedback you may have.

For Q5:

Table 5 only demonstrates that MEDTRINITY-25M includes the training sets of VQA-RAD, PathVQA, and SLAKE, but it does not confirm the absence of these datasets in the test set. Given that the dataset spans over 20 sources and LLaVA-Tri achieves 99% accuracy on PathVQA, have you considered conducting statistical experiments to verify this point?

In our experiments, we have thoroughly reviewed the data sources for VQA-RAD, PathVQA, and SLAKE, as detailed below:

We ensured that the data sources for VQA-RAD and PathVQA do not overlap with those in MedTrinity-25M, as we have not included MedPix or the PEIR Digital Library as data sources. Furthermore, we confirmed that our other data sources do not include data from VQA-RAD or PathVQA. However, SLAKE includes two overlapping sources: MSD and CHAOS. As noted in Table 5 of Appendix A in our original paper, the SAMMed-20M dataset [6] is included as a data source. This dataset integrates multiple sources, including MSD and CHAOS. To address this issue, in our experiments, we trained our model on a subset of MedTrinity-25M that excludes data from MSD and CHAOS. This ensures that the test sets from these datasets were not present in the training data. We appreciate your pointing this out, and we will clarify this adjustment in the revised version of the paper.

References:

1. MedPix: https://medpix.nlm.nih.gov/
2. Jones, Kristopher N., et al. "PEIR digital library: Online resources and authoring system." Proceedings of the AMIA Symposium. American Medical Informatics Association, 2001.
3. Antonelli, Michela, et al. "The medical segmentation decathlon." Nature communications 13.1 (2022): 4128.
4. Wang, Xiaosong, et al. "Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases." IEEE CVPR. Vol. 7. sn, 2017.
5. Kavur, A. Emre, et al. "CHAOS challenge-combined (CT-MR) healthy abdominal organ segmentation." Medical Image Analysis 69 (2021): 101950.
6. Ye, Jin, et al. "Sa-med2d-20m dataset: Segment anything in 2d medical imaging with 20 million masks." arXiv preprint arXiv:2311.11969 (2023).