Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks

We are immensely grateful to you for recognizing the importance and novelty of our work. Thanks for your precious comments.

W1: "Human validation of the automatically generated annotations is needed to prove the quality of the dataset."

Thanks for your suggestion. We have conducted human validation for the automatically annotated data, as detailed in the Common Questions Q2.

W2: "The paper lacks comprehensive comparisons in report generation with state-of-the-art methods, ..."

Thanks for your advice. We have performed additional comparisons in report generation on ReXrank with official splits and metrics. The tables below demonstrate the results.

Results on IU-Xray benchmark.

Results on MIMIC-CXR benchmark.

In the table, Bold values indicate the best results, and '*' indicates the second-best results.

While our model's performance on certain metrics does not surpass all the models on the ReXrank benchmark, we believe the following points highlight the contribution of our approach:

1. Comparable Results with Specialist Models: Our model achieves results that are comparable to specialist models, such as RGRG [2] and RaDialog [3], which are specifically trained on report generation tasks for chest X-ray datasets. However, unlike these specialist models, our model is designed as a generalist framework capable of handling multiple tasks, such as VQA and diagnosis, and supporting more modalities.
2. Competitive Performance Among Generalist Models: Within the generalist model category, our model ranks second. It accomplishes superior scores in certain metrics compared to MedVersa [1], and outperformed other generalists including RadFM [4] and BioMedGPT [5]. In comparison with MedVersa [1], our model offers broader versatility in report generation for different modalities of various organs, including X-ray, CT and MRI scans of the brain, spine, abdomen, and pelvis, in addition to chest X-rays.
3. Contributions Beyond Benchmark Performance: While this benchmark focuses solely on chest X-ray report generation, our model introduces unique capabilities for region-specific tasks.

We have updated the results on ReXrank benchmark in Table 9 of Appendix C.1.2.

We are deeply grateful for your recognition of our paper's performance and potential academic impact. Thanks for your constructive comments.

Weaknesses

W1: "While the proposed dataset is novel and important to the field, ..."

Thanks for your careful review. We appreciate your recognition of the significance of our proposed dataset. We would like to highlight and clarify the contributions of our research.

Our key contribution lies in enhancing the region-centric ability into the medical generalist models, which remains unexplored in previous works. Considering the limitation of existing works in detecting or reasoning about specific regions within a medical scan, we contributed to enhancing spatial awareness of medical generalist models and performing specific adaptation on region-centric tasks. This significantly improves the model's ability to generate high-quality and interpretable outputs in complex medical scenarios. Moreover, the proposed regional CoT strategy enables interpretable and region-specific reasoning, inspired by the diagnostic process of clinicians who rely on precise and localized information for diagnosis. By incorporating this strategy, the regional knowledge within our model can be leveraged to further enhance downstream tasks like VQA and diagnosis, demonstrating stronger potential in clinical applications.

W2: "Similarly, it seems that the baselines used in the evaluation are not all fine-tuned..."

Thanks for your advice. We have evaluated the finetuning results with two baselines, Med-Flamingo and LLaVA-Med, as demonstrated in the Common Questions Q1.

W3: "Another concern about this paper is the quality of the data..."

Thanks for your valuable suggestion. We have conducted human validation for the automatically annotated data, as detailed in the Common Questions Q2. We will also verify the generated data to improve its quality before release.

Questions

Q1: "How is the English version of the text report/caption generated for the in-house Chinese data?..."

Thanks for your question. Actually, we did not generate English version reports for the in-house Chinese data, and vice versa. We only kept the original Chinese text due to the following considerations:

1. Aligned with real clinical scenarios: In clinical practice, it is not necessary to generate bilingual reports because they serve people who speak respective languages. Besides, the image data distributions may vary across different populations. Separating the languages allows the model to distinguish between data distributions, thereby encouraging a better understanding of the images from different populations.
2. Challenges of Translation Quality: Creating data with both languages requires either manual translation, which is labor-intensive, or automated translation with translation models or large language models. However, due to the specialized terminology in medical reports, automated translation often yields errors or unnatural phrasing. This makes it difficult to ensure the quality of the translated data and could introduce noise into the model.
3. Difference in Writing Styles: There are differences in the writing style between Chinese and English reports. Simply using translated texts for training would affect the consistency of styles and introduce noise that would limit the model’s ability to mimic the writing style of different languages. This would lead to difficulties for doctors with different native languages in fully understanding the generated reports.

Q2: "According to the “Structure Detection” section on page 5, ..."

Thanks for your insightful question.

1. Evaluation on the finetuned MLLM for structure detection: We have conducted human validation for the automatically annotated data, as detailed in the Common Questions Q2.
2. Comparison with specific detection models: We applied the finetuned MLLM for structure detection instead of training a specific detection model because the MLLM allows for flexibility in customizing the target organs. Our medical report for different scans may cover different subsets of organs, and detection models with predefined targets cannot adapt dynamically to these changing objectives. Furthermore, training multiple detection models for different organs requires extra effort in dataset preparation and model deployment. By using the finetuned MLLM, we consolidate these tasks into a single model, providing a more adaptable and efficient solution.

Q3: "...the reviewer wonders how much will the other baseline medical MLLMs improve ..."

Thanks for your question. Please refer to the Common Questions Q1 for a detailed response to this question.

Q4: "...the reviewer wonders if the data will be released as well..."

Thanks for your suggestion. Our data will be released to the public.

We are grateful for the reviewer's appreciation of our model and the paper's clarity. Thanks for your time in the review process and the insightful comments.

Weaknesses

W1: "The article’s description of the model structure in the main text is incomplete,..."

Thanks for your advice. We revised the main text in Section 4 and included additional training details.

W2: "The fairness of the model comparison is questionable..."

Thanks for the question. Certain models (LLaVA-Med [1] and BioMedGPT [2]) were marked as "fine-tuned" because the checkpoints provided have been fine-tuned on specific datasets. For LLaVA-Med [1], the fine-tuned models on VQA benchmarks, SLAKE, VQA-RAD, PathVQA, are open-sourced. For BioMedGPT [2], the fine-tuned results of these three VQA benchmarks are reported in the original paper. Although fine-tuning on targeted tasks before evaluation can often enhance performance, the fine-tuning process may reduce deployment efficiency. Our MedRegA model does not require the fine-tuning stage, and outperforms the finetuned models on overall results in a zero-shot manner.

[1] Li, Chunyuan, et al. "Llava-med: Training a large language-and-vision assistant for biomedicine in one day." Advances in Neural Information Processing Systems 36 (2024).

[2] Zhang, Kai, et al. "A generalist vision–language foundation model for diverse biomedical tasks." Nature Medicine (2024): 1-13.

W3: "...it would be insightful to know if fine-tuning on the authors’ proposed dataset could improve this..."

Thanks for your suggestions. We would like to clarify that our proposed MedRegInstruct dataset is a core contribution, as it incorporates region-centric ability into medical MLLMs, which has not been explored in previous research. Specifically, we applied an automatic labeling system to curate the grounded medical reports in the dataset. Our dataset covers comprehensive region-centric tasks, aimed at encouraging the model to focus on critical areas. Besides, these baselines were all trained using different data sources and different data construction methods, making direct comparisons challenging. Notably, the design of dataset construction is one of the key contributions to the development of MLLM.

Moreover, we have evaluated and analyzed the fine-tuning results on our proposed dataset with two baselines, Med-Flamingo and LLaVA-Med, to demonstrate the effectiveness and novelty of our proposed dataset, as demonstrated in the Common Questions Q1.

W4: "The role of the CoT component is somewhat unclear, ..."

Thanks for your suggestion. We have presented a more specific ablation study in Appendix E to further demonstrate the effectiveness of regional CoT.

W5: "While the authors claim that the proposed model can perform diagnostic tasks, ..."

Thanks for the advice. We have included detailed evaluation results and user studies for diagnostic tasks in Appendix C.1.3 to demonstrate the model performance on diagnostic tasks.

W6: "Several prior studies focus on medical image segmentation, ..."

Thanks for your careful review. We agree that some segmentation models, like MedSAM [1], possess regional knowledge that helps capture the position of structures and lesions in medical images. However, these models lack training on paired image-text data, which limits their ability to align with textual representations. As a result, they are not effective on other clinically significant tasks, such as generating medical reports. While segmentation models could be used as backbones, this would require additional alignment training with image-text pairs, which is a time-consuming process that could also deviate the original segmentation semantics. Therefore, we use a multi-modal large language model as our backbone, as it already incorporates aligned image-text knowledge.

[1] Ma, Jun, et al. "Segment anything in medical images." Nature Communications 15.1 (2024): 654.

W7: "The experimental results on report generation are insufficient, ..."

Thanks for the suggestions. We further provided user studies and result analysis in Appendix C.1.2 to validate the model on the report generation task.

Questions

Q1: "Please explain why certain models were fine-tuned in the comparison while others were not."

Thanks for your comment. Please refer to W2 for a detailed response.

Q2: "The authors state that adding CoT improves model performance..."

Thanks for your careful review. Please refer to W4 for a detailed response.

Q3: "Have the authors considered comparisons using other large models as the backbone?"

Thanks for your question. Please refer to W3 for a detailed response.

Q4: "Do the authors plan to open-source this dataset?"

Thanks for your question. The MedRegInstruct dataset will be released to the public.

We are grateful for your recognition of our paper's soundness and novelty. Thanks for your efforts in the review and valuable comments.

Weaknesses

W1: "The section of the report grounding dataset is quite confusing as to how its constructed..."

Thanks for your questions. We would like to further illustrate the construction of report grounding dataset.

To construct report grounding dataset for Chest Xrays in MIMIC-CXR dataset, no automated process is required since all the labels can be retrieved in the MIMIC-CXR and Chest ImaGenome datasets.

To construct report grounding dataset for our in-house data, we implemented two automated processes: (1) we employed InternLM to segment each report into detailed descriptions for each organ; (2) We first fine-tuned an MLLM based on the Region-Text dataset to enable it to generate regions based on organ names. Then, we input the collected images into the finetuned MLLM, and prompted it with the organs mentioned in the reports to obtain the corresponding bounding boxes. The automatically constructed part makes up approximately 10% of all the report grounding data. We also present the result of the human validation for generated data in the Common Questions Q2.

W2: "The paper claims to work across a broad set of modalities, ..."

Thanks for your question. In addition to Chest X-rays from the MIMIC-CXR public dataset, we include report grounding data constructed from our in-house data, covering more body structures such as lung, liver, and pancreas, across MRI and CT modality.

W3: "It's unclear if the comparison to existing baselines are exactly fair as the trained MedRegA has access to more data..."

Thanks for your advice. We would like to clarify that our proposed MedRegInstruct dataset is a core contribution, as it incorporates region-centric ability into medical MLLMs, which has not been explored in previous research. Specifically, we applied an automatic labeling system to curate the grounded medical reports in the dataset. Our dataset covers comprehensive region-centric tasks, aimed at encouraging the model to focus on critical areas. Besides, these baselines were all trained using different data sources and different data construction methods, making direct comparisons challenging. Notably, the design of dataset construction is one of the key contributions to the development of MLLM.

Moreover, we have evaluated the fine-tuning results on our proposed dataset with two baselines, Med-Flamingo and LLaVA-Med, to demonstrate the effectiveness and novelty of our proposed dataset, as demonstrated in the Common Questions Q1.

Thank you to the authors for the in-depth response my concerns and questions, as well as clearing up a few points I was confused with. I agree that the MedRegInstruct dataset will be a valuable contribution to the research community, and that with the additional results on attempting to tune the baselines for fair comparison that I am raising my score to an 8.

Q2: Human validation of the quality of automatically generated annotations.

To construct report grounding dataset for our in-house data, we implemented two automated processes: (1) Report Refinement: we employed InternLM to segment each report into detailed descriptions for each organ; (2) Structure Detection: We first finetuned an MLLM based on the Region-Text dataset to enable it to generate regions based on organ names. Then, we input the collected images into the finetuned MLLM, and prompted it with the organs mentioned in the reports to obtain the corresponding bounding boxes.

To evaluate the quality of the automatically annotated part of our dataset, we randomly selected 50 samples and asked 2 experts to create manual labels for comparison.

1. Human Validation of Report Refinement

   Report segmentation is a relatively straightforward task for a large language model, as it does not require generating new information but simply categorizing existing content based on the organs described. The sentence-level accuracy is 93.33%, where 2.67% of the sentences are missing and 4% of them are wrongly classified.

2. Human Validation of Structure Detection

   Compared with the human annotation, the accuracy of generated bounding boxes is 72%, and the IoU score is 55%. We also conducted a visual evaluation and found that although most bounding boxes are slightly larger, they can still encompass the target region. This is sufficient for our approach since we only require localization rather than a tight and accurate bounding box.

We have included the human validation in Section 3.2 to provide a quantitive assessment of the data quality.