ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models

We sincerely thank the reviewer for their thoughtful and supportive feedback. We also appreciate your positive evaluation of the core contributions of our work, including identifying the limitations of current medical MLLMs, introducing the multi-graph alignment objective, and proposing an efficient solver grounded in theoretical insights. Below we address your remaining concerns.

Q1. Does EXGRA-MED use full fine-tuning? How does it perform with parameter-efficient fine-tuning methods such as LoRA, and could this further improve efficiency?

A1 We thank the reviewer for this insightful question. To clarify, all experiments in the main paper use full fine-tuning to ensure fair comparisons with existing baselines, which are also fully fine-tuned.

To investigate the reviewer’s suggestion, we conducted additional experiments using LoRA-based parameter-efficient fine-tuning on both ExGra-Med and LLaVA-Med, and evaluated their performance on the VQA-RAD and SLAKE datasets. The results are summarized below:

These results show that ExGra-Med with LoRA still outperforms LLaVA-Med with LoRA, indicating the robustness of our approach even under parameter-efficient tuning. However, as expected, LoRA-based tuning does lead to a drop in performance compared to full fine-tuning.

We believe that to bridge this gap, pre-training on a larger-scale medical instruction-tuning dataset (e.g., the 21M samples curated from MedTrinity [1]) would allow the LoRA setup to more closely match the performance of fully fine-tuned models. If the paper is accepted, we plan to explore this direction further and release a variety of pretrained checkpoints, including LoRA-compatible versions, to support the community.

[1] Xie, Yunfei, et al. "MedTrinity-25M: A Large-Scale Multimodal Dataset with Multigranular Annotations for Medicine." ICLR 2024.

Q2. The proposed method appears to be domain-agnostic - could it also be effective in the general domain beyond the medical field?

A2. This is a very insightful question. We thank you for raising it. While our starting focus in this work is on medical multimodal LLMs, we highlight a broader challenge: autoregressive models tend to be data-hungry, especially when adapting across domain shifts (e.g., from natural images to medical images).

We believe this issue might not be limited to the medical domain. For instance, in the robotics domain, particularly in vision-language-action (VLA) models [1,2], similar challenges can arise. These models often face limited access to high-quality training trajectories and require significant adaptation when deployed in complex real-world environments. As a result, they may demand large-scale pretraining to bridge the domain gap.

Overall, we see this as an important direction for future research and believe our proposed multi-graph alignment approach could generalize well to such domains, potentially improving efficiency and cross-modal alignment where labeled data is scarce.

[1] Kim, Moo Jin, et al. "OpenVLA: An open-source vision-language-action model." Arxiv 2024

[2] Black, Kevin, et al. "π0: A vision-language-action flow model for general robot control. Arxiv 2024

Q3. Adding a more recent survey on MLLMs to provide readers with a comprehensive background and better contextualize your contribution within the broader MLLM research landscape.

A3. We thank the reviewer for this valuable suggestion. We agree that incorporating recent surveys on medical multimodal large language models (MLLMs) would strengthen the contextual grounding of our work. In the revised version, we will update the Introduction and Related Work sections to include relevant and up-to-date surveys, thereby better situating our contributions within the evolving MLLM research landscape.

We sincerely thank the reviewer for the thoughtful and encouraging feedback. We greatly appreciate your recognition of the key contributions of our work, including the design of the multi-graph alignment framework, the efficient use of limited instruction-following data, and the successful integration of IMLE for scalable graph alignment. Below we address Reviewers’ concerns.

Q1. Examine ExGra-Med’s generalization to other base models (e.g. Qwen).

A1. We thank the Reviewer for your suggestion. During the short period of rebuttal, we integrated the ExGrad-Med to the Qwen-VL 2.5, 7B version. This model is pre-trained with the original 10% medical instruction-tuning data and the extended version with multi-graph alignments. We summarize below our findings confirming that ExGra-Med still helps to enhance the Qwen model in this case. We’ll update this setting with 100% pre-training to the revised version if the paper is accepted.

Q2. Using proprietary GPT-4 APIs to generate enriched captions may hinder reproducibility or deployment in restricted environments.

A2 We thank the reviewer for highlighting these important concerns related to reproducibility and practical deployment.

To support reproducibility, we will publicly release the GPT-4, generated extended instruction-following dataset used in our experiments. This will allow researchers to reproduce our results without needing API access.

Regarding deployment in restricted environments, for example, in healthcare settings with strict data privacy constraints where sending data to external APIs (like GPT-4) may not be permissible, we address this by validating alternative caption generation using open-source models. Specifically, we employ the Qwen3-8B LLM [1], which has demonstrated superior performance compared to previous Qwen variants (e.g., QwQ, Qwen2.5) on reasoning-intensive tasks like mathematics, coding, and commonsense reasoning.

Due to the time constraints of the rebuttal phase, we were only able to evaluate ExGra-Med pre-trained on 10% Qwen-generated instruction-following data. We summarize the obtained results in the table below

As can be seen that, ExGra-Med (Qwen-enhanced captions) achieves:

+17.74% higher average accuracy on VQA-RAD (70.13 vs. 52.39),

+4.08% higher average accuracy on SLAKE (84.7 vs. 80.62), compared to the LLaVA-Med baseline.

While the performance of ExGra-Med with GPT-4-enhanced captions is notably higher on VQA-RAD (72.52 vs. 70.13), the two models achieve comparable results on SLAKE (85.01 vs. 84.7), suggesting that our multi-graph alignment remains effective even when using alternative instruction-following data sources. We will add these findings more explicitly in the revised manuscript if the paper is accepted.

[1] Yang, An, et al. "Qwen3 technical report." Arxiv 2025

Q3. The graph formulation is mathematically intensive and may benefit from simplified descriptions.

A3. We thank the reviewer for this valuable suggestion. We agree that the current mathematical formulation may be too technical and could benefit from a more accessible presentation to better reach a broader audience. If the paper is accepted, we will revise the text to simplify the mathematical descriptions and provide more intuitive explanations, including visual illustrations and video teasers on the project website, to make the core ideas of the method easier to understand and follow.

Q4. How can ExGra-Med remain robust when replacing GPT-4 with potentially lower-quality caption generators?

A4. We appreciate the reviewer’s question and have addressed this concern in conjunction with Q2. Specifically, we evaluated the robustness of ExGra-Med by replacing GPT-4 with a state-of-the-art open-source model, Qwen3-8B, to generate extended instruction-following captions.

As reported, even when using captions generated by Qwen3-8B, arguably of lower quality than GPT-4, ExGra-Med still achieves a significant improvement over the LLaVA-Med baseline, confirming the robustness of our multi-graph alignment approach to variation in caption quality.

Q5. Is the triplet graph structure essential, or would simple pairwise alignments between image and extended captions suffice?

A5. We thank the reviewer for this insightful question. We had the same curiosity and conducted an ablation study to investigate the necessity of the triplet graph structure.

In this study, we compared the full triplet alignment setup (image, original caption, and extended caption) against a simplified version using only pairwise alignments (image and extended caption). Both models were pre-trained using 40% of the medical instruction-tuning data, and results are shown in Table 6 of the main paper. For convenience, we summarize the key results below:

These results indicate that the triplet alignment structure provides a meaningful improvement, likely due to its ability to better capture the semantic relationships across all three modalities. This richer structure helps the model learn more robust and informative representations compared to simpler pairwise alignments.

Q6. Can the authors release a toolkit or provide open-source code for the black-box optimization with IMLE to support reproducibility?

A6. Yes, we fully support open science and reproducibility. Upon acceptance, we will release:

• The pretrained checkpoints (based on multi-graph alignment) for downstream task adaptation, and

• The complete training code, including our efficient solver with barycenter graph construction and end-to-end learning using black-box gradient estimation via IMLE.

This will enable the community to reproduce our results from scratch, starting from the official LLaVA checkpoints, and facilitate further research based on our framework.

Q7. Quantitative results when training EXGRA-MED with smaller frozen LLMs to evaluate downstream performance trade-offs

A7. We thank the reviewer for this thoughtful suggestion. We fully agree that evaluating ExGra-Med with smaller language models would provide valuable insight into the trade-off between model size, efficiency, and performance.

Due to the limited timeframe of the rebuttal period, we were unable to conduct additional experiments with significantly smaller models (e.g., 3B or below). However, we partially addressed this concern by validating the robustness of ExGra-Med on the Qwen-VL 7B model, as discussed in Answer 1. This evaluation demonstrates that ExGra-Med generalizes well beyond LLaVA-Med and retains its performance gains when applied to a different backbone.

We recognize the importance of further exploring the performance of ExGra-Med on smaller frozen LLMs to better support deployment in resource-constrained settings. If the paper is accepted, we plan to extend our analysis to smaller models, such as Qwen-VL 3B or MiniGPT variants, and release the corresponding checkpoints and results to benefit the community.

We thank the reviewer for the encouraging feedback and thoughtful observations regarding both the strengths and areas for improvement in our work. Regarding the training efficiency concern:

Q1. Clarifying additional costs of 0.5 hour longer of ExGraMed compared to LLavA-Med.

A1. We thank the reviewer for raising this important point. The additional 0.5-hour training time required by ExGra-Med compared to LLaVA-Med is primarily due to the multi-graph alignment module. This includes the following computational steps:

• Constructing graphs using the K-nearest neighbors algorithm,

• Performing GNN-based message passing to aggregate features among nodes in each graph,

• Solving three graph alignment problems during the forward pass using a barycenter graph, and

• Executing the corresponding backward steps to compute gradients for optimization.

To ensure that the observed performance gain of ExGra-Med is not simply due to longer training time, we conducted an additional ablation experiment where LLaVA-Med was trained for one extra hour. As shown in Table 3, ExGra-Med still outperforms LLaVA-Med under these settings, confirming the effectiveness of our proposed alignment method beyond just training duration.

Q2. Clarifying the language models LLAMA versions used in ExGra-Med and LLaVa-Med

A2. Thank you for your careful evaluation and valuable question. To ensure a fair comparison, we confirm that both ExGra-Med and LLaVA-Med in our study are built upon the same original LLaMA-7B model, as provided on Hugging Face [1]. Specifically, we do not use any variant such as the Mixture-of-Experts mistral-7B-v0.2 checkpoint. This ensures consistency and fairness across all experiments.

[1] huggyllama/llama-7b

Q3. Make consistent use of the terms “pre-training” and “fine-tuning”

A3. Thank you for the helpful comment. We acknowledge the potential confusion caused by the use of the terms “pre-training” and “fine-tuning.”

Following the original LLaVA paper (Figure 3), Stage 1 is intended for the medical concept alignment, and Stage 2 is used for fine-tuning on generic medical instruction tuning. However, both these Stages indeed can be seen as the pre-training phase because the model is still not fully optimized for specific downstream tasks. Additional fine-tuning is required on each target dataset (e.g., VQA-RAD, SLAKE, or PathVQA) to achieve the best performance.

Regarding line 281, we intended to state that ExGra-med takes slightly longer to complete the full pre-training process (Stage 1 + Stage 2) on the 600k medical instruction tuning samples compared to LLaVA-Med.

We will revise this sentence for clarity and ensure consistent use of terminology in line with the original LLaVA definitions throughout the manuscript.

We sincerely thank the reviewer for the thoughtful and detailed feedback. We greatly appreciate your recognition of the strengths of our work, including the effectiveness of the multi-graph alignment mechanism, the strong data efficiency, the significant gains on VQA-RAD, and the scalability enabled by our black-box optimization approach. We also appreciate the constructive suggestions regarding architectural clarity, evaluation scope, and broader applicability. Below, we address each of these concerns in detail.

Q1. Architectural complexity of multi-graph alignment, reproducibility, and practical adoption

A1. We thank the reviewer for highlighting these important aspects. Regarding the architectural complexity of the multi-graph alignment module, if the paper is accepted, we will simplify the mathematical formulations and include more intuitive explanations - such as visual illustrations and video teasers on the project website to make the method easier to understand and follow.

In terms of reproducibility and practical adoption, we are committed to open-sourcing:

• The pretrained checkpoints are used for downstream adaptation tasks, and
• The complete training pipeline for multi-graph alignment

This will ensure that our method can be fully reproduced and easily integrated by the research community.

Q2. Limitation of ExGra-Med’s conversational format in handling multi-turn dialogues or clinical decision-making

A2. We thank the reviewer for this question. We acknowledge that Figure 2 provides only a simplified illustration of how we extend the original captions. However, in practice, ExGra-Med is capable of handling multi-turn dialogues, as it is trained on curated multi-turn conversational data provided in the pre-training set of LLaVA-Med (Figure 5 in the LLaVA-Med paper).

Specifically, for each input image, medical experts generate sequential follow-up questions based on the model’s prior responses, enabling the system to learn context-aware dialogue behavior. This design allows ExGra-Med to better support extended interactions that resemble clinical reasoning and decision-making processes.

Q3. Evaluations are limited to simple benchmarks (VQA-RAD, SLAKE, PathVQA) and primarily compared against LLaVA-Med. More challenging benchmarks such as OmmiMedVQA and recent SOTA medical multimodal LLMs are needed.

A3. We thank the reviewer for these valuable suggestions and fully agree that including more challenging benchmarks and broader comparisons strengthens the paper’s contributions. Fortunately, we have already included these evaluations but may not have highlighted them clearly enough in the main text:

(i) The zero-shot results on 23 segmentation tasks are actually sub-tasks derived from the OmmiMedVQA benchmark (Appendix Section G, Figure 4). Additionally, we conducted a medical visual chatbot evaluation (zero-shot) using GPT-4 as the evaluator due to the complexity of the model responses (Appendix Section F).

(ii) We have also compared ExGra-Med with state-of-the-art medical MLLMs beyond LLaVA-Med, such as RAD-FM (pre-trained on 16M 2D/3D scans vs. 600k for LLaVA-Med), MedDr (40B parameters vs. our 7B model, trained on 2M instruction-tuning samples), and GPT-4o API for zero-shot evaluation on the medical visual chatbot task.

Beyond medical MLLMs, we have benchmarked the multi-graph alignment module itself against advanced alignment algorithms across different domains such as PLOT, SigLIP, VLAP (2-domain alignments), and GeoCLAP, PAC-S, ImageBIND (3-modality alignments).

In summary, we believe these diverse benchmarks (from standard to highly challenging) and comparisons with multiple SOTA baselines effectively validate ExGra-Med’s strengths. If the paper is accepted, we will revise the manuscript to emphasize these points more clearly to address the reviewer’s concerns.

Q4. Comparison with baselines trained using GPT-4-generated instruction-following data, not just used in multi-graph alignment

A4. We appreciate the reviewer’s suggestion and agree that clarifying this point is important. We would like to clarify that we have already conducted analyses to assess the effect of using GPT-4-generated extended instruction-following data beyond just multi-graph alignment. Specifically:

(i) We trained LLaVA-Med models using the GPT-4-extended instruction data, and the results are presented in Table 3, denoted as LLaVA-Med (10%) ext. cap and LLaVA-Med (40%) ext. cap, representing training with 10% and 40% of the extended captions, respectively.

(ii) For all multi-modal pretraining algorithms in Tables 1 and 2, including ExGra-Med and relevant baselines, the same GPT-4-extended instruction-following data were used to ensure a fair comparison. This setup helps isolate the effect of the multi-graph alignment itself, and results confirm the consistent improvements of ExGra-Med across all three VQA datasets.

We will revise the manuscript to better highlight these evaluations and ensure this point is clearly communicated to readers.

Q5. Unclear whether the performance gains of ExGra-Med are robust across other foundation models beyond LLaVA-Med

A5: We thank the reviewer for raising this important point. To evaluate the robustness of ExGra-Med beyond the LLaVA-Med backbone, we have integrated the multi-graph alignment framework to the Qwen-VL 7B model, a strong vision-language foundation model.

We compared the following three configurations:

• Qwen-VL 7B baseline without additional pretraining on medical instruction-following data,

• Qwen-VL 7B pre-trained with 10% GPT-4-enhanced medical instruction-following data, and

• ExGra-Med uses the Qwen-VL 7B architecture, also pre-trained with the same 10% instruction-following data, but incorporating our multi-graph alignment strategy.

All models were fine-tuned on the VQA-RAD and SLAKE datasets (closed-ended questions, evaluated with accuracy). Our results show that ExGra-Med consistently improves performance over both baselines on these benchmarks, with gains of 3–6%, demonstrating that our approach is model-agnostic and generalizes well across different foundation architectures.

We will add these findings more explicitly in the revised manuscript to highlight the generalizability of our method.