Thank you for your positive and insightful review, which greatly helps to refine the manuscript's quality and clarity. Below are our point-by-point responses.

1. Additional Details

We appreciate the reviewer’s insightful questions and welcome the opportunity to further elaborate on the relevant concepts. Below, we provide detailed clarifications.

1.1 Fig.2 setting

Regarding the experimental setup in Figure 2, its purpose is to illustrate conflicts that arise in traditional joint modeling methods when comprehension and generation tasks are trained on heterogeneous data. The experiment consists of two stages:

1. In the first stage, lightweight adapters align visual and textual features to build the visual-language pretraining base.
2. In the second stage, while preserving single-task data integrity, we gradually introduce heterogeneous data from the other task (25%/50%/100% generation or comprehension data).

We observe that increasing heterogeneous task data significantly degraded performance due to task interference. In contrast, our method introduces only 2% additional repeated data (used in the second phase of TLS), greatly improving the performance of both tasks.

1.2 Diminishing returns clarified

We would like to clarify a possible misunderstanding of the “diminishing returns” description. Our intent was not to undervalue methods like MetaMorph, but to highlight the difficulty of transferring joint training paradigms across tasks with distinct structures and imbalanced data scales.

In the medical domain, comprehension and generation differ significantly in objectives and data availability, making mutual gains through joint training challenging and sometimes prone to negative transfer (see Fig. 2). We will revise the phrasing in the next version to avoid broad descriptions:

Joint training of comprehension and generation may face challenges such as task interference, data bias, or optimization saturation.

1.3 Fusion embedding layer

Additionally, we appreciate the reviewer’s attention to the fusion embedding layer. Our design introduces multimodal tokens into the original vocabulary to integrate discrete indices from the visual encoder (VQGAN). This includes 8,192 discrete tokens (from the VQ codebook) and two special tokens, <START_IMG> and <END_IMG>, forming the fusion embedding layer to seamlessly integrate visual information into the language model input. We will clarify and define this term in the method section.

2. Comparative Experiments

We understand the reviewer’s request to compare our method with more powerful LVLMs for better evaluation. To address this, we add experiments with the enhanced HealthGPT-XL version and additional comparisons:

We carefully evaluate the concurrent work MedMax, which performs well, especially in OmniMedVQA. We have updated the experimental comparisons content in the manuscript.

It is important to emphasize that HealthGPT aims to provide a unified, scalable framework adaptable to diverse tasks. It can seamlessly incorporate high-quality datasets like MedMax to boost downstream performance, and its design is highly complementary to data-driven approaches such as HuatuoGPT-Vision and MedMax.

3. Model Details

3.1 VQGAN

We appreciate the reviewer’s focus on the model's resource consumption and component compatibility, which are essential for the method's reproducibility and practicality. Below are our detailed responses:

The VQGAN model, pretrained on OpenImages, is validate on three medical modalities: X-ray, CT, and MRI. The results demonstrate its strong performance in preserving key structural and lesion region information, with good fidelity and consistency:

3.2 Training details

Our method demonstrates strong resource efficiency and reproducibility. In the default configuration, the M3 model completes a full training cycle in about 35 hours using 8 A100 GPUs, which we consider reasonable given its performance across multiple tasks and the modular design.

3.3 Code Release Plan

We recognize the importance of open sourcing for the community and have nearly completed organizing the code, data processing scripts, model weights (M3, L14, and XL versions), and inference workflows. We will release the full codebase publicly to ensure reproducibility and ease of extension.

Thank you again for your recognition and suggestions. We believe the response better highlights HealthGPT’s technical value and practical potential, and we hope it contributes to the development of medical multimodal models.

Thank you for your insightful review. Your insights have helped us clarify HealthGPT's contributions in unified architecture design, task modeling efficiency, and real-world application potential. Below are our point-by-point responses.

1. Contribution Statement

1.1 Core Contribution

We appreciate the reviewer’s detailed focus on our method and contributions. It is worth noting that Reviewers wUUr, ugJa, and B8gX have recognized the novelty of our work, describing it as innovative and meaningful. Building on this recognition, we would like to respectfully clarify a potential misunderstanding regarding the core contribution of our paper.

• Paradigm Innovation. Our primary innovation lies in proposing the first unified Med-LVLM paradigm, which goes beyond alignment or fine-tuning strategies. Specifically:
  • At the global-task level, we enable the model to learn and handle heterogeneous characteristics and formats across both comprehension and generation.
  • At the local-task level, we design mechanisms to efficiently learn and transfer shared knowledge across diverse sub-tasks.
• Method Innovation. We carefully designe H-LoRA to align with our proposed paradigm, and further optimize TLS and HVP to support its integration. Thus, we emphasize that alignment and fine-tuning are techniques we adopt to support this unified paradigm, not the main novelty itself.

To further clarify this point, we briefly summarize the key contributions:

1. Lines 139–142 (left): We propose the first unified Med-LVLM to solve the challenge of integrating diverse medical tasks under a single LVLM framework.
2. Lines 74–100(right): We introduce H-LoRA to solve the significant conflict between comprehension and generation tasks, greatly improving both the performance and efficiency over MoE+LoRA-based methods.
3. Lines 104(right)–126(left): We propose HVP and TLS to solve the challenge of heterogeneous knowledge fusion and hierarchical visual feature requirements.
4. Lines 127–136(left): we introduce VL-Health to solve the lack of high-quality datasets that jointly support unified tasks in the medical domain.

Thus, this work goes beyond benchmarking by establishing a unified Med-LVLM framework that expands the capabilities of current medical models, which is crucial for advancing practical medical solutions.

1.2 H-LoRA batter than MoE+LoRA

Our results show that while MoE+LoRA improves performance, it violates the PEFT principle of efficiency. In contrast, H-LoRA significantly outperforms MoE+LoRA in both performance and training efficiency, as supported by theoretical analysis (Reviewer wUUr (Section 2: H-LoRA)) and experiments (Fig.5, Tab.5, Tab.10). H-LoRA achieves higher average scores (+3.6/+0.38 on comprehension/generation) with only 67% of the training time. We kindly hope these results clarify the necessity and advantages of H-LoRA.

1.3 Med-LVLM Significance

We appreciate the reviewer’s thoughtful concern regarding the application of Med-LLMs. As highlighted in prior studies [1,2], developing Med-LLMs is both a timely scientific challenge and a transformative opportunity for healthcare. These models can bridge multimodal medical data, support informed decision-making, and ultimately improve clinical outcomes.

2. Efficiency Analysis

We appreciate the reviewer’s focus on efficiency, vital in resource-limited medical scenarios. To validate efficiency, we compared H-LoRA with MoELoRA under identical settings. H-LoRA outperformed MoELoRA across tasks and reduced training time to 67% of MoELoRA’s:

Besides, our three-stage learning strategy eliminates unnecessary padding caused by task sequence length mismatches, reducing training FLOPs by 22% (from 6.4e19 to 5.0e19) and lowering memory usage per GPU from 36GB to 30GB, saving 17%.

Finally, HealthGPT-M3 completed training on 8 A100 GPUs in 35 hours, significantly fewer GPU hours than most general LVLMs, demonstrating its potential for rapid deployment in medical scenarios.

3. Content Refinement

We thank the reviewer for their helpful suggestions. We will simplify Figure 1 by streamlining labels, standardizing colors, and improving the layout to focus on structural logic.

Regarding the formulas, we appreciate the reviewer pointing out redundancy. We have identified and will simplify or merge unnecessary functions and symbols (e.g., in Equations 4 and 5), ensuring the focus remains on our key innovations.

We sincerely thank you for your valuable suggestions. We believe this work addresses critical gaps in medical multimodal LVLMs and supports their deployment in real medical scenarios.

Reference

[1] A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics
[2] A Survey of Large Language Models in Medicine: Progress, Application, and Challenge

We sincerely thank the reviewer for the detailed and thoughtful feedback, which greatly helped us refine our experiments and methodology, and better highlight HealthGPT’s innovation and practical value. Below are our point-by-point responses.

1. VQA Analysis

1.1 Comparison with LLaVA-Med++

We wish to clarify that LLaVA-Med++ is not a chat LVLM and is fine-tuned on individual downstream tasks, which compromises its generalization ability. For fairness, we use LLaVA-Med++'s training setting and the results are below:

1.2 Comparison experiments

Notably, we further release the HealthGPT-XL with a stronger LLM (Qwen2.5-32b). We believe this model is the new SOTA Med-LVLM compared with baselines:

The aforementioned results further demonstrate HealthGPT's effectiveness.

1.3 Failure case

After analysis, the failure cases stem from two sources:

1. Broad questions (e.g., 'What is present?')
2. Same question, different answers (Q → A1 vs. Q → A2)

Addressing (1) and (2) typically requires fine-tuning on the corresponding training set. However, our model still achieves the best performance through strong generalization and instruction-following capabilities.

1.4 VQA qualitative

We visualize heatmaps for different questions and masked key regions to assess answer quality: https://anonymous.4open.science/r/HealthGPT-9533/visualization.png. Results confirm our model relies on critical regions for reasoning.

2. H-LoRA

2.1 Performance of H-LoRA

We believe the reviewer may have misunderstood—we did compare performance with MoELoRA. Fig.5, Tab.4, and Tab.10 in our paper systematically demonstrate the performance and efficiency advantages of H-LoRA.

2.2 H-LoRA better than LoRA

• Efficiency. Existing research confirms that LoRA with MoE and router mechanisms improves performance for medical tasks [1]. To enhance efficiency, we propose internalizing LoRA experts as matrix space separation: $\sum ka w_i A_i B_i / r \rightarrow ka A \odot \mathcal{W} B / r$.

• Effectiveness. We find the conventional router mechanism reduces the expected scaling factor from $a/r$ to $a/(rk)$, leading to performance degradation [2]. We correct the scaling factor and experimentally validate its effectiveness in our paper. Additionally, we included an ablation study on the router module:

We hope the above analyses and experiments further demonstrate the potential of H-LoRA in the medical domain.

3. Image Metrics

3.1 LPIPS & FID

We appreciate the reviewer’s concern about perceptual consistency in images. As requested, we’ve added LPIPS and FID metrics (https://anonymous.4open.science/r/HealthGPT-9533/lpips_fid.png), showing HealthGPT’s strength in both pixel-level and perceptual quality.

3.2 Human evaluation

We also conduct human evaluation of the modality conversion task across five dimensions, comparing against the best-performing BBDM method. H-LoRA achieves higher average scores (4.30/4.52 vs. BBDM's 3.54/4.16):https://anonymous.4open.science/r/HealthGPT-9533/human_eval.png.

4. Other Issues

4.1 Mutual Gains between Comprehension and Generation

We would like to respectfully clarify that our work aims to mitigate the significant conflicts between comprehension and generation tasks, rather than assuming inherent complementarity. Given current modeling paradigms and data limitations in the medical domain, achieving mutual benefit is particularly challenging (see Fig. 2). To this end, we design dedicated mechanisms to reduce such conflicts and enable unified modeling.

4.2 Clinical Potential

We highly value clinical evaluation and are already collaborating with two public hospitals, which provide ocular disease reports and case data for validation. These efforts represent a concrete step toward assessing the clinical applicability of our approach, and preliminary evaluations on the provided clinical data show promising advantages of our model in ocular disease understanding.

4.3 Dataset Bias

Dataset bias is an important concern. Due to space limits, please see our response to Reviewer ugJa (Section 4: Dataset Bias).

We hope our response clarifies the model’s rationale and its potential in medical applications.

Reference

[1] When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications
[2] A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA

We greatly appreciate the reviewer’s recognition of our work. Your recognition of the model’s originality, innovation, and effectiveness greatly encourages us and has helped improve HealthGPT. Below are our point-by-point responses.

1. Unified LVLM Comparison

Thank you for the suggestion. Below, we provide further comparisons with SoTA unified LVLMs:

Meanwhile, we present an improved HealthGPT-XL with enhanced performance. We believe this additional experiment demonstrates significant advantages of our method.

2. Generalization Capabilities

2.1 Generalization ability

To demonstrate HealthGPT’s generalization and practical value, we highlight two aspects:

1. Benchmark generalization: The model performs well on unseen tasks and modalities (e.g., OmniMedVQA, MMMU-Med), showing robustness beyond the training distribution.
2. Clinical collaboration: Ongoing partnerships with public hospitals on ocular disease diagnosis have yielded promising early results, with further evaluation in progress.

These results underscore HealthGPT’s potential for both benchmark-level generalization and practical clinical impact.

2.2 Complex medical tasks

Thank you for your thoughtful concerns. We recognize their importance and provide the following clarifications:

1. Unseen medical conditions: HealthGPT demonstrates strong generalization and medical knowledge coverage, enabling it to handle most previously unseen conditions.
2. Unseen modalities: In cases lacking prior modality-specific data, relying solely on a pre-trained LLM may introduce bias. However, our plugin-based learning approach allows for rapid adaptation using high-quality modality-specific data.
3. Simultaneous multimodal inputs: This engineering challenge can be effectively addressed through targeted data collection and training. We are actively collaborating with hospitals to tackle this issue in real-world clinical settings.

We will incorporate the clarifications in the next version.

3. Differences with MoE Mechanism

We appreciate the reviewer’s detailed attention to the model's structure, allowing us to clarify the differences between H-LoRA and MoE architectures and highlight the necessity and advantages of our design.

Existing work explore LoRA and MoE combinations, including symmetric[1] and asymmetric structures[2]. While performance benefits are evident, we observe that introducing LoRA experts significantly increases resource consumption, compromising the efficiency of PEFT.

To validate this, we compare the training efficiency of several LoRA+MoE structures with the same LoRA setting (r=64, k=4):

DS-LoRA is a LoRA mechanism based on the powerful MoE model Deepseek-V3. We find that MoELoRA, HydraLoRA, and DS-LoRA significantly reduce training speed, while H-LoRA avoids this limitation.

Unlike traditional MoE scheduling, H-LoRA uses low-rank matrix subspace dynamic routing, preventing performance degradation and instability from issues like routing jitter and load imbalance. It also adaptively activates parameter subspaces, creating soft isolation and weak coupling to enhance adaptability to task heterogeneity.

Additionally, we emphasize that our approach is not a dense-to-MoE alternative for expanding FFN. While traditional MoE increases capacity by expanding parameters, our method focuses on balancing performance and computational resources in efficient fine-tuning.

4. Other Issues

4.1 Potential Limitations

We appreciate the reviewer’s attention to potential limitations. Our work explores the feasibility of efficiently handling unified comprehension and generation tasks in medical scenarios. As shown in Figure 2, joint training faces a significant bottleneck in data-scarce medical tasks. However, task cooperation enhancement is a long-term focus, and we aim to explore the complementary potential between the two tasks for synergistic effects.

4.2 Dataset Bias

We understand that bias in sample distribution and task labeling can impact model generalization. To improve chat-LVLM practicality, we balanced general instruction data with curated medical datasets during training, enhancing generalization and medical knowledge. A section on dataset bias has been added to the appendix.

Thank you again for your positive and encouraging feedback and insightful comments. We hope this response further clarifies our design, contributions, and the model's potential.

Reference

[1] When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications

[2] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning