From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning

1. Can you compare your method with more baseline models? Why you chose to fine-tune particular VLMs?

We expanded our comparisons to include additional baseline models such as LLaVA-7B v1.5, InternVL2-8B, and Qwen2-VL-7B-Instruct. As shown in Table 2, fine-tuned Qwen2-VL-7B and other models with standard tuning generally achieve comparable or worse performance than fine-tuned InternVL2-2B in most cases. Considering InternVL2-2B's smaller size and robust performance, we selected it as our base model for experimentation.

2. Verification of the proposed method on medical VLMs.

We applied our method to LLaVa-Med v1.5 to assess its effectiveness on medical VLMs. As shown in Table 2, our method improves LLaVa-Med's accuracy and F1 score by 10.3% and 11.6%, respectively, demonstrating its capability to enhance both general and medical VLMs.

Additionally, we verified that our method preserves the base VLM's original capabilities while enabling it to handle new tasks. As shown in Table 1, the InternVL2-2B achieves comparable performance on its original benchmarks before and after fine-tuning, confirming that our method generalizes beyond task-specific specialization.

3. Concerns about fine-tuning time and resources used.

In our experiments, training a TSM for classification with ~500K samples takes 2.5 hours on 4 Nvidia A100 GPUs. Standard visual instruction tuning of InternVL2-2B takes 2 hours, while our VITask method takes 4 hours to fine-tune the same base VLM.

Given the substantial task-specific performance improvements, we consider the computational cost of training a TSM and applying VITask acceptable. Moreover, training the TSM from scratch on the same dataset is done purely for fair comparison in this study but is not always necessary in practice. Our method is flexible and can leverage pre-existing models as TSMs, which will significantly reduce computational cost.

To demonstrate this flexibility, we utilized SAM-Med2D [12] as the TSM for medical phrase grounding. Results in Table 3 show that our method improves Recall@0.5 by 4.63% on average, confirming its flexibility and effectiveness across tasks, including classification and phrase grounding.

4. Validation on non-medical domains.

Thank you for the suggestion. We conducted additional experiments on three natural image classification datasets under a challenging open-world setting, where no candidate answers are provided in the instruction. Results in Table 4 demonstrate that our method significantly enhances the fine-tuned VLM's performance, achieving results comparable to the TSM, which represents the upper bound for classification accuracy. These findings validate that our method is effective and broadly applicable across both general and medical domains.

Thank you for your rebuttal. Most of my concerns are addressed and the overall experiment results on medical domain are impressive.

However, I concern about the results on natural images. The performance when using TSM and your method are quite equal, which do not show significant improvement of your method compared to TSM. I think because TSM have so good performance on these dataset, which make the result do not show any significant improvement.

In addition, your method now only focus on classification task and I think there are many tasks in medical domain which are more important than classification such as VQA, Medical Report, so I keep my original score.

1. Evaluation on tasks beyond classification.

We appreciate the suggestion. To assess whether our method generalizes to tasks other than classification, we tested it on medical phrase grounding using SAM-Med2D [12] as the TSM on the Med-GRIT-270 dataset [11]. This dataset includes 270k question-answer pairs across eight medical imaging modalities and evaluates bounding box predictions for given textual findings.

As shown in Table 3, our VITask method improves the Recall@0.5 by 4.63% on average compared to standard instruction tuning. These results demonstrate the effectiveness of VITask in enhancing the grounding ability of fine-tuned VLMs, suggesting its potential for a wide range of vision-language tasks.

2. Why use VLMs with a high computational budget instead of improving TSMs?

We acknowledge that directly improving a TSM is an effective approach for optimizing specific downstream tasks. However, fine-tuning a VLM aligns better with the broader goal of creating an AI agent capable of handling multiple tasks, as VLMs offer superior generalizability and user-friendliness.

Our work emphasizes teaching a pre-trained VLM to handle new tasks while preserving its original capabilities, rather than optimizing for state-of-the-art performance on individual tasks. Using our method, the fine-tuned VLM acquires new capabilities without compromising existing ones.

To validate this, we used InternVL2-2B as the base model and evaluated its performance before and after fine-tuning with our method on benchmark datasets originally designed for evaluating InternVL2. As shown in Table 1, our method introduces only minor degradation, confirming that the VLM's original capabilities are preserved.

3. Improving interpretation of results for better readability.

We appreciate your feedback. The comparisons between our methods and baseline models have been revised for clarity and better readability in the updated manuscript.

4. Missing figures, equation formatting (3–6), and typos.
Thank you for pointing these out. We have corrected the formatting issues, added the missing figures, and fixed typos throughout the paper.

5. Empirical analysis of VITask vs. TSMs in cost and performance.
Training the TSM for classification on 500K samples takes 2.5 hours using 4 Nvidia A100 GPUs, while the standard visual instruction tuning of InternVL2-2B takes 2 hours. Our VITask method requires 4 hours to fine-tune the same base VLM, which improving the F1 score of vanilla-tuned InternVL-2B from 60.4% to 77.1% and outperforming the TSM by 3.1% in average accuracy.

We consider this computational cost acceptable given the significant performance improvement. Moreover, VITask is flexible and can utilize pre-trained TSMs, avoiding the need to train one from scratch. For instance, in medical phrase grounding, we use SAM-Med2D [12] as the TSM. As shown in Table 3, VITask is effective even for tasks beyond classification.

The optional RDA strategy adds 2 hours of training but removes the reliance on TSMs during inference. RDA resolves scalability challenges when dealing with multiple tasks and numerous TSMs, which would otherwise require substantial storage and computation. With RDA, the fine-tuned VLM operates independently of TSMs while maintaining strong performance, offering a trade-off between test-time efficiency and effectiveness.

6. Why do results decline significantly when embeddings are completely replaced by task-specific representations? Did you use the CLS token or entire patch embeddings from ViT?

Task-specific features are not well-aligned with the LLM, which is pre-trained to process original vision embeddings. Replacing these embeddings entirely disrupts the generative capabilities of the VLM, leading to a significant drop in performance.

In our method, we use the CLS token from the TSM as the feature for exemplar prompting. While we also tested using the entire patch embeddings, the results showed no substantial improvement. Therefore, we chose the CLS token for better efficiency.

7. Why use Stage 1 loss functions without scaling in Stage 2 optimization?

Thank you for pointing this out. In Stage 2 training, we downscaled the CRT weight by setting the hyperparameter $\beta = 0.1$, ensuring all loss terms are in a comparable range. This adjustment prevents any loss term from dominating, maintaining balance in the fine-tuning process.

8. What is the "novel" vision-language connector in Line 423?

We apologize for the confusion. The word "novel" was a typo and should be removed. The TSM connector is a simple MLP, similar to the one used in the original VLM.

Dear Reviewer YD9v,

Thank you for revisiting our work and for your positive assessment. We deeply appreciate your acknowledgment of the improvements made and the constructive feedback provided. Below, we address your follow-on questions:

1. Inclusion of results in the paper.

We completely agree that the results shared in the rebuttal are essential to demonstrating the significance of our method. We will incorporate these results and their accompanying discussions into the final version of the paper. We believe this addition will significantly enhance the clarity and impact of our work.

2. Reasons behind the steep drop in performance in Table 1.

For the LLM component of the VLM, we always use LoRA for fine-tuning. For the vision encoder and connector, we directly fine-tune their weights without employing any PEFT techniques. This direct weight adjustment explains the observed steep drop in performance following fine-tuning.

We sincerely thank you for your thoughtful suggestions and insights. Please do not hesitate to reach out if you have further questions or additional feedback.

Best regards,
The Authors

Thank you for your detailed comments and valuable suggestions, which have greatly improved the quality of our paper. Below, we address your concerns and clarify potential misunderstandings:

1. Justification of what makes EP successful.

1.1 Does the performance gain of EP come from fine-tuning additional learnable model parameters?

The performance improvement of EP stems from the specialized features from the external TSM, rather than the fine-tuning of additional learnable modules such as the connector between the TSM and LLM.

To validate this, we conducted additional experiments comparing several baselines:

• Standard LoRA tuning (Baseline)
• Fine-tuning the original VLM connector and LoRA parameters of LLM (LLM+Conn)
• LLM+Conn with prefix tuning (LLM+Conn+Prefix)

Table 5 shows that EP >> LLM+Conn+Prefix > LLM+Conn > standard tuning. Specifically, fine-tuning the VLM connector improves performance to some extent, as expected. Adding prefix tuning further enhances LLM+Conn, albeit marginally. However, both LLM+Conn and LLM+Conn+Prefix consistently underperform compared to EP, often by a significant margin. These results demonstrate that EP's performance gain is not attributed to fine-tuning additional learnable parameters.

1.2 Why does fine-tuning the vision model in VLM distort vision-language alignment? If so, wouldn’t trained LoRA weights also be harmful?

Fine-tuning the vision model in a VLM distorts vision-language alignment in the sense that it alters the entire latent space of image embeddings, causing the VLM to forget pre-trained knowledge and tasks. However, this does not imply that fine-tuning the vision model is harmful for learning new downstream tasks. In comparison, instruction-tuned LLMs are more robust to forgetting because their tuning is conditioned on new instructions, which has less impact on their pre-trained instruction-following abilities.

To validate this, we compared EP with the following baselines:

• Fine-tuning the vision encoder and LoRA parameters of LLM (LLM+Vision)
• Fine-tuning both the vision encoder and connector with LoRA parameters (LLM+Vision+Conn)
• EP with a learnable vision encoder (EP+Vision)

As shown in Table 5, making the vision encoder learnable significantly improves downstream task performance. However, further tuning the connector does not yield additional benefits, and EP still outperforms or performs on par with VLM+Vision and VLM+Vision+Conn. Notably, EP with a learnable vision encoder does not lead to better performance. These findings suggests that learning from the TSM features is more effective than tuning the original vision encoder.

Furthermore, as will be discussed later, fine-tuning the vision encoder causes the VLM to completely forget pre-trained knowledge, which is impractical for generalizability. These results further highlight the advantages of EP.

Table 5. Classification performance (Acc/F1) of InternVL2-2B with different fine-tuning approaches.

1.3 Comparison with Graph Neural Prompting (GNP)

While EP and GNP share the general concept of using encoded multimodal features to prompt LLMs, this is a standard practice for multimodal LLMs and does not imply that EP is similar to GNP. The key differences are as follows:

• Research Problem: GNP focuses on enabling LLMs to understand knowledge graphs, while our work aims to teach pre-trained VLMs to handle new tasks while preserving its original capabilities.
• Technique: GNP follows the standard instruction-tuning paradigm to train a multimodal LLM for knowledge graph based QA. In contrast, EP introduces a new approach that improves the standard instruction-tuning paradigm to enhance task-specific performance.

2. Explanations of Response Distribution Alignment (RDA)

2.1 Differences between RDA and knowledge distillation

RDA is fundamentally different from knowledge distillation, which involves replicating the outputs of a teacher model using a student model. In RDA, the TSM features act only as additional prompts and do not serve as the "teacher" outputs to be replicated. Instead, RDA ensures the VLM produces consistent outputs from two slightly different inputs—one with EP and one without EP.

RDA minimizes the discrepancy between the output distributions of the VLM from these two forward passes, enabling the fine-tuned VLM to function effectively even without EP. Another key difference of RDA from knowledge distillation is that the involved output distributions are dynamic and iteratively updated together during training, rather than mimicking static outputs as in knowledge distillation.

Therefore, despite its similarity to knowledge distillation in aligning embeddings, we specifically name it Response Distribution Alignment (RDA) to highlight its unique role in ensuring output consistency between forward passes w/ and w/o EP.

2.2 Why gradient detachment is needed for RDA?

In RDA, gradient detachment is applied to the VLM outputs from the input with EP, not to the TSM, which is frozen all the time. This ensures that the VLM without EP learns to mimic the behavior of the VLM with EP, eliminating the reliance on EP for better test-time efficiency.

3. Comparison between Contrastive Response Tuning (CRT) and Visual Contrastive Decoding (VCD)

CRT is a fine-tuning method aimed at enhancing visual instruction tuning, while VCD is a sampling technique designed to mitigate hallucinations during inference. Despite their shared focus on contrastive learning, CRT and VCD address entirely different challenges and are independent approaches.

To evaluate how VCD affects fine-tuned VLMs, we applied it to the fine-tuned LLaVA-Med v1.5 with standard instruction tuning, as VCD's official implementation targets LLaVA. For fair comparison, we also applied our method to LLaVA-Med v1.5.

Table 6 shows that VCD reduces performance across all datasets, suggesting that post-fine-tuning distribution adjustments may not improve downstream tasks. Conversely, our method significantly enhances LLaVA-Med's performance, further highlighting its effectiveness across different VLMs.

Table 6. Classification performance (Acc/F1) of the fine-tuned LLaVA-Med v1.5.

4. Notations and Grammar Issues

Thank you for identifying these issues. We have corrected the equations (1), (2), (3), (5), (6), and addressed the noted typos.

5. Does the TSM use the same vision encoder as the target VLM? How about the connector?

In our experiments, the TSM employs a ViT-Base model pre-trained on ImageNet21K, which differs from the vision encoder of the target VLM (InternViT-300M for InternVL). For the TSM connector, we use a simple MLP similar to the original VLM connector, with slight modifications to match the feature dimensions between the VLM and TSM.

1. The model can only be used for a specific task, which may hinder the original generalizability of the VLM.

Thank you for highlighting this concern. We agree that task-specific fine-tuning might affect the generalizability of the original VLM. However, with visual instruction tuning, our fine-tuned VLM gains the ability to handle new downstream tasks while preserving its original generalizability. This is possible because different tasks are guided by specific instructions and contexts, allowing them to be learned separately without too much interference. To verify this, we evaluated our fine-tuned VLM on eight benchmark datasets originally used to assess the base VLM (InternVL2-2B). Table 1 presents the results, comparing the original VLM and our fine-tuned model. The results demonstrate that our VITask method only introduces minor degradation and thus preserves the original generalizability of the VLM.

2. EP is most useful, while other modules bring marginal improvements. Are the other two modules necessary?

We appreciate this observation and would like to clarify the contributions of all proposed strategies. Each module is designed for a specific purpose:

• Exemplar Prompting (EP): Addresses the limitation of ineffective features from the pre-trained vision encoder for new tasks by leveraging specialized features from external task-specific models (TSMs).
• Contrastive Response Tuning (CRT): Mitigates the suboptimal text generation loss, optimizing downstream task performance in a more explicit way.

These two strategies independently improve task-specific performance from different perspectives, as shown in Table 2 of our paper. They can work together to achieve the best performance.

While EP is highly effective due to its use of external TSM features, it requires maintaining TSMs alongside the tuned VLM, increasing inference-time computation and memory costs. To enhance the test-time efficiency, we introduced Response Distribution Alignment (RDA), which eliminates the reliance on TSMs during inference. This improves the scalability of our method by reducing dependency on multiple TSMs for different tasks, offering a balance between effectiveness and efficiency.

3. Differences between EP and LISA.

EP and LISA serve different purposes:

• LISA: Combines a VLM with a segmentation model for interactive segmentation. It uses VLM features as prompts for the segmentation model to enable referring image segmentation, primarily allowing the VLM to control the predictions from the segmentation model.
• EP: Incorporates specialized features from TSMs into the VLM, enabling the pre-trained VLM to perform new tasks as effectively as task-specific models.

4. Comprehensiveness of the experiments.

To improve comprehensiveness, we compared our method with additional baseline models such as LLaVAv1.5-7B, Qwen2-VL-7B, InternVL2-8B, and LLaVA-Med v1.5. Results in Table 2 demonstrate that highlight that our method outperforms the standard fine-tuning approach. Moreover, as a new visual instruction tuning paradigm, our method can also be applied to medical VLMs and further enhances the performance of the fine-tuned LLaVA-Med.

5. Effectiveness beyond the medical field.

Thank you for the suggestion. To validate the generality of our method, we conducted additional experiments. As shown in Tables 2, 3, and 4, our method consistently enhances VLM performance across different models, downstream tasks, datasets, and domains, including both medical and non-medical fields.

Dear Reviewer fSNV,

We hope this message finds you well. Over three days ago, we submitted our detailed rebuttal, thoroughly addressing all your valuable feedback. We sincerely appreciate your insights, which have been instrumental in improving our work, and we are grateful for the opportunity to clarify and strengthen our contributions.

We kindly request your attention to review our rebuttal and to reconsider our work in light of the clarifications and additional evidence we provided. Please let us know if there are any remaining questions or areas requiring further elaboration.

Thank you once again for your time, thoughtful feedback, and consideration.

Best regards,
The Authors

3. According to the feedback and suggestions from all reviewers, we validated the effectiveness of our method on tasks beyond classification. Specifically, we used the pre-trained Med-SAM2D [12] as the TSM and applied our method to medical phrase grounding on the Med-GRIT-270 dataset [11]. This dataset contains 270k question-and-answer pairs across eight medical imaging modalities and evaluates bounding box prediction for given medical findings.

Table 3. Grounding performance (Recall@0.5) of the fine-tuned VLMs on Med-GRIT-270.

---

4. Following the suggestions of Reviewer fSNV and Z7hL, we evaluated our method on natural images and demonstrated its effectiveness. Additional experiments were conducted using three natural image classification datasets: Stanford Cars [8], Flowers 102 [9], and Caltech 101 [10].

Table 4. Classification performance (Acc.) of the fine-tuned VLM on natural image datasets.

---

5. Following the feedback from all reviewers, we corrected typos and improved the overall presentation of the paper. All revised contents are highlighted in blue for clarity.

---

References

[1] Liu, Y., Duan, H., Li, B., Zhang, Y., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., Chen, K., & Lin, D. (2023). MMbench: Is your multi-modal model an all-around player? arXiv:2307.06281.

[2] Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., & Wen, J.-R. (2023). Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355.

[3] Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2023). MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. arXiv:2311.16502.

[4] Tong, S., et al. (2024). Eyes wide shut? Exploring the visual shortcomings of multimodal LLMs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. [5] Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., & Wang, L. (2024). MM-Vet: Evaluating large multimodal models for integrated capabilities. ICML.

[6] Hudson, D. A., & Manning, C. D. (2019). GQA: A new dataset for real-world visual reasoning and compositional question answering. CVPR, 6700–6709.

[7] Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., & Farhadi, A. (2016). A diagram is worth a dozen images. ECCV, 235–251.

[8] Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., & Fei-Fei, L. (2017). Fine-grained car detection for visual census estimation. AAAI.

[9] Nilsback, M. E., & Zisserman, A. (2008). Automated flower classification over a large number of classes. Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 722–729. IEEE.

[10] Li, F. F., Fergus, R., & Perona, P. (2006). One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).

[11] Huang, X., et al. (2024). A refer-and-ground multimodal large language model for biomedicine. International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland.`

[12] Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T., Wang, H., ... & Qiao, Y. (2023). SAM-Med2D. arXiv preprint arXiv:2308.16184.