Enhancing Large Vision Language Models with Self-Training on Image Comprehension

Thank you for the detailed feedback. Please find our responses below. We hope that our clarifications and additional experiments resolve the misunderstanding.

W1/Q3: In Table 2, much of the performance gain comes from DaR.

We respectfully point out that there is a misunderstanding of the results presented in Table 2. As we have discussed in line 278-283 of our submission, Table 2 (row 1-2) showed that DaR prompting alone even results in degradations, while only combining with STIC achieves the best result. And contrary to the your conclusion, the 3rd row for STIC without DaR clearly showed that STIC fine-tuning itself leads to significant improvements across all tasks, with the average increasing from 54.8 to 57.6. Notably, it achieves the best results on MMBench and MM-Vet, proving its soundness. In the 4th row, we combined STIC with DaR. While DaR prompting alone is not sufficient, the integration of STIC significantly boosts the model's image comprehension ability and consequently effectiveness of DaR.

More analysis of the stages is provided in our global rebuttal (Global A2).

W2: Existing self-training methods for vision models are not discussed.

Thank you for mentioning the references, we will include them in discussion in our revision. Specifically:

• Similarity: The major goal shared by the mentioned papers and STIC is designing an effective way of leveraging unlabeled data to improve the performance of current models.
• Differences:
  1. (Focus/Model) The mentioned papers focus on representation learning of deep learning models. Meanwhile, STIC focuses on the vision LLMs, and the backbone remains an LLM. While previous deep learning models are grounded in representation and further specialized in specific task, LLMs are autoregressive and easily generalize to different tasks. Instead of training for better image representations, STIC aims to gather synthetic data for the LLM to produce higher-quality responses to a query on an image.
  2. (Algorithm) We focus on alignment fine-tuning. Notably, classic self-training algorithms for vision models do not employ alignment algorithms like RLHF or DPO. While providing a positive training signal is beneficial, having negative examples is crucial for the success of LLM alignment. As shown in Table 3 of our submission, including only positive examples for SFT is not as effective as having pairwise preference data.

W3: A small amount of unlabeled images

In our discussion with POVID in Section 6 Figure 5, we highlighted the data efficiency of STIC. While POVID uses 17k SFT data, STIC achieves better results with a total of 11k data (5k SFT and 6k unlabeled). STIC requires a smaller amount of data to achieve significant improvements.

We conducted an additional experiment using 30k unlabeled images, as shown in Figure 1 of our attached one-page pdf. The results demonstrate the scalability of STIC when applied to larger datasets.

Q1: Why aren’t model-generated description 𝑦_des used for fine-tuning?

We will correct this typo in Algorithm 2 in our revision. The correct data used for fine-tuning should be $([v^{(i)}, y_{des}, x^{(i)}], y^{(i)})$. Algorithm 2 indeed aims to use the model-generated description and append it before the question prompt for fine-tuning.

Q2: Motivation of Stage 2

As explained in our method section, stage 2 is aimed to further fine-tune the model to leverage self-generated image descriptions for downstream tasks, and thus help ground its reasoning ability on such descriptions. While stage 1 improves the model’s ability in image description, it does not focus on leveraging these descriptions for subsequent tasks such as question answering. Stage 2 fine-tunes the model by reusing a small amount of its SFT data and infusing it with self-generated descriptions to specifically strengthen model's reasoning ability based on descriptions.

Q4: How much does each stage contribute to performance improvement?

Thank you for raising this important point. Please see our response in Global A2.

Q5: DaR was not mentioned early.

DaR was discussed in line 211-213 of our submission and was not elaborated due to page limit. Please see our elaborated explanation in Global A3 of our global rebuttal. We will add a paragraph in our method section in revision.

Q6: Why STIC randomly selects unlabeled images.

Data selection is indeed a promising future direction and we will include this discussion in revision.

The current implementation of STIC randomly selects unlabeled images as it simplifies the process and ensures a broad and diverse sampling of data. In Table 3 of our attached one-page pdf, we included an experiment using the same amount of images but more diverse distribution (Vision Flan) for stage 1. Notably, the increased diversity led to further improvements in STIC, suggesting the potential for enhancement with better sets of unlabeled images.

Q7: How can we guarantee that all samples generated by LVLM on clean images are preferred?

Regarding the specific question on clean images vs. corrupted images, we note that this approach has become widely-used and tested in concurrent works focusing on LVLM alignment [1]. In our paper, Figures 3, 9, and 10 also showed that image corruptions indeed cause observable declines in model output quality. Please also see our response in the global rebuttal (Global A1) for a detailed explanation on preference alignment.

[1] Aligning modalities in vision large language models via preference fine-tuning.

We appreciate your feedback and make clarifications as below.

W1. If there is any method needed to ensure the correctness of the well-crafted prompt?

Thank you for raising this important question. The concern regarding the complexity of prompts is indeed crucial. To address this, we implemented restrictions and human filtering on multiple candidate prompts generated by GPT-4 to ensure they function as intended. This process can be viewed as a behavior distillation from a stronger model.

To ensure effectiveness, we tested these prompts on MSCOCO samples and verified the LVLM’s instruction-following quality through human evaluation. We provide a detailed explanation along with additional experiments in our global rebuttal (Global A1), and we will include this discussion and the additional experiments in our revision.

Regarding the concern about instruction-following abilities, which may be weaker in untrained models, we found that DPO alignment fine-tuning significantly enhances this capability. This allows the model to learn not only the preferred response but also to identify and avoid dispreferred and often erroneous responses.

Verification remains an intrinsically challenging problem, especially since there is no ground truth answer in the context of image description tasks with unlabeled images. To mitigate this, we applied a human filtering strategy for the prompts, which has proven effective and not costly. To scale up our method and move towards a fully autonomous framework, we plan to involve a critic model in the process in our future studies.

Q1. Have the authors changed the default latex template?

We apologize for any confusion. We did not intentionally alter the template. It appears we inadvertently included the submission ID. We appreciate your understanding and will ensure this mistake is corrected. We checked that with the new template, the paper can still be fit into the page limit.

Thank you very much for your support and the constructive feedback that helped us improve our work. Please see our detailed response with additional experiments below.

W1a: Principles behind the prompt design.

In short, we use GPT-4 to generate and sample multiple initial prompts, which are then refined through human filtering. To ensure effectiveness, we test these prompts on MSCOCO samples, verifying their ability to produce well-structured and relevant responses from the model. Using DaR performance as an evaluation to the prompts (Table 2 of the attached one-page pdf), we showed that the better crafted prompts result in better performance for DaR even on plain models.

Please see our detailed response in Global A1 in the global rebuttal. We will include the discussion and experiments in our revision.

W1b: Is the performance gain dependent on MSCOCO data?

Thank you for raising this important point. To address this, we conducted additional experiments using images from various sources. Specifically, we utilized the Vision Flan dataset (VFLAN: https://huggingface.co/datasets/Vision-Flan/vision-flan_191-task_1k) for stage 1 image comprehension self-training. This dataset includes images from 191 diverse vision tasks, providing a broad spectrum of image types.

We ensured a fair comparison by maintaining the same sample size (randomly sampled 6,000 images) and have presented the experimental results in Table 3 of the attached one-page pdf. The results indicate that our approach improves consistently across different datasets, demonstrating its robustness and adaptability. Notably, the increased diversity of VFLAN led to further improvements in STIC, suggesting the potential for even greater enhancement with better sets of unlabeled images. This finding aligns with our analysis in Figure 8 of the main paper, where we observed a positive correlation between the overlap of MSCOCO's image distribution with a benchmark and the performance gains achieved by STIC on that benchmark.

W2: It’s better to compare model generations before and after using STIC.

Thank you very much for the suggestion. In Figure 1 of our submission, we showed an example of model generation before and after using STIC. In line 75-76, we provided a discussion that STIC improves the model response by successfully identifying the key visual information for subsequent reasoning.

In the attached one-page pdf, we included two additional model generation examples before and after applying STIC, as illustrated in Figures 2 and 3. Despite the task being focused on mathematical reasoning, STIC enhanced the model’s response by improving its image comprehension capabilities. While the original model merely identified one of the recognized numbers in the image as the final answer, the STIC fine-tuned model was able to interpret the meaning of each number, describe them accurately, and perform reasoning based on this understanding.

Furthermore, in Figure 8 and its corresponding ablation study of our main paper, we examined the improvement in ScienceQA, where it shares a great overlap between the MSCOCO image distribution.

W3: Further explanations on DaR prompting.

We apologize for the lack of clarity in our initial presentation on DaR. Please see our explanation in Global A3 of our global rebuttal. We will add a paragraph in our method section to fully illustrate DaR in the revision.

Q1: Difference between the image captioning prompt set and the well-curated prompt.

We detailed the image captioning prompt set in our answer to Q6. Here are the key differences between the image-captioning prompt and well-curated captioning prompt:

Image Captioning Prompt Set: This set comprises concise and straightforward prompts designed to elicit basic image descriptions. These prompts typically ask the model to describe the image in simple terms without additional guidance or structure.

• Purpose: The prompts in set P serve as the target task.

Well-Curated Prompt: These prompts are designed to be more elaborate and structured, crafted to elicit higher-quality responses by encouraging the model to engage in a more systematic reasoning process.

• Purpose: Generate superior responses to provide a learning signal for preferred/positive responses. This process alone (without the negative/dispreferred responses) is similar to the currently popular method called system 2 distillation [1], where the model is fine-tuned on its responses generated from a more complex, step-by-step prompt. The goal is to teach the model to apply the enhanced reasoning patterns induced by the well-curated prompts when responding to the simpler prompts (e.g. in set P).

We will add the above explanation in our next revision.

[1] Distilling System 2 into System 1

Q2: What is the prompt 𝑥 used in DPO in stage 1?

We included some of the prompts below due to character limit and will include the full set of eight prompts in our future revision.

• "Illustrate the details of the picture.",
• "Summarize the visual content presented.",
• "Explain what is depicted in the photograph.",
• …

Q3: Model performance if only stage 1 is performed.

Thank you for raising this important point. In short, while stage 1 focuses exclusively on enhancing the perception capabilities of LVLM, it still notably improves performance on downstream VQA tasks (1.1% accuracy gain on ScienceQA).

Please see our detailed response in Global A2 on the progression of stages.

Thank you for your strong support and valuable feedback! We address your major comment as follows.

W1: The scalability of STIC

To explore STIC's applicability to models with higher representation capacity, we conducted supplementary experiments using LLaVA-v1.6 (Vicuna-13B).

Table 1 in the attached one-page pdf shows the detailed and comprehensive experiment results. Due to compute and time constraints, we included the three benchmarks (LLaVA-Bench, MM-Vet and MMBench) that include various different tasks and can comprehensively evaluate the model’s performance. We used the same images for STIC fine-tuning as in our experiments for LLaVA-v1.6 (Mistral-7B) to ensure fairness and the same set of hyperparameters due to time constraint. The improvements observed with LLaVA-v1.6 (Vicuna-13B) demonstrate that STIC is not only effective with smaller models but also scales well with larger or more capable LVLMs. It also shows potential for further improvement through hyperparameter tuning, data filtering, and enhanced data generation.

We hope that our additional experiments address your raised concerns. Let us know if there remain further questions, and we are happy to discuss them.

Q1: What if we try the proposed method with different LVLM models?

Thank you for this insightful question. In our original and additional experiments, we employed STIC with LLaVA-v1.5 and LLaVA-v1.6, incorporating various LLM backbones at different scales, specifically Vicuna-7B, Mistral-7B, and Vicuna-13B. These models, all incorporating strong visual encoders from the CLIP family, demonstrated effective improvements with STIC. While our current exploration of different models was constrained by time and computational resources, we recognize the importance and potential of exploring a wider range of LVLM models. Future work could investigate models with diverse architectures and LLM backbones, such as Llama-3, to further explore the potential of our proposed method.

I keep my original score and tend to accept this paper.