Mitigating Object Hallucination in Large Vision Language Model with Human-Free Reinforcement Learning

2. In addition, the title of this paper claims to mitigate object hallucinations in large vision language models. Can other vision-language tasks like VQA or visual chatting also benefit from the proposed RL tuning process? Only considering image captioning tasks cannot fully support the claim of mitigating object hallucinations of LVLMs.

In this paper, we focus on image captioning to provide a more in-depth analysis, leaving other tasks for future investigation. To answer the reviewer’s question, our reinforcement learning framework is flexible and can work with different queries, not limited to captioning or Q&A, without specific adjustments. However, evaluating its effectiveness on tasks beyond captioning is outside this paper’s scope.

As a preliminary test, we adapted the COCO dataset to a VQA task with the prompt, "What is in the given image?" Our model achieved CHAIRi and CHAIRs scores of 7.9 and 20.5, respectively, compared to instructBLIP's 22.8 and 40.1. This indicates potential but suggests further exploration is needed for VQA tasks.

3. From the qualitative results shown in Section I of the Appendix, the outputs from the proposed method are shorter than those of the base model. As we know, the outputs from LVLMs are generally more detailed and comprehensive than the ground truth of the COCO dataset. The results in the Appendix show that the detailed captioning properties of LVLMs are destroyed. More clarifications are encouraged.

It is worth noting that even the shortened captions generated by our method remain significantly longer than the ground-truth captions. The average lengths are 85, 110, and 10 for our method, the baseline, and the ground truth, respectively. This concern is similar to Reviewer 2’s comment, and we briefly address it in the Discussion section, "On the detailed caption length" (lines 519–520). To clarify further, we provide a detailed explanation here.

The proposed framework under reinforcement learning (RL) optimizes the output to satisfy the reward function, which may implicitly result in shorter captions. However, explicit constraints are in place to prevent undesirable behavior. These include preserving object coverage (the Recall component of the F1 Score) and ensuring that the KL divergence from the original output does not deviate excessively. These constraints ensure that the method does not simply shorten captions arbitrarily but focuses on improving object-mention accuracy (the Precision component of the F1 Score) and coverage. Therefore, the shortening effect is merely a side effect of optimization which the primary goals can dominate.

As discussed in the Discussion section, checking the actual outputs reveals that, while the captions are shorter, they maintain factual accuracy and eliminate imaginative or hallucinatory content. We refer the reviewer to the qualitative results in the Appendix for further validation. We argue that this shortening effect is not necessarily negative, as it offers a viable alternative: a concise, accurate caption may be preferable to a longer one with imaginative details, depending on user needs.

Additionally, we provide an option to encourage the model to generate longer captions by incorporating a length component into the reward function (lines 523- 525). This adjustment results in slightly longer captions, with a modest decrease in hallucination reduction. However, the hallucination rate remains significantly lower compared to the baseline, demonstrating the flexibility of the proposed framework.

Question

1. This paper only considers InstructBLIP to be the LVLM. Are other commonly used LVLMs, such as LLaVA or VILA, also applicable to this proposed RL tuning framework?

Yes, our method is agnostic to the specific large vision-language model (LVLM) used. The framework simply augments the model by prefixing a soft prompt to the query and adding a linear layer on top of the latest hidden latents. This design is compatible with and can be easily implemented on any LVLM.

As suggested by another reviewer, we conducted experiments with LLaVA and provide the results below:

These results demonstrate that our method can be seamlessly integrated into LLaVA without any issues. Other LVLMs, such as VILA, can also be incorporated similarly.

We thank the reviewer for the constructive feedback and provide our response as follows:

Weaknesses:

1. This paper trains and evaluates the models on the COCO dataset. While this training manner avoids additional effort for collecting and annotating human preference data, I’m wondering whether the original capabilities of LVLMs (i.e., base model) can be adequately preserved. This paper only demonstrates the results of captioning quality on the COCO dataset in Figure 5. However, does the tuned model still maintain the captioning capability of the base model compared to other datasets? My concern is whether the original generalizability of the base model would be destroyed after being tuned by the proposed method.

Thank you for your insightful concern. The preservation of captioning quality is ensured by the use of KL divergence in our framework. KL divergence helps maintain alignment with the baseline model (referred to as the reference model in our framework), preventing the tuned model from deviating too much from the original output. Without this mechanism, training could prioritize specific metrics like F1 score, potentially disrupting the caption structure, as we observed in our initial experiments without KL divergence.

3. Using prompt tuning to reduce object hallucination has indeed shown some effectiveness. However, the prompts used by the authors are automatically generated without human intervention, which means these prompts themselves might contain hallucinations or errors. Thus, evaluating the accuracy and validity of these generated samples becomes a critical issue. The authors should provide additional analysis to demonstrate the reliability of these prompts to better support the effectiveness of this approach.

We may not have fully captured the reviewer’s concern, but we interpret it as a suggestion that the method may be suboptimal and requires further experimentation to validate its effectiveness. To clarify, the "prompt" in our method, more specifically the “soft prompt,” consists of trainable parameters (not human-readable prompts). These parameters are optimized via reinforcement learning (RL) to guide the model in avoiding hallucinated objects.

Regarding optimization, the effectiveness of this approach—like any RL-based method—depends on how well RL is performed. If RL is executed correctly and converges to an optimal solution, it can generally provide significant reductions in object hallucination across datasets. While suboptimal solutions may still exist (as you pointed out), our experiments show that the optimization process in this paper has been effectively executed. Empirical evidence demonstrates consistently low object hallucination rates across datasets while maintaining good caption quality (e.g., maintaining BLEU, BERTScore, and SPICE metrics).

In response to Reviewer 1’s suggestion, we further applied our method to the base LLaVA model. Training was conducted for 50 epochs using the same hyperparameters as InstructBlip. The results are as follow

The results may improve further with careful hyperparameter selection. Nevertheless, these results demonstrate that our method can be effectively applied to a broader range of baseline models.

We thank the reviewer for the constructive feedback and provide our response as follows:

Weaknesses:

1. This paper currently validates the effectiveness of the proposed method only based on InstructBLIP, making it difficult to verify its broader applicability. The method’s generalizability across different baselines, such as mPLUG-Owl and LLaVA, should be demonstrated.

We provide additional experimental result, further applying our method to the base LLaVA model. Training was conducted for 50 epochs using the same hyperparameters as InstructBlip. The results are shown below:

The results may improve further with careful hyperparameter selection. Nevertheless, these results demonstrate that our method can be effectively applied to a broader range of baseline models.

2.1 The authors use the F1 Score to address object hallucination, but this approach is not well-motivated. While F1 Score balances accuracy and recall, it does not have a direct connection with the issue of object hallucination. The authors should provide more theoretical and experimental analysis to support their rationale

The reviewer may have misunderstood the formulation of the F1 Score used in our method. The F1 Score combines precision and recall, where precision is calculated as the ratio of correctly predicted objects to the total number of predicted objects. This can be expressed as:

An increase in precision directly correlates with a decrease in the object hallucination rate. Thus, improving precision inherently reduces hallucination, making the use of the F1 Score well-suited for this task.

2.2 In lines 521-523, they mention a minor side effect of their method: the average caption length is shorter than the baseline (85 tokens compared to 110 tokens). This raises the question of whether their proposed framework reduces object hallucination simply by shortening the output length.

The proposed framework under reinforcement learning (RL) will optimize the output to satisfy the reward function by any means, which may implicitly involve shortening the output length. However, there are explicit constraints that prevent undesirable behavior, such as preserving object coverage (Recall part of the F1 Score) and ensuring that KL divergence does not deviate too far from the original output. These constraints prevent the method from simply shortening the sentence, focusing instead on improving object-mention accuracy and coverage. The shortening is merely a side effect.

As we point out and discuss in the Discussion section, "On the detailed caption length," checking the actual output shows that even though the caption is shorter, it maintains factual content and eliminates imaginative content. We direct the reviewer to the qualitative results in the appendix for further validation. We argue that this shortening effect is not necessarily negative but offers a viable option: a concise, accurate caption may be preferable to a longer one with imaginative details, depending on user needs. It is also worth noting that even the shortened captions generated by our method are still significantly longer than the ground-truth captions. The average lengths are 85, 110, and 10 for our method, the baseline, and the ground truth, respectively.

Furthermore, the authors also provide a way to encourage the model to generate longer captions by incorporating longer captions into the reward function. This results in a slight decrease in hallucination but still shows a significant reduction compared to the baseline.

Thank you for taking the time to review my paper. I have carefully addressed your feedback in the rebuttal, providing clarifications and updates that I believe significantly enhance the paper. As the discussion period approaches its deadline, I kindly remind you to review the feedback and, if appropriate, consider revising your score. Thank you again for your valuable input and consideration.

3. The paper relies pretrained an object detector to eliminate human annotations. Therefore, the performance of the model relies on and might be biased to the object detector used. There is no discussion on this point in the paper.

In case no ground true object is available, the performance of hallucination reduction depend on the accuracy of the object detector used. More accurate ground true object extraction result in better hallucination reduction. To provide a sense of how the object detector compares to ground-truth labels, we conducted experiments using YOLOv8x and YOLOv11x to perform object extraction on COCO images. We perform detection and select objects with a detection confidence score > 0.7 to form a list of ground-truth objects.

Using our framework to fine-tune InstructBlip with the YOLO-based ground-truth objects and evaluating using COCO ground-truth objects, we obtained the following results:

As observed, the hallucination rate is not as low as when using ground-truth objects. However, the results demonstrate that even with less accurate object detectors, our framework effectively reduces hallucination. This also show that the more accurate the object detector is, the better the hallucination reduction that can be achieved.

4. More recent baselines should be compared. The baselines (LLAVA, mPLUG-Owl) in Table 4 are from 2023. New versions of LLAVA and mPLUG-Owl are introduced in 2024.

We provide additional results for the latest version of LLaVA (LLaVA-Next):

For the latest mPLUG-Owl3, we encountered bugs during the prediction of the given model, so the results are currently unavailable.

Question

1. What is the reason for adopting a different metric for the Genome dataset? why not use F1 or CHAIR?

The CHAIR metric has a limitation due to its reliance on string matching for a fixed set of objects. While this makes it suitable for COCO, it is less accurate for datasets like Visual Genome, which contain a larger number of objects per image and a broader range of object categories. Additionally, we follow the evaluation methodology of HallE-Switch to provide a comparable evaluation that can be referenced in both papers.

We thank the reviewer for the constructive feedback and provide our response as follows:

Weaknesses:

1. The following two relevant works are ignored in the paper.

We appreciate the reviewers’ suggestions to consider additional cited papers. These studies, which propose valuable benchmarks and corresponding metrics, offer important insights for open-vocabulary evaluation beyond the COCO dataset. We will incorporate them into the related work section to enhance the paper’s comprehensiveness.

Specifically, the study “Mitigating Open-Vocabulary Caption Hallucinations” introduces the OpenChair benchmark, which extends CHAIR to a broader object vocabulary than COCO and suggest a evaluation method using large language models (LLMs) to identify hallucinated objects. This approach is claimed to complement CHAIR for experiments beyond the COCO dataset. Similarly, ALOHa highlights limitations in CHAIR due to its reliance on string matching for a fixed set of objects, making it suitable for COCO but less generalizable to other datasets. To address this, ALOHa employs LLMs to identify objects in open-vocabulary settings, enhancing its adaptability.

In our approach, we evaluate both a closed dataset (COCO) and an open-vocabulary dataset (Visual Genome). For COCO, we selected CHAIR due to its specialization for this dataset, which ensures reliable and consistent results. For Visual Genome, we chose CCEVAL, which extends CHAIR’s methodology by incorporating LLMs to more effectively capture objects in open-vocabulary settings, with a focus on the Visual Genome dataset. It is worth noting that CCEVAL, OpenChair, and ALOHa all identify similar limitations of CHAIR and share a common approach of using LLM support for a more generalized application.

2. The motivation of using reinforcement learning is not clearly explained.

We appreciate the feedback regarding our use of reinforcement learning (RL) to reduce hallucinations in large vision-language models (LVLMs). Our motivation for employing RL stems from the need to minimize human effort while ensuring effectively reduct hallucination.

Traditional approaches to mitigating hallucinations often require identifying specific sources of hallucination and designing targeted strategies to counter them. While effective, these methods are labor-intensive. Data-driven alternatives like supervised learning provide some level of automation but rely heavily on labeled datasets, which still require significant human input for data annotation and curation—an increasingly costly and time-intensive process, particularly for large-scale models.

In contrast, reinforcement learning in the literature not only demonstrates strong alignment capabilities for LVLMs in tasks like image captioning but also offers a promising path to automation by significantly reducing the need for explicit labels (e.g., relying only on simple binary feedback for reward modeling). We are motivated to push this approach to its limits by completely eliminating human-labeled data, fully leveraging RL’s potential through the exclusive use of automatic metrics to reduce hallucinations. These metrics are gradually improving in their alignment with human feedback in terms of both accuracy and reliability. Our approach enables the model to iteratively refine its outputs based solely on automatic feedback, providing an efficient and scalable solution that aligns with the trend toward larger LVLMs.