HallE-Switch: Rethinking and Controlling Object Existence Hallucinations in Large Vision-Language Models for Detailed Caption

Thanks for your feedback! We have largely updated Appendix and Figures to incorporate your suggestions. We also updated the related work section to discuss additional LVLMs. Here, we address the major concerns as following:

1. Although it is interesting to argue that not all hallucination is bad, I don't think the authors successfully supported the argument with examples showing when hallucination is beneficial. With that said, more visualizations like example captions may help better explain the hallucination behavior.

We added a section A.1 in Appendix page 1 to support our argument around benefits from parametric knowledge which may also be called “hallucination”. In our study, we define "object existence hallucination" to be a phenomenon where a description makes reference to objects that are not present in the image. However, such hallucinations, when properly harnessed, can be regarded as instances of imagination. Human beings frequently use imagination to successfully accomplish tasks, often without even realizing it.

2. There could be more specific illustrations on how the training data was generated using GPT-4.

From our understanding, the reviewer needs an illustrations for the data generation process in our method. We update Figure 2 in page 6 to show how we use an open vocabulary detection model to filter objects in ground truth, and recreate detailed caption data based on the filtered objects with GPT4. For how to use GPT4 to generate detailed caption data, we followed a method from LLaVA, by using objects, corresponding bounding boxes, and short captions to generate detailed captions.

3. Related work section doesn't really provide any useful information connecting existing work and the proposed work. For example, some references are missing such as [1].

We rewrite the related work and notably cite the paper you mentioned. We find this paper proposes the that label distribution may influence hallucination in image caption, which is much earlier than POPE [2] and LRV [3]. It reduces caption hallucination by balancing ground truth object labels, and adding detection labels into language models, which is really impressive to raise the idea before LLMs were widely used. Our work looks into another aspect which is the alignment issue - hallucination happened in detail captions may be caused by the perceivable-vision and language misalignment. Our finding does not contradict with this previous work that balances the labels distribution to the hallucination in captions. As we can treat balancing objects as a reduction of parametric knowledge, which makes the imagination not so clear and confident. Therefore, it is really nice to see, our work’s findings and experiments can echo previous works and provide new ways to understand the reason for hallucination.

[1] Biten, Ali Furkan, Lluís Gómez, and Dimosthenis Karatzas. "Let there be a clock on the beach: Reducing object hallucination in image captioning." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.

[2] Li, Yifan, et al. "Evaluating object hallucination in large vision-language models." arXiv preprint arXiv:2305.10355 (2023).

[3] Liu, Fuxiao, et al. "Aligning Large Multi-Modal Model with Robust Instruction Tuning." arXiv preprint arXiv:2306.14565 (2023).

Thanks for your feedback! We have fixed the typos and double checked the paper again. We also updated the related work section to discuss your suggestion on additional LVLMs. Here, we address the major concerns as following:

1. The overall story is not very coherent. First, the details of CCEval are not very clearly described. Then, analysis is conducted on two or three methods with some conclusions drawn. However, the observation mentioned in the paper seems not to have a specific relation with the proposed Hallu-Switch method. The technique is also only evaluated on CCEval, but previous benchmarks are discussed and used in this paper.

We frame our overall story as defining, analyzing, and addressing the problem.

1. How defining: Benchmarks that are based on VQA cannot reflect object hallucination for detailed caption. Models did a great job in MME, POPE, does not promise detailed captions without hallucination. We propose a new evaluation metric, CCEval, to measure object existence hallucination reliability.

2. How defining leads to our analysis: Using CCEval, we can evaluate LVLMs by controlling components to find out the cause of hallucination; How analysis: We analysis language decoder, resolution, and data size to conclude that misalignment between perceivable object in image and object caption form parametric knowledge, which is hallucination.

3. How analysis leads to our approach: We can decrease the amount of parametric knowledge within the model to reduce hallucination; How our approach addresses the problem: We use HallE-Switch to control the contextual and parametric knowledge.

2. Why technique is only evaluated on CCEval, but previous benchmarks are discussed and used in this paper?

Technique is only evaluated on CCEval because benchmarks based on VQA cannot reflect object hallucination for detailed caption. Models did a great job in MME, POPE (benchmark for evaluating object existence hallucination by using VQA task), does not promise detailed captions without hallucination. Previous benchmarks are compared with CCEval for showing that model performance on VQA-based benchmark is not correlated to performance on CCEval in the analysis part, so that we can only adhere to CCEval when it comes to our technique, which controls hallucination specifically for detailed captions.

3. The reviewer would expect more insights or explanations about why HallE-Switch works.

We provide theoretical explanations on why HallE-Switch works in A.9 in Appendix page 5 and 6. In the new version, we updated the section for readability. Here is a short summarization of why it works:

Assuming LLM is good enough to represent an equivalent distribution with Hidden Markov Chain, then there exists an matrix $W$, so that after transferring word embedding $E$ to $WE$, the LLM's originally simulate the text distribution starting with the initial state $\pi$ will turn out to be equivalent to a different initial state. Corresponding to our analysis, the switch method changes the model from a language model with parametric knowledge to a model cutting the parametric knowledge.

Thanks for your feedback! We updated the related work section to reflect your suggestions. HaELM is a LLM-based hallucination evaluation framework, which uses LLM trained on human annotations, and can evaluate a hallucination in image captions, with low cost. This work can make our CCEval GPT-4 free. I think this work considers realistic things for academic researchers. In the future, we will consider to follow this work to decrease the cost of CCEval.

Thanks for your feedback! We have updated the paper for better presentation. Here, we address the major concerns as following:

- Presentation needs polishing. The paper is very dense in text and requires some effort for reading. Also, please address the points in "Questions" section.

Please refer to the Questions section.

- The first part and second part of the paper looks more like two separate and condensed papers. Due to this problem, especially the first part fails to deepen our insight about the root cause of the problem. I would expect a deeper treatment on each part.

We refer the reviewer to revised Section 2 Conclusion in page 6 where it summarizes the entire first part and proposes an explanation on the root cause of the problem. We also revised Section 3 Intro in page 6 to better connect with Section 2. In short, the root cause is the misalignment between perceivable vision and language data, developing contextual and parametric knowledge, which is hallucination. We also refer the reviewer to our first reply for reviewer DU9c for a run through of the overall story.

- Results are mostly quantitative. It would be better to show more qualitative examples of hallucination.

We add a section A.8 in Appendix page 5 to show the caption differences with different switch values on 5 images. The qualitative results align with the quantitative ones.

Questions:

1. In page 4, how are "consistent constraints" enforced?

We want to clarify that we do not strictly enforce consistent constraints, neither this is realistic to achieve. Although our intention is to make the model comparable and push the model to express more objects, it is sometimes not possible for some of the models, such as BLIP2, to express detailed captions, and our coverage, average length, and average number of object metrics in CCEval can accurately reflect if any model has such limitations.

In the paper, "consistent constraints'' are mainly achieved through prompting. We use the same prompt - “describe this image in detail” - across all four models in Table 2. Simply prompting can result in similar length and number of objects captions in this case since the instruction finetuning data for detailed captioning all comes from a similar source (LLaVA synthetic data). For models with different instruction finetuning data, we suggest further adjustment of the prompt or set the min_length and max_length in the inference function.

2. Section 3.2 is not very clear to me. Does W correspond to "Projector" in Fig 2? According to Fig 2, W comes after LLM. However, equation says the opposite. Is the epsilon parameter applied on word-by-word basis or sentence-by-sentence basis?

To explain, $W$ is a projector and it is inserted between LLM backbone and LLM head. Therefore, we have to pass the projector into $M$, where $M$ is LLM backbone + head, resulted in the equation $M’ = M(eps*W)$. We update the Figure 2 and Section 3.2 in page 6 and 7. In the new version, we separately represent LLM backbone as $B$ and LLM head as $H$ in both equation and figure, so it should be clearer.

The epsilon parameter is the same value for every word in one caption and multiplied with the projector W. Therefore, the answer is word-by-word basis since the model processes each word, but it is the same for every word in one caption.

3. In page 2, object relationship hallucination is mentioned but this concept does not seem to appear again later pages, in metrics or methods presented in the paper. Did I misunderstood?

In this work, we intentionally only focus on object existence hallucination as mentioned in Introduction. We consider object attributes and relationship hallucination as our future works. The reason is that - it is not surprising that existing LVLMs fail on expressing object attribute and relationship, because MSCOCO dataset does not include this information, and existing LVLMs’ caption data heavily rely on this dataset. However, it is surprising that object existence hallucination is still an issue for these models, because current detailed caption training data is accurate in object existence. Therefore, we decided to tackle this problem first.

4. Do you observe any downsides or limitations of using this method that were not expressed in the result Table?

One of the limitations is that the method reaches the best performance with larger image resolution, which aligns with our finding. However, increasing image resolution inevitably leads to computation cost.

Another limitation is the extrapolation ability of the switch. Since we train with epsilon +1 and -1, it makes sense to interpolate well when epsilon is between +1 and -1 during inference, and our Table 8 result on CCEval meet this expectation. However, we thought it would be very interesting to test the extrapolation ability of the switch. Specifically, given a switch is trained using epsilon +1 and -1, can we use epsilon < -1 during inference? It would be very nice if the hallucination continued to decrease for epsilon < -1. Unfortunately, hallucination did not decrease further with -1.2, -1.5. In addition, the model starts to repeat and loss language ability if epsilon is outside of range [1, -1]:

Example captions with switch value -1.2: The scene unfolds on a dining table, which is neatly arranged with a variety of food items. There are two plates of food, one containing a sandwich and the other with a piece of cake. The sandwich is located towards the left side of the table, while the cake is positioned towards the right. In addition to the food, there are several utensils scattered across the table. A knife is located towards the left side of the table, while a fork is positioned towards the right. There are also two spoons, one near the center of the table and the other towards the right side. (The table is not just for food; there are also several books scattered across the surface. One book is located towards the left side of the table, while another is positioned towards the right. The third book is located towards the center of the table.) x N times

Example captions with switch value -1.5: The scene unfolds on a bustling city street, where a man is walking across the street. He is wearing a backpack and appears to be in a hurry. There are several other people scattered around the scene, each engaged in their own activities. Starting from the left side of the scene, there's a car parked on the side of the street. Moving towards the right, there's another car, followed by a truck. Further to the right, there's a traffic light, indicating the direction of the flow of traffic. Starting from the top of the scene, there's a person standing near the car on the left side. Moving down, there's another person standing near the car on the right side. Further down, there's a person standing near the truck. (Starting from the top of the scene, there's another person standing near the car on the left side. Moving down, there's another person standing near the car on the right side.) x N times