Discovering Compositional Hallucinations in LVLMs

We thank the reviewer for the feedbacks and questions. Our response is as follows:

W1: Distinction between the proposed benchmark and existing VQA benchmarks

We would like to clarify that our proposed VLComp benchmark differs fundamentally from existing knowledge-based VQA benchmarks. As stated in Lines 56–59 in the main paper, while challenges related to textual knowledge can arise in these VQA benchmarks, VLComp focuses on questions with simpler textual sub-questions, with its primary challenge being compositional hallucination. To further emphasize this distinction, we present the following comparison table as follows:

W2: Concerns about VLComp scale.

The representativeness and generalizability of VLComp have been validated from the following aspects:

• VLComp is specifically designed to diagnose and analyze SCHall. Inspired by existing hallucination evaluation benchmarks for LVLMs—such as LLaVA-Bench-in-the-Wild, MMHal-Bench, and MM-Vet, which comprise approximately 60, 96, and 128 samples, respectively—VLComp significantly expands both scale (323 samples with 951 compositional and decomposed questions in total) and scope. Importantly, it ensures strong diversity across key categories, including perception, science, commonsense reasoning, factual knowledge, linguistic capability, scene understanding, and mathematics.
• While VLComp is relatively small than more general-purpose LVLM benchmarks, our experiments show that it effectively identifies and supports exploration of this novel form of hallucination. More importantly, SCHall also emerge in general-purpose benchmarks (Figure2), suggesting that our benchmark captures a representative and generalizable issue.
• In addition, the semi-automated process for constructing VLComp involves minimal manual annotation, primarily limited to reviewing and refining the questions. This design allows for efficient and scalable expansion. In future work, we plan to extend VLComp with broader coverage and increased human involvement to further enhance its reliability and generality.

W3: Concerns on writing and logical flow.

We would like to clarify that our paper is structured to follow a logical progression centered around a concrete motivation: the discovery of a previously underexplored hallucination type in LVLMs, which we term SCHall (Simple Compositional Hallucination). The paper is organized as follows:

1. We define SCHall and motivate its importance based on empirical observations (Introduction & Section 3.1).
2. We construct VLComp, a targeted benchmark to systematically study this phenomenon (Section 2).
3. We analyze its underlying causes, identifying two contributing factors: (Section 3.2 & 3.3) (i) difficulty with visual abstraction, and (ii) interference of visual input with language reasoning.
4. We then propose VLR-distillation, a mitigation method informed by our analysis (Section 4).

In the final version, we will improve the transitions between sections to better highlight the logical flow and the relationship between sections.

Thank you for your response and for engaging in the discussion! We highlight the fundamental differences between our compositional hallucination benchmark and OK-VQA, as well as the differing treatments of hallucination and error in this study, as follows:

---

Our compositional hallucination benchmark vs. OK-VQA

We would like to clarify that our work is centered on the proposal and investigation of Simple Compositional Hallucination (SCHall), and the benchmark we introduce is purposefully designed to support this goal. The key distinction between our benchmark and existing ones is that ours is uniquely capable of revealing compositional hallucinations in a significant and systematic way. For instance, the OK-VQA benchmark is not suitable for analyzing SCHall, for the following reasons:

1. Lack of Compositionality in OK-VQA: OK-VQA is not specifically designed to evaluate compositionality but rather focuses on knowledge-based question answering. Neither the data construction process nor the evaluation in OK-VQA takes compositionality into account. As a result, a significant portion of its questions lack compositional structure. For instance, questions like "What toy is this?" are included in OK-VQA because their answers (e.g., "stuffed animal", "teddy bear") require world knowledge. However, the type of compositionality we focus on generally involves combining and reasoning over recognized visual elements in a structured manner. Moreover, in OK-VQA, some questions such as "What do they call running around the bases on a single hit?" can be answered purely based on knowledge without any visual input. Due to time constraints, we analyze the first 100 questions in the OK-VQA’s validation set and find that 59% are non-compositional in OK-VQA: 49% are direct knowledge-based recognition questions, and 10% could be answered without visual information at all.
2. The Performance Bottleneck in OK-VQA Is Not Due to Compositionality: Unlike ours, the performance bottleneck in OK-VQA does not stem from compositionality, but rather from the model’s limitations in external knowledge. To illustrate this, we decompose the all remaining compositional problems in the OK-VQA subset into visual and textual subproblems, following the same procedure as in our benchmark. The results reveal accuracies of 93.6% for visual subproblems, 73.2% for textual subproblems, and 70.7% for the original task—similar to the textual subproblems. This indicates that the bottleneck lies in the textual subproblems, which, for OK-VQA, corresponds to limitations in external knowledge. This phenomenon differs from what we observe in our SCHall benchmark (Figure 1c) and general-purpose LVLM benchmarks (Figure 2). One possible reason is that the OK-VQA has been extensively used to train LVLMs (e.g., LLaVA 1.5), and thus it may not effectively reflect SCHall during evaluation.

Error vs. Hallucination

Errors broadly refer to any incorrect model outputs, which can result from missing knowledge, reasoning failures, instruction-following misalignment, hallucinations, or other limitations. In this study, we adopt a more precise definition to differentiate hallucinations from general errors. Specifically, errors are defined as failures caused by insufficient capabilities (i.e., missing knowledge or recognition errors). In contrast, hallucinations occur when the model possesses the necessary knowledge and recognition capabilities to answer the question correctly, yet still produces a response that is inconsistent with the input or factual reality. Accordingly, SCHall is considered a hallucination rather than a general error: while the model successfully solves the subproblems it is capable of answering—demonstrating both the required knowledge and recognition capabilities—it hallucinates when addressing compositional questions derived from them. Whereas benchmarks like OK-VQA primarily capture errors arising from knowledge gaps, our benchmark is specifically designed to target SCHall.

Thanks for your feedback again. We welcome further discussion should you have any questions or require additional clarification.

We sincerely thank you for raising the score and for the timely discussion! As you have agreed, prior VQA datasets were not tailored for compositional hallucinations. In addition, we would like to gently point out that datasets such as OK-VQA are not only not specifically designed with this focus in mind, but also are less well-suited for evaluating compositionality. This is mainly because the accuracy on original OK-VQA questions closely matches that on their text-centric sub-questions, indicating that OK-VQA’s main challenge lies in external knowledge within the textual sub-questions rather than compositionality. Consequently, despite 41% of questions being decomposable, OK-VQA is not well-suited for studying compositionality.

As our benchmark, covering diverse fields, identifies compositional hallucination issues as significant (Figure 1c), and given that these issues are also prevalent in general benchmarks (Figure 2), we respectfully believe that the current benchmark is sufficient for investigating and evaluating compositional hallucinations. That said, we agree that scaling up—whether through our semi-automatic method or by first diagnosing existing benchmarks and extracting subsets containing compositional hallucinations—would further strengthen our claims. Nonetheless, we view this as an enhancement and expansion, rather than an essential prerequisite. Specifically, since the benchmark represent only part of our contribution, the main contribution of our paper also lies in being the first to identify, define, and study compositional hallucinations, as well as proposing effective novel solutions.

Once again, we sincerely thank you for raising the score, providing the timely discussion, and offering the valuable perspective on potential scaling!

We thank the reviewer for the insightful feedbacks and questions. Our response is as follows:

W1: Concerns about generalizability and annotation bias.

Thank you for your feedback. The generalizability and quality of VLComp can be validated from the following aspects:

• VLComp is specifically designed to diagnose and analyze SCHall. Inspired by existing hallucination evaluation benchmarks for LVLMs—such as LLaVA-Bench-in-the-Wild, MMHal-Bench, and MM-Vet, which comprise approximately 60, 96, and 128 samples, respectively—VLComp significantly expands both scale (323 samples with 951 compositional and decomposed questions in total) and scope. Importantly, it ensures strong diversity across key categories, including perception, science, commonsense reasoning, factual knowledge, linguistic capability, scene understanding, and mathematics.
• While VLComp is relatively small than more general-purpose LVLM benchmarks, our experiments show that it effectively identifies and supports exploration of this novel form of hallucination. More importantly, SCHall also emerge in general-purpose benchmarks (Figure2), suggesting that our benchmark captures a representative and generalizable issue.
• In addition, the semi-automated process for constructing VLComp involves minimal manual annotation, primarily limited to reviewing and refining the questions. This design allows for efficient and scalable expansion. In future work, we plan to extend VLComp with broader coverage and increased human involvement to further enhance its reliability and generality.

W2: Interpretation on failure of existing hallucination mitigation strategies on SCHall.

Thank you for the insightful comment. The ineffectiveness of existing mitigation strategies in addressing SCHall can be attributed to two primary factors:

1. While methods like PAI are effective at mitigating object hallucination and improving recognition accuracy, they are less suited to the SCHall setting, which primarily involves compositional reasoning rather than recognition failures. For instance, in Figure 5 (Math) of the Appendix, the image displays the equation “13 + 8,” which the baseline model answers correctly. However, PAI misleads the model to predict “53”—a partially visually plausible (due to the “3”) but semantically incorrect answer. This highlights that the difficulty in SCHall stems more from compositional challenge than from visual recognition errors.
2. Meanwhile, REVERIE attributes hallucinations to a lack of fine-grained supervision and improves reasoning via Reflective Instruction Tuning. However, it does not explicitly teach selective integration of visual and textual cues. For instance, as illustrated in Figure 3(a) in Appendix, where a person is clearly chopping wood at sunset, the baseline model incorrectly predict "thermal energy" based on the visual salience of sunlight. REVERIE fails to correct this, likely because both the ground-truth and hallucinated answers are visually plausible.

We appreciate the suggestion and will include additional qualitative failure cases and comparative analyses for existing methods in the final version to enhance interpretability.

W3: More evaluation results on hallucination and other VQA benchmarks.

Thank you for your suggestion! We evaluate our method with LLaVA-1.5-7B on additional hallucination benchmarks, including HallusionBench, as well as general-purpose VQA datasets, including MMBench, ScienceQA, and MM-Vet. Results are shown below.

Our method achieves consistent improvements across multiple benchmarks, demonstrating both the generality of SCHall and the effectiveness of our approach.

Q1: Types of compositional hallucinations and how do models fail in them?

Thank you for your valuable suggestion! We identify several common types of compositional hallucinations observed in our VLComp:

1. Irrelevant Visual Feature Attribution. When an image contains rich or distracting content, the model may attend to semantically irrelevant visual regions, leading to incorrect reasoning. For instance, in Figure 3a in Appendix, the image shows a person chopping wood at sunset, while the question concerns the type of energy consumed during the activity. Earlier models (e.g., LLaVA) are often misled by salient but contextually unrelated elements (e.g., the sunset), resulting in hallucinated answers.
2. Superficial Visual-Linguistic Mapping. When there is superficial alignment between visual content and candidate answers, models may rely on pattern matching instead of grounded reasoning. In Figure 3b, a dog is resting on a chair. The question asks about typical behaviors of the animal shown. Most open-source models mistakenly select “being very lazy” based on the dog’s current state, rather than reasoning about its typical behavior in general.
3. Cross-modal Misalignment. In some cases, replacing the image with its caption leads to correct answers, indicating poor alignment between visual and textual modalities. In Figure 4 in appendix, the image contains a clearly visible ruler, and the question asks what it can be used to measure. While caption-based inputs (e.g., “A ruler is shown”) yield correct responses, the same models fail with the full visual input. Such failures on simple and unambiguous images are typically observed only in earlier models (e.g., LLaVA).

Nevertheless, the interplay of these challenges often leads to failure cases that challenge even close-source models. There remain many types of compositional hallucinations that warrant deeper investigation. We will incorporate the above discussion and related analyses in the final version.

Q2: Discussion with general shortcut learning or heuristic answering strategies.

We thank the reviewer for the thoughtful question. We agree that shortcut learning and heuristic answering are important considerations and likely contribute to the observed SCHall behaviors, particularly those related to Superficial Visual-Linguistic Mapping (see response to Q1).

However, we argue that SCHall extends beyond these mechanisms. In addition to superficial patterns, SCHall captures deeper model limitations such as Irrelevant Visual Feature Attribution and Cross-modal Misalignment. These cannot be fully accounted for by shortcut learning alone. We view our analysis as an initial step, and we welcome future work that further disentangles these contributing factors.

We thank the reviewer for the insightful feedbacks and questions. Our response is as follows:

W1: Writing mistakes in Section 4.1

Thank you for pointing out the formatting issues. We will ensure that the correct condition $s \leq r$ is clearly stated and carefully revise the formatting in the final version.

W2: Lacks of deeper theoretical analysis

Indeed, our analysis of SCHall is primarily empirical. We perform error analysis on both VLComp and general benchmarks to reveal its widespread presence, and use experiments and visualizations for preliminary attribution. The competitive performance of our method on VLComp further substantiates our analysis.

We acknowledge the need for deeper theoretical analysis of why compositionality leads to hallucination at the model architecture and optimization level. Our hypothesis is that the training paradim, which supervises only the final textual output without intermediate guidance, leads the model to rely on spurious correlations and dataset biases instead of grounded compositional reasoning, thereby increasing SCHall risk. In response to this suggestion, we plan to conduct further analyses, including intermediate representation probing with logit lens and intervention validation via attention transfer.

Q1-3: Experimental Setting Details

Thank you for your question. During training, in cases where ground-truth captions are not available, we employ the baseline model itself as the caption generator. In the ablation study (left table of Table 5), we used 4 VLRs for both the with and without distillation learning (DL) settings. We further conduct additional experiments by increasing the number of VLRs without applying distillation learning (DL). The results are shown below:

These results indicate that simply increasing the number of VLRs does not match the performance achieved when combining VLRs and DL. Notably, performance even degrades when using 16 VLRs. This may be related to the nature of the POPE benchmark, where 4 tokens are sufficient to capture the necessary visual information. Adding more VLRs may introduce redundancy and noise, which could lead to hallucinations—this also addresses the concern raised in Q3.

Q4: Key differences with LLaVA-Mini

Thank you for your question. LLaVA-Mini employs explicitly designed fusion and compression modules to generate compressed and pre-fusion tokens. In contrast, our approach introduces VLRs, a novel token type that operates entirely within the LVLM, enabling it to autonomously learn to extract question-relevant image information while also engaging in textual understanding. Specifcally, the key differences are as follows:

• Our VLRs are independent tokens that bridge vision and language, whereas LLaVA-Mini uses pre-fusion tokens to replace text tokens.
• The multimodal interaction in our method is conditioned on the LVLM’s rich internal knowledge, while LLaVA-Mini performs a simpler fusion of modalities. By leveraging this internal knowledge, VLRs achieve more flexible and semantically meaningful multimodal representations.

Thank you for the valuable feedbacks and suggestions. Our response is as follows:

W1 & Limitations 1.1: Statistical information and construction process of the benchmark.

We appreciate your suggestion to provide more detailed information about the statistical properties and construction process of our benchmark.

1. Statistical Information:

   We provided detailed statistical information in the supplementary materials (Appendix C.1 and C.2), including data sources, data volume (951 total questions with 323 compositional and 628 decomposed), and category distribution (18% Perception, 15.2% Science, 12.1% Commonsense Reasoning, 9.3% Factual Knowledge, 9.3% Language Capability, 18.6% Scene Understanding and 17.6% Math). We will include more details (e.g. sub-problems distributions) and make sure to reference these clearly in the main text for better accessibility.

2. Benchmark Construction Process:

   We agree that a clearer explanation of the benchmark construction process would improve the clarity of the paper. Our benchmark construction process includes two stages: Atomic Questions Generation and Simple-Composed Question Generation.

   Atomic Question Generation

   • Image Collection: Images are collected from datasets (MMBench, MME, SEEDBench) and online sources.
   • Caption Generation: LVLMs (LLaVA, Qwen-VL, MiniGPT-4, InternVL) generate captions for each image.
   • Visual-centric Question Formulation: GPT-3.5 generates recognition questions from captions, e.g., "What animal is shown in the image?"
   • Textual-centric Question Formulation: GPT-3.5 generates diverse questions related to the visual content, e.g., "What is typical behavior of the rabbit?" along with answer options grounded in the visual context.
   • Evaluation and Filtering: Questions are filtered based on LVLM performance, retaining only "easy" questions.

   Simple-Composed Question Generation

   • Image Substitution: Textual-centric questions are converted into Simple-Composed Questions by replacing text with images.
   • Filtering and Refinement: Difficult questions are identified through LVLM performance and manually refined for the final benchmark.

   We will include a visual pipeline diagram to make this process more transparent and easier to understand.

W2 & Limitations 1.2: Evaluation on the current mainstream mllms.

We include evaluations on several mainstream models, including open-source (Qwen2VL) and closed-source models (GPT-4.1, Gemini 2.0, Flash). As shown in the table below, these models demonstrate high accuracy on both visual- and textual-centric questions individually. However, when combined, they exhibit noticeable drop in performance (over 10%) compared to the individual task accuracies.

This performance drop actually supports our motivation and quality of benchmark. Current mainstream MLLMs still show room for improvement on our VLComp benchmark, highlighting the quality and challenge of our benchmark.

To further validate our benchmark, we also evaluated three human experts (with access to search engines) and report their average score. Their consistently high performance demonstrates that the benchmark is solvable by humans and confirms its clarity and diagnostic value.

W3 & Limitations 2.2: In the method part, the mllms used is a bit outdated

Thank you for the suggestion. We apply our method on mainstream Qwen2VL and evaluate them on VLComp. The results are as follows:

The results indicate that our method also achieves the improvement on the mainstream baseline, which exhibits fewer hallucinations. Due to time constraints, we use only a very limited amount of training data from QwenVL (previous version in Qwen Family).

Q1.1 & 2.1: Discussion with multimodal alignment problem

Thank you for the question. SCHall arise not only from insufficient multimodal alignment, but also from failures in compositionality. To further probe the role of modality alignment, we conduct an additional experiment using Qwen2VL. Specifically, we first prompt the model to generate a caption for the image alone, then combine this caption with the original textual question for inference. Interestingly, the model’s performance drop from 67.3% to 62.2%. Among these, 8.3% of cases show improvement, indicating that alignment is indeed a bottleneck. However, the majority of new errors fall into two categories: (1) 45.5% where the caption lack question-relevant visual details (reflecting visual abstraction failures, as discussed in Section 3.2), and (2) 54.5% where the caption is informative, but reasoning still failed—highlighting limits in compositional reasoning. These findings support our claim that multimodal alignment, while necessary, is not sufficient to resolve SCHall cases. Visual abstraction and other underlying factors remain critical challenges.

Our proposed method also contributes to improving alignment. Among the 8.3% alignment-limited cases, our method improves 22%, demonstrating its effectiveness in addressing this subset of failures.

Q1.2: From Figure 1, the effect of simple-text is lower than that of simple-visual. Is there a corresponding explanation?

Since most existing hallucination benchmarks focus on visual recognition, we intentionally craft visually simple questions to minimize the impact of recognition errors and better diagnose SCHall.

Q2.2 & Limitations 2.1: Motivations and effects of the training stages in the method part.

The motivations and effects of two stages is as below:

Training Stage 1: VLR pretraining. Motivation: As discussed in Section 3.2, a key contributor to SCHall is the failure of visual abstraction. To address this, we introduce explicit VLRs to encourage the model to abstract visual information effectively. Specifically, we pretrain the model by blocking access from the output tokens to the image, encouraging it to rely entirely on VLRs for extracting and transforming key information. Effect: This pretraining improves the model’s ability to extract and represent visual information accurately. However, it does not yet equip the model to jointly reason with images and VLRs, which is addressed in subsequent training stages.

Training Stage 2: Distillation learning. Motivation: As shown in Section 3.3, the presence of visual inputs can degrade the model’s language processing ability. To address this issue, we distill language reasoning knowledge from a branch that uses only image captions to guide the main branch, which jointly reasons over images and VLRs. Effect: This stage improves the model’s language understanding in multimodal settings and encourages more effective use of VLRs.

Thank you for the feedback. We will make the motivations and effects of each training stage clearer in future revisions of the paper.