Thank you for your thorough and insightful review of our paper. We are happy that you appreciate our novelty and found our analyses thorough. We addressed your questions and concerns below. If any residual concerns remain, we would be glad to discuss further. If no concerns remain, we would appreciate it if you could raise your score.

W1: Missing Experiments.

We have added a comparative table in our revised manuscript that clearly presents the performance of OPERA, RLHF (MiniCPM-V), and other relevant methods on our multi-object benchmark. Due to the limit of space, we kindly redirect the reviewer to the Rebuttal Supplement PDF.

• Decoding algorithm (OPERA): In the default multi-object setting, OPERA marginally improves the performance of LLaVA-1.5 in some settings, especially in settings with higher object homogeneity. OPERA decreases the performance on heterogeneous tests.
• RL-based finetuning (MiniCPM-V): MiniCPM-V shows robust performance across different settings, consistently handling various object types effectively. This approach enhances model performance in multi-object benchmarks that even surpasses single-object settings in Wild and Hom sets. Upon inspection, we found that this model demonstrates strong visual in-context learning capability and improves correct recognition when objects of the same classes are probed together.

In summary, the heterogeneous test split remains challenging given recent advances in decoding and alignment.

W2: Limited Significance on Real-world Use Case

Real-World Use Case 1: Common Scenarios in Embodied AI

Multi-object querying is particularly relevant in embodied AI applications, such as in cooking scenarios. In a typical kitchen setting, a robot might need to identify and manipulate multiple ingredients and tools simultaneously. For example, preparing a meal might require the robot to locate and retrieve a knife, cutting board, vegetables, and spices all at once. This ability to handle multiple objects at the same time enhances the robot's efficiency and effectiveness, reducing the time taken to complete tasks and improving overall performance.

We further present a case study in Autonomous Driving (see Figures 1 and 2 in Rebuttal Supplement PDF). This case study demonstrates how our approach can be utilized in the automotive industry to enhance the accuracy and efficiency of object recognition systems in vehicles. Figures 1 and 2 are taken from the nuScenes dataset for autonomous driving. Figure 1 illustrates the single-object case, where each object is identified independently. Figure 2 demonstrates the multi-object case, where multiple objects are detected simultaneously. The multi-object case exhibits more hallucination errors compared to the single-object case. This finding underscores the importance of studying multi-object hallucination, especially in real-world scenarios like autonomous driving, where multiple objects need to be detected accurately at the same time.

Real-World Use Case 2: Cost and Time Efficiency

Evaluating multiple objects simultaneously, rather than querying each object individually, can significantly save both time and resources.

Q1: Figure 3 Example

• For Q1-a, prompting with binary inference questions "Is there an orange in this image?" has been discussed in prior work, specifically in the POPE [24]. In our paper, we focus on the multi-object setting and our goal is to compare it with a single-object setting and ensure a fair comparison. Our single-object prompts are designed to align with our multi-object prompts for controlled studies, allowing us to directly evaluate and compare the performance under both settings.
• For Q1-b, the instruction type used for the example in Figure 3 is the default multi-object setting, the same as Figure 2.
• For Q1-c, we utilize student-forcing and teacher-forcing techniques to mitigate the model's dependence on the output format. These techniques allow the model to focus on predicting the next class rather than conforming to a specific output structure. Furthermore, student-forcing and teacher-forcing help us analyze whether the model learns shortcuts from the text.

Q2: Instruction Following

In fact, our student/teacher forcing probing strategies are designed "to separate out errors due to following instructions, we force the model to follow the format template and decode only the object tokens for each of the five objects. Ideally, this setting allows the model to focus solely on object recognition." By decoding the object tokens conditioned on a correct template, student/teacher forcing probing strategies allow us to evaluate the model's performance in both single-object and multi-object settings more fairly, as it factors out the template dependence. See lines 140-150 and Figure 7 in the Appendix for details.

We appreciate your input and the reference to the recent paper [A]. We acknowledge its relevance to our work and will discuss this paper in our revised manuscript.

Thank you for your thoughtful feedback. We are happy that you found our paper well-written, the problem novel and under-explored, and our experiments/analyses extensive and detailed. We addressed your questions and concerns below. If any residual concerns remain, we would be glad to discuss further. If no concerns remain, we would appreciate it if you could raise your score.

Q1. Initial ideas or hypotheses to mitigate multi-object hallucinations

We present an initial ideas on addressing multi-object hallucinations from a training perspective in Section 4.3. Our findings indicate that LVLMs tend to hallucinate less in homogeneous cases and more in heterogeneous cases. This suggests that LVLMs may get distracted by other objects and struggle to pay attention to the referred objects. Consequently, we propose a training-free method to mitigate this issue, see details below.

W1, Q2. Adding a baseline for multi-object hallucinations

One of the most natural way to make LVLMs "pay more attention to" what is inside the bounding box is to enlarge the amount of attention spent on the bounding box region. Since our dataset comes with ground truth bounding box regions and dimensions for each image, we retrieve the corresponding patches in the ViT that contains the bounding box region, and increase the self-attention on those tokens.

After some trials and errors, we found:

• Following ATMAN [1], we keep the selected tokens’ attention the same and scale all other tokens’ attention uniformly down by 0.7. Then after softmax, those bounding box regions will naturally obtain more attention.
• We have to freeze the attention ratio between vision and text, and manipulate the attention within visual attention. Otherwise, LVLMs output random meaningless tokens.

[1] AtMan: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation. Björn Deiseroth, Mayukh Deb, Samuel Weinbach, Manuel Brack, Patrick Schramowski, Kristian Kersting. NeurIPS, 2023.

W2. Latency Analysis

Thank you for your valuable feedback. We appreciate your suggestion to include a latency analysis in our evaluation. In response to your comments, we have conducted additional experiments to compare the latency of five single-object evaluations with that of a multi-object evaluation.

We greatly appreciate the reviewer’s time and effort reviewing this paper. We thank the reviewer for the positive feedback on the motivation, novelty, and “meticulously designed” experiments. Please see our responses to your questions below.

W1: Improvement in Writing

Specifically, we describe our improvements for presentation in the experiment sections, especially Section 5.

• Enhanced Definitions and Explanations in Model Behaviors: We will clarify the definitions and explanations for factors relevant to the mechanistic behaviors of the models. Specifically, we refined the descriptions and formulas for Object Token Entropy and Visual Modality Contribution to ensure they are more comprehensible.
• Refined Table and Figure Captions: We will refine the table and figure captions to provide more context and detail about the settings and the content of each figure. This ensures that readers can better understand the visual data presented and the specific conditions under which the experiments were conducted.
• Detailed Analysis in Section 5.2: We will rewrite and expand the section “When Do LVLMs Experience Multi-Object Hallucinations?” to provide a more detailed and clear analysis of each factor. Each factor is now thoroughly examined, with a deeper discussion of its impact on the model’s performance.

We kindly redirect the reviewer to the General Author Rebuttals for more details.

Q1: Legend

We apologize for the confusion caused by a mistake here. The legend for Figure 5 was misaligned with its content. The hallucinated objects are represented in Yellow and the non-hallucinated are Green. We will correct these errors in the revised manuscript and ensure that the descriptions accurately reflect the results.

Q2: Visual Prompt

We apologize for any confusion regarding the visual prompting strategies in Table 2 (Lines 194-198). For all LVLMs, we experiment with the bounding box as visual prompts. Specifically for CogVLM-G, we additionally experiment with their special grounding tokens as visual prompt input, as this is their natural visual prompt format and outperforms naive bounding box prompts. We report their performance using special grounding tokens in the Table.

Q3: Format

Thank you for catching this presentation error. We have now corrected this in the revised manuscript.

Response to Reviewer 4TNS

We greatly appreciate your dedicating your valuable time to this detailed review! We address your concerns as follows.

Weakness

Due to the limited space, we kindly redirect the reviewer to the General Author Rebuttals.

Q3: Seen and Unseen

The Seen and Unseen datasets are divided based on whether the images originate from the training or test/validation sets of the original dataset. We use MSCOCO Panoptic and ADE20K datasets to build our datasets. We didn’t use Visual Genome as it does not have an official train/val split and people use the GQA splits conventionally.

Q4: Single-Object Setup

Statement (1) is correct, the setup has “each image with 5 bounding boxes results in 5 independent sessions with the VLM.” The wild/hom/het dataset division is identical to the Multi-Object setups. The purpose is to keep all irrelevant factors untouched thus the comparisons between setups are fair and controlled.

Why is the single-object recognition accuracy so low, especially for LLaVA-34B…For LLaVA-34B, performance for single and multiple objects is similar…

We note that LLaVA-34B performance for single and multiple objects is similar ONLY for the situation of Seen+Heterogeneous data split. For all the other setups, performance for single is better than multiple objects. We are also aware that the performance of LLaVA-34B on Seen+Heterogeneous+Single Object is abnormal. We reran the experiments with 16 bit precision and remove model quantization, the updated number is 19.36%, which is almost the same as the previous score. Upon closer inspection, LLaVA-34B appears to memorize images encountered during pretraining and is prone to focusing on salient objects or objects with multiple occurrences.

How does having 5 bounding boxes in a single-object setup impact results? There is a very marginal performance increase when keeping only one bounding box compared to five. We report the results with 5 bounding boxes to keep all irrelevant factors untouched thus the comparisons between setups are fair and controlled.

Q5: Shortcut Hypothesis

For hypothesis 1, the model might learn to perform class-agnostic object recognition better due to the ability to perceive context through a few-shot setting. For hypothesis 2, the model might learn to recognize one specific object in a few-shot setting. We expect improved performance on the last object B in the AAAAB setup for hypothesis 1, compared to the single object setting. For hypothesis 2 we expect the performance to be roughly the same.

Q6: Grounded Tuning

We refer specifically to the results for CogVLM-G presented in Table 2. The model performs poorly in the default multi-class setting, but shows relatively better performance under the single-object setting and teacher-forcing setting. We hypothesize that the grounding model fails to improve performance due to either a lack of instruction-following ability. This suggests that grounding alone does not effectively reduce multi-object hallucination in our benchmark, and conversational tuning is critical. We will add more detailed experimental results to support this conclusion in the revised manuscript.

Q7: Figure Caption Clarity

When calculating hallucinated and non-hallucinated objects, we consider every object from all the unseen images, and classify the model’s prediction into hallucinated and non-hallucinated. We will improve the caption in the revised manuscript.

Q8: Figure 5 Error

Thank you for catching this bug in the plots, we realize that the legend for Figure 5 was misaligned with its content, which led to confusion. The hallucinated objects are represented in Yellow and the non-hallucinated are Green. We will correct these errors in the revised manuscript and ensure that the descriptions accurately reflect the results.

Q9: Last Token

This is a confusion arising from the way we described the algorithm. In our implementation, which uses a causal transformer, the model generates tokens sequentially. At each step, while generating the next token (which is the most recent token in the context of the transformer), the model attends to all previously generated tokens. This means the model’s attention mechanism considers the entire history of tokens up to that point. We apologize for any misunderstanding caused by our writing and will clarify this in the revised manuscript.

Q10: Actual Class and Predicted Class

Sorry for the confusion. For a model that hallucinates object A into object B, object A is the actual class, and object B is the predicted class. Indeed, presenting them as two distributions loses this one-to-one pairing information, but Zhou et al. [67] have already studied this setup and we emphasize on the statistical comparison instead.

Q11: OCR

Yes, we agree that our benchmark setting may entangle hallucinations with the model’s ability to correctly recognize visual text information within the bounding boxes. To address this, we conducted controlled experiments to isolate these factors. In the single-object setting, the model performed very similarly for each of the objects 1/2/3/4/5 within the same image, which indicates that its performance is stable with each visual prompt (See Appendix for the full tables). Also, since the same issue remains for both single and multi-object settings, our findings hold as the factors are controlled. Besides, our visual prompt setups mostly follow that of the set-of-mark prompting [54].

Q12: User Studies

Thank you for the valuable suggestion. Incorporating a user study where users provide bounding boxes and answer questions would indeed offer a meaningful evaluation of the model’s performance in real-world scenarios with potentially unclear boundaries. Due to time and resource constraints during the rebuttal process, we were unable to implement this approach. However, we consider it an excellent direction for future work and plan to explore it in subsequent research.