Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models

Q1 Open Data. Please refer to the answer for W3. Lack of open-sourcing of the benchmark data.

Q2 Comparison to other benchmarks. As highlighted by reviewer 4cSE, the rigorous analysis in MERLIM discovers ‘Hidden Hallucinations’, a phenomenon where seemingly correct answers lack any visual grounding and therefore constitute an error. To the best of our knowledge, MERLIM is the only tool that can identify and quantify such a source of error. As stated by dTrJ, MERLIM works with the open-set natural language generated by the language decoder. This allows for a more realistic assessment of the capabilities of the IT-LVLMs. As outlined by 1AUQ, MERLIM focuses on fundamental visual tasks, which are overlooked by other benchmarks. Critically, we identify that despite the large-scale pre-training and all the in-domain returning IT-LVLMs, they still underperform in these tasks. We hypothesize that more complex tasks, such as captioning and VQA, could be affected by these fundamental errors in identifying and locating objects in an image.
Unlike others, MERLIM constitutes a large-scale Benchmark with 2 orders of magnitude more data than POPE, HallusionBench, and NExT-QA. Moreover, MERLIM’s clever way of editing visual data allows it to disentangle some of the most common regularities in visual data. We conclude that given its scale, evaluation strategy, and visual manipulation strategy, we see MERLIM as a more real-world evaluation benchmark than the currently available benchmarks.

We thank reviewer YPpW for their comments and questions. Below, we provide detailed clarifications and responses to each of the questions.

W1 MS-COCO Bias. Similar to MERLIM, POPE and NExT-GQA are based on a single dataset (MS-COCO and NExT-QA), HallusionBench contains only 346 images, many of them are 2D plots and visual illusions, with very few natural images. In comparison, MERLIM uses the full validation set of MS-COCO (POPE uses only 500 images) and generates new data (changes are subtle, but the images are never semantically identical). Due to these changes, MERLIM can break some standard visual biases (for example, in Figure 1, there are no visible tennis racquets on a tennis court). Moreover, we continue with such a visual edition procedure and evaluate its effects in 3 different tasks. Therefore, we politely disagree with YPpW and state that we consider MERLIM as a direct tool to break some of the standard visual biases and analyze the effects of this visual disentanglement in multi-modal tasks. Finally, we invite YPpW to reconsider its requirement for a fully unbiased dataset. Computer vision datasets (and, by extension, the models trained on them) are highly biased despite modern techniques that collect large-scale data from diverse sources. Current (and classic) datasets suffer from strong bias to the point that tuned models can easily identify the source dataset by simply analyzing one image [A].

[A] Liu, Z., & He, K. (2024). A Decade's Battle on Dataset Bias: Are We There Yet?. arXiv preprint arXiv:2403.08632.

W1 Leakage. We welcome YPpW observation in leakage, as this is a common issue in evaluating LLMs. However, we politely disagree as MERLIM does not have a higher risk of leakage than any other benchmark. It is impossible to fully guarantee that MERLIM images will not be used as training data. We share YPpW enthusiasm about open data, thus we have already made available the entirety of MERLIM data (labels included). Even if we created a new test-set with no ground-truth data, those new images must be downloaded for inference, then they can be manually labeled and used for training. We further clarify that MERLIM uses only COCO VALIDATION data, and the 31K edited images correspond to altered versions of MS-COCO. A significant data leakage can only occur if someone directly trains on the validation data of COCO along with the derived data of MERLIM. In summary, MERLIM does not pose a greater (or smaller) risk of leakage than any other dataset new or old. leakage will only occur if an actor (by omission or malice) decides to train our data, which is the case for any other benchmark regardless of its recency.

W2 Laking of evaluations of newer MLLMs. The field of LVLMs is advancing rapidly. This makes it challenging to evaluate every newly developed IT-LVLM. Nevertheless, we assessed 11 state-of-the-art models to demonstrate the advantages of our benchmark, MERLIM, in highlighting the visual grounding limitations and language biases of IT-LVLMs. We plan to continuously update the leaderboard to include newly released state-of-the-art models, ensuring MERLIM remains a relevant and comprehensive evaluation framework.

W3. Lack of open-sourcing of the benchmark data. Similar to POPE, NExT-GQA, and HallusionBench, our data is open and free of charge to whoever wants to use it. Our license is the very same as MS-COCO “Creative Commons Attribution 4.0 License”. The original MS-COCO data can be downloaded from the official website.

We made the MERLIM’s inpainted data, queries and code available at submission time (anonymous repository). The link to the data (inpainted images and queries) can be found at the end of the introduction (line 101) on the word ‘here’. The format does not show as a standard hyperlink but can be clicked. We have corrected the style of this link in the revisited version.

For completeness, we have updated our anonymous repository to explicitly include the license of our inpainted data and related queries.

State-of-the-Art Models

Our benchmark evaluates 11 models, including the recent xGen-MM (BLIP-3), which was first submitted to arXiv on August 16, 2024. We recognize that this is a rapidly evolving field, making it challenging to evaluate every newly released model. Nevertheless, our benchmark provides a comprehensive analysis of state-of-the-art models, offering valuable insights that can inspire future research. Additionally, in our updated revision, we have incorporated evaluation results for GPT-4o-mini, now included in Figure 8 of the supplementary material.

We thank Reviewer YPpW for engaging in the discussion and for his/her comments. Regarding YPpW assertion that “The author's response does not address my concerns” In the original review, YPpW refers to weakness regarding the lack of open-source data and potential data leakage in MERLIM, highlighting the first one as “ a substantial drawback”. We note that after our rebuttal, these weaknesses are no longer discussed by YPpW. If concerns about the license of the data or its potential leakage remain, we invite YPpW to address them directly in this discussion. We would be more than happy to provide additional feedback and clarification. Otherwise, we politely invite YPpW to acknowledge that these concerns have been addressed and revisit his/her rating, given that the only drawback that was highlighted as “substantial" has been addressed.

Regarding the remaining concerns. We proceed item by item over YPpW ‘s last message.

1. "While many hallucination benchmarks based on COCO have been accepted, that is not a justification for accepting this paper. There are already too many hallucination benchmarks based on COCO.”

We politely disagree with YPpW and invite the reviewer to list other hallucination benchmarks that evaluate instructional models (which are distinct from visual-language models). Moreover, we invite YPpW to carefully analyze the performance of modern IT-LVLMs in such benchmarks, POPE and MERLIM are far from being saturated. Certainly MS-COCO is a classic computer vision dataset, but the recent empirical evidence indicates that it remains a challenging test-bed for current IT-LVLMs.

2. “Rigidly adhering COCO in the multimodal field”.

Our benchmark does not represent the standard practice of object detection/recognition in MS-COCO. Instead, MERLIM provides a more real scenario than the existing hallucination benchmarks. We evaluate the hallucination on open-ended questions (not part of the MS-COCO dataset) and provide a set of questions to analyze the language bias (no language biases are assessed in the MS-COCO dataset). Finally, the LLM of the instructional model can output open answers, we also consider if the predicted nouns are synonyms of the ground truth objects, this is not part of the MS-COCO dataset or its evaluation procedure.

3. “If an MLLM was trained using the COCO train set, it might achieve better scores on the benchmark proposed in this paper”.

We invite the reviewer to YPpW to revisit figures 2 and 3. xGen-MM (BLIP-3) does not rely on MS-COCO during training but outperforms models such as BLIP2, InstructBLIP, and LLaVA-1.5, which are actually trained on MS-COCO data. This empirical evidence is in direct contraposition to YPpW assertion about dataset biases. We remind YPpW that MS-COCO is just one of several datasets used in the training of IT-LVLMs and that the training typically comprises a visual-language alignment step followed by an instructional tuning step. None of these steps directly includes the task of object recognition, and these training steps only use MS-COCO training data. Our empirical results suggest that the difference in the training objectives of IT-LVLMs and MERLIM’s primary tasks does not generate a clear overfit or constitute any evident data leakage in the MS-COCO dataset.

4. “However, this does not reflect the true hallucination level of the model, i.e., its real-world hallucination performance, but rather its hallucination performance within the COCO domain.”

We politely invite YPpW to clarify what he/she calls “real-world hallucination performance”, specifically, how can the hallucination rate of a model be estimated independently of the benchmark data?. Moreover, we also invite YPpW to elaborate on what kind of dataset can effectively evaluate real-world hallucinations regardless of the domain of the data. We will proceed to directly compare the most relevant design choices of MERLIM with the outlined requirements of YPpW. We must note that collecting new data (as suggested by YPpW) would also receive the YPpW complaint. YPpW argues that COCO data can only assess hallucinations in the COCO domain. Likewise, the newly collected dataset will only assess hallucinations in its collection domain.

Finally, we would like to highlight that MS-COCO was “collected images from Flickr, which tends to have fewer iconic images. Flickr contains photos uploaded by amateur photographers with searchable metadata and keywords”(Lin et al., 2014),. Thereby reducing potential biases in collection and labeling. Furthermore, we consider the different open-ended questions would mimic a closer to the real-world working scenario.

We thank reviewer 4cSE for the insightful review. We will take care of introducing hidden hallucinations right from the abstract. As highlighted by 4cSE, this is a central contribution of our benchmark and an under-explored topic in the research community.

W2, Q1. Focus of the article. Indeed, the "hidden hallucinations" are a direct contribution of this paper, but our contribution is not limited to them. Many hallucination benchmarks follow the style of VQA by querying the IT–LVLM with multiple-choice answers. These benchmarks do not control for alternative questions with similar semantics or allow for open-set responses. Since MERLIM design includes these two aspects, we see our benchmark as a more realistic scenario for IT–LVLMs. These models are commonly used as multi-modal assistants that reply with natural language.

In addition, MERLIM is the first benchmark to employ subtle visual edits, thus establishing a connection between effective visual grounding and IT–LVLM performance. We think MERLIM can bring valuable insights.

Finally, as outlined by 4cSE, we also focus on fundamental vision tasks. As most benchmarks follow the VQA approach, the community has overlooked these tasks. MERLIM shows that, despite building on large visual encoders pre-trained with large-scale data, current IT–LVLMs underperform in fundamental vision tasks. In fact, MERLIM shows that none of them can reach a 50% F1 score in object recognition.

Q2. Difference between IT-LVLM and LVLM? The key difference between these modes is the instructional training and the capacity of the language module. CLIP would be an LVML, it lacks any instructional training and has a relatively smaller language module. Therefore, it can only align multi-modal data. Meanwhile, IT–LVLMs like LLaVA, BLIP-2, and Kosmos-2 build upon a stronger language component and can output language with their decoder component. After instructional training, the IT–LVLMs can follow natural language instructions and produce open-set responses according to the language instruction and the image data.

Q2. Why use only one prompt in 3.3? As described in Section 3.3 (lines 301-304), we evaluate two prompts. First, we ask the model, 'How many {object category} are there? Just answer the number.' This instructs the model to provide a numerical response, which we cast into an integer using the int() function. Next, we assess the model's congruency with a follow-up prompt, we ask, 'Are there {model answer} {object category}?' or 'Is there one {object category}?' This approach allows us to systematically evaluate the counting ability of the model and if the answer is effectively visually grounded or if it depends on a language bias. Clearly, there could be more equivalent language prompts, but MERLIM already captures some language diversity to validate the numerical answers.

Q3. Why models are, on average, worse on edited images. This phenomenon is particularly intriguing. The performance gap observed between the original and edited images highlights the spurious visual grounding that is prevalent in most IT-LVLMs. The text output of the IT-LVLMs does not completely correlate with the visual changes in the image, and changes significantly according to the language input. This indicates that IT-LVLMs exhibit limited robustness to modifications in both visual and language inputs. In our paper, MERLIM serves as a diagnostic tool to identify and study this lack of visual grounding and robustness to the input.

We thank dTrJ for the careful review and favorable view of our work. We are working on the typos and style errors and will notify dTrJ when we update the PDF.

W1. Examples of tasks. We have added some example output for each task evaluated by MERLIM in Figure 9 in the revisited PDF supplementary material section (A.6).

W2 Examples of the edited images. As outlined in the main paper (Line L723), we focus on edited images with a similarity score exceeding 0.7. To further analyze this criterion, we sampled additional images meeting this threshold (Refer to Figure 5). Images with high similarity scores (approximately 0.9) are virtually indistinguishable from the originals. Even when large objects are removed, the edits are in line with the image context and are hard to spot. Likewise, the images with the lower similarity score are still highly similar. While some inpainted images may contain localized artifacts, these artifacts are subtle, do not significantly alter the appearance of other objects, and these artifacts provide minimal clues for identifying the removed object.

W2 Inpainting. We use the image inpainting method of Li et al. (2022b) (Lines 200-201). Currently, no inpainting method is perfect for completely replacing pixel data with imperceptible and in-context modification. Therefore, we apply the quality control described in Appendix A1.

W2 Edges of the racquet. Some minor details of the inpainted object can remain. We identify that some artifacts can also remain at the edge as the polygons from MS-COCO might miss (or over-segment) some pixels at the very edge of the object. Therefore, we perform the YOLO test, described in Appendix A1, to verify the object is no longer recognized in the image.

W3 open set responses for relation and counting tasks. We believe our editing and parsing strategy offers significant potential for extension. In future work, we plan to broaden our open-ended question evaluation to account for not only nouns but also adjectives, such as those describing color, position, and other attributes, thereby also evaluating relations between nouns (Relationship task).

W5,W6,W7. Typos and style. We have addressed them in the new version of the PDF.

W8 Number of images. MERLIM stars with 4.1k original images (MS-COCO val), and from those images we obtain 31K inpainted images (total 35K). Since we have 3 tasks and multiple questions for each individual instruction, we have a total of 300k image-question pairs across 3 tasks. In comparison, hallucination benchmarks like POPE, MMVP, and HallusionBench have between 300 and 500 images (two orders of magnitude less). And contain about 1K to 2K image question pairs (again, two orders of magnitude less). When compared with other hallucination benchmarks, MERLIM is far larger.

Q1. Details on the output processing. For yes/no questions, we preprocess the answer by converting it to lowercase. If the answer contains the word 'no', it is classified as 'No'. Likewise, if the answer contains 'yes', it is classified as 'Yes.' For counting questions, as we outline in L301-302, we prompt the model with the instruction, 'Just answer a number,' allowing us to cast its output using the int() function and evaluate the difference between the prediction and the ground truth. Only answers that can be successfully casted by the cast function' int()' are evaluated. The models produce, on average, 32683.28 valid answers, corresponding to 91.9% of the total questions, considering both edited and original images. We will update the final version to reflect all the details in this process.

Thanks for your detailed response. In general I believe the work is meaningful research in the right direction.

However my biggest concern was whether the inpainting actually works well enough. To verify this, I scrolled through the dataset images and found that it can fail quite often. 102331_1 and 107094_0 show cases were humans were only half-erased and still visible. In 100723_3 a person is erased but replaced with meaningless noise. So at least a part of the hallucinations is potentially cases where the object is still visible. These are just two examples but it happens more often, it seems the inpainting AI is not strong enough to reliable produce data of real-world quality.

Therefore, I decided to keep my rating at 6.

We thank reviewer Qhva for the careful reading and overall positive opinion of our submission.

W1. Overlap with other benchmarks. As outlined in Lines L139–146, MERLIM overlaps with other benchmarks, such as POPE and MMT-Bench, in tasks like Object Recognition and Counting. However, MERLIM focuses on analyzing hidden hallucinations and tackling more challenging scenarios involving open-ended questions. Unlike traditional benchmarks that assess object recognition through yes/no questions, MERLIM automatically extracts predicted objects from the model's output and compares them to ground truth, enabling a more realistic evaluation. Additionally, MERLIM evaluates the model's performance on both original and edited images, revealing whether its answers are visually grounded. In the inter-object relationship and counting task, we also include the performance for original and edited images and show a decrease in performance in every evaluation. This demonstrates that the hidden hallucination problem extends beyond object recognition to more complex tasks. This highlights a core contribution of our work: hidden hallucinations are a general issue for IT-LVLMs, affecting not just fundamental tasks but also their broader capabilities.

W2. The field of LVLMs is constantly developing. Many models evaluated in the paper may not be that strong nowadays. We agree that the field of LVLMs is advancing rapidly. This makes it challenging to evaluate every newly developed IT-LVLM. Nevertheless, we assessed 11 state-of-the-art models to demonstrate the advantages of our benchmark, MERLIM, in highlighting the visual grounding limitations and language biases of IT-LVLMs. We plan to continuously update the leaderboard to include newly released state-of-the-art models, ensuring MERLIM remains a relevant and comprehensive evaluation framework.

Q2. MERLIM Expansion. Yes, we see MERLIM as the first step on a larger IT-LVLM Benchmark. Our core strategy (editing the visual data) allows for direct control of the ground truth, thus allowing us to explore visual arrangements that do not match the regularities of language. We think further visual properties and more complex relationships can be explored, especially with regard to adjectives.

We thank 1AUQ for the careful and detailed review of our work, spotting typos and style errors. They have been corrected on the updated PDF.

W1 MS-COCO data. We chose MS-COCO as its overlap with Visual Genome allows us to establish the object relationship task. Additional datasets can be used as input for MERLIM. As long as segmentation data is available, our inpainting method and query strategy can be directly applied, thus generating evaluation data. No dataset is completely unbiased thus extending MERLIM to multiple datasets with instance segmentation will still retain some sort of bias from the original dataset. We have already released MERLIM, and we plan on extending it once the performance gap between inpainted and original images begins to close.

W2 Key motivations in the submission. Our key motivation is to create a large-scale benchmark that provides a more realistic evaluation scenario for IT-LVLMs. To this end, we create a set of open-ended questions and develop a straightforward way to manipulate the expected outcome of those questions (image inpainting). The result is a flexible and extensible benchmark that shows that IT-LVLMs still have relevant fail-case scenarios in fundamental computer vision tasks. Our flexible analysis also results in the discovery of hidden hallucinations, which are answers that are not effectively visually grounded. A visual grounding error that has not been studied yet but represents a percentage of the performance of IT-LVLMs. We think MERLIM can serve as a tool to advance the area of IT-LVLM by improving the performance in fundamental tasks and better understanding how their performance is affected by text replies that have no effective visual grounding.

W3 Inpainting Details. Our inpainting procedure is straightforward, we use COCO masks to define the area to be inpainted. To minimize changes, we remove exactly 1 object from the image. Therefore, we can create multiple inpainted versions of the image (as many as annotated objects are). We replace only the pixels in the COCO mask using the pixel data generated by the inpainting method proposed by Li et al. (2022b). No further modifications are made. After inpainting, we apply the quality control described in Appendix A1.

W4 More inpainted images. As outlined in the main paper (Line L725), we focus on edited images with a similarity score exceeding 0.7. To further analyze this criterion, we sampled additional images meeting this threshold. Our findings reveal that images with high similarity scores (approximately 0.9) are almost identical to the originals. Even when large objects are removed, the edits remain imperceptible in such cases. Likewise, those images with the lower similarity scores are still highly similar. While some inpainted images may contain minor, localized artifacts (around the edits), these artifacts are subtle. Our quality control process ensures that these artifacts do not alter the appearance of other objects and provide no clues for identifying the removed object.

Q1. We have corrected all of them and can be found in the revisited PDF.

Q2. Qwen-VL performance. We use Qwen-VL-Chat. Qwen-VL is one of the best-performing IT-VLMs in other benchmarks and is also the best-performing in the Object relationship task (Table 2 curated set). However, the performance in the Object Recognition task is lowered by object hallucinations. Figure 6a) (supplementary material) shows that the precision of Qwen is among the 3 lowest. Figure 6b) shows that its recall is not too far behind the other models, but questions 1,2, and 5 are particularly bad cases. Overall, we observe Qwen to be very sensitive to the syntax of the question and prone to hallucinations. This combination lowers the F1 score in Figures 2 and 3.