VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning

Thank you for your thoughtful review of our paper, VOILA. We are pleased that you recognized our work as novel, well-defined, and comprehensive with various properties, rule configurations, and difficulty levels designed to reveal the shortcomings of state-of-the-art MLLMs through detailed and comprehensive evaluations. We are delighted that you recognize the significance of VOILA in making advancements to abstract relational reasoning in MLLMs. Below, we provide detailed responses to your specific inquiries:

“Why the proposed problem is an essential and meaningful problem to solve. In what applications or scenarios is the capability essential?”

Analogical reasoning, commonly used in IQ tests, is a critical component for assessing cognitive abilities, encompassing problem-solving, decision-making, classification, and learning [7]. It involves the process of mapping abstract relationships between concepts and applying these relational patterns, which is fundamental for higher-level cognitive assessments. This reasoning is crucial to improve the generalization ability in which the model implements the identified pattern to a new set of data. Transferring knowledge from identified relationships allows models to make human-like decisions with complex abstract reasoning abilities across various contexts.

Analogical reasoning is essential when transferring the previous relational learnings from one concept to another. It can be applied in many application areas from medical diagnostics to creative industries. One example is autonomous vehicles that can use analogical reasoning for decision-making during navigation by recognizing the relationship between signs, intersections, or traffic flow patterns.

We will add more information to the paper to explain the significance of analogical reasoning.

“The comparison between the model and human performance may not be fair: (1) Humans are given two examples (2) It seems the task for humans is to choose properties from available options, while MLLMs need to predict those properties.”

We believe that the comparison between human and MLLM performance is fair. Human performances are conducted by providing two examples to explain the Voila-ND and Voila-WD dataset discrimination. However, no direct instructions were provided to human performers as they were given to the models with prompts. Instead of image examples, we implemented text-based instructions for models by following the L2M approach. Since our ablation study 6.2 shows that MLLMs are better at processing textual inputs than visual data, the experimental setup benefits them to understand the task better than visual examples.

Providing options may appear to influence human performance for the first step, however, to detect the distraction property relations (any), participants need more than just selection; they must perform a logical reasoning process to understand and apply the relevant and irrelevant relations provided in the instruction to the models. To facilitate the assessment of the human participants’ answers, we provided 14 actions, 15 subjects, and 5 numbers including the ‘any’ option. This configuration results in an accuracy of approximately 0.095% for random selection which is significantly lower than the human performance of 70%. As participants complete the visual analogy questions in a single step—selecting the best answer from the available options without intermediate stages—the gap between random selection and human performance indicates that the provided options do not offer a meaningful advantage.

Table 3, including the detailed model performance on VOILA-WD, shows that most of the models, excluding CogVLM2 and LLaVa, correctly recognize three properties with above 80% accuracy at the first step. This demonstrates that models are capable of describing the images by predicting the properties. Providing options might increase the accuracy of the first step, however, the performance of the models drops significantly after identifying relations and application relationships step, as seen in Table 3 with comprehensive property analysis at each step. This confirms that establishing logical relations, discovering irrelevant properties, and applying relationships require more than simply recognizing the properties; it demands high-level abstract reasoning capabilities. Another study in section 6.1 supports these findings. Although models are provided with ground truths at each step, their performance in the third step only achieves 17% accuracy. As offering examples and options does not compromise the core logic of the task, we believe our human vs model comparison is fair.

[7] Goswami, U. (1992). Analogical Reasoning in Children. Psychology Press. https://doi.org/10.4324/9781315804729

Based on the above answers, it is still not clear why the evaluation on the image generation step is helpful in evaluating the MLLMs, given you already have the score for "Applying Relationship."(what ability does the image generation score can reflect and the score of "Applying Relationship." can not?) Because in my understanding, the generated image is based on the output of the "Applying Relationship" step (Please correct if I was wrong). Besides, using the external image generation model DALL-E 3 and using GPT4o for evaluation both introduce noise from the final score of image generation to the MLLM's ability -- DALL-E 3 can make errors in image generation given the correct text input, GPT4o can also make mistakes during evaluation.

Thank you for considering our previous responses and for raising your score. We would like to address the remaining concerns you have. Below, we provide additional clarifications on the reasoning ability of the image generation step and evaluation performance of GPT-4o:

Why the evaluation on the image generation step is helpful in evaluating the MLLMs, given you already have the score for "Applying Relationship." DALL-E 3 can make errors in image generation given the correct text input.

The image generation errors made by DALL-E 3, EMU-2, and SEED-LLAMA, despite correct text inputs, highlight their limitations in reasoning and synthesis processes. Although these models can generate high-quality images, they struggle to render abstract and complex relationships because they rely on text-based embeddings rather than contextual understanding.

In the visual analogy task, models, capable of generating visual outputs, are required to understand semantically complex relational contexts between properties and visualize them accurately. When utilizing the “any” keyword for irrelevant relations, the models are expected to map abstract concepts and their text-based relations onto visual representation. The challenges models face when integrating multiple concepts and relations in the visual domain reveal their limitations in generalization.

VOILA evaluates the complicated relational logic in both text-based and vision-based forms. The ability to translate textual relationships into visual content measures the models’ performance in connecting semantic interpretation with visual reasoning. Evaluating the image generation step provides insights into the models’ shortcomings in understanding the context and effectively generalizing within the visual domain.

What ability does the image generation score can reflect and the score of "Applying Relationship." can not?

Although applying relationships is the most critical and challenging step in VOILA, the task evaluates diverse abilities at each stage. In the first step, the models are evaluated for perceptual and semantic context reasoning via image description. The second step focuses on measuring the models’ relational and comparative reasoning by identifying changed and unchanged properties between images. The third step tests the transfer learning and generalization capability of the models with relational mapping. The last step, image generation, assesses the models’ contextual understanding and visual representation mapping with multimodal alignment.

Although models accept the output of the "Applying Relationship" step as the input prompt in the last step, the abilities measured during the image generation process are distinct from the earlier step. While in the third step, models transfer relations to a new set of data, in the image generation step, models convert textual information to a visual representation by working across multiple modalities.

GPT4o can also make mistakes during evaluation.

We have already conducted a comprehensive ablation study to evaluate the accuracy of GPT-4o in each step, including image generation. This study shows that GPT-4o has a 10% error rate when evaluating generated images against the ground truths. For a more detailed explanation and analysis of GPT-4o’s performance, along with calculations, please refer to the response for Reviewer YkdC (part 2). This study ensures that GPT-4o’s evaluation capabilities are sufficiently robust for the comprehensive assessment process, despite its’ challenges of applying identified relationships.

We believe our response addresses your remaining concerns comprehensively and encourage you to reevaluate our submission. We look forward to further discussion.

We are encouraged by your review and appreciate your comprehensive evaluation of our paper. We are pleased that you recognize VOILA as a well-motivated, unique, well-formulated, well-defined benchmark, and thoroughly supported by detailed analyses based on manually cleaned data. We are especially pleased that you acknowledge the VOILA’s contribution to a dataset creation pipeline. Please find our detailed responses to your specific inquiries below:

“Most evaluations focus on the first three steps of the analogy completion task with only a few data points on the image generation stage.”

Because of the limited number of MLLMs capable of generating and processing multiple/collaged images and understanding the instructions in literature, we obtain more experimental results on text-based answers than image-based answers. We tested VOILA on 5 image generation MLLMs: GPT-4o, SEED-LLaMa, EMU-2, Chameleon, and Gemini. However, in most cases, Chameleon did not generate the output: “I'm unable to meet that request.” and for the rest of the several cases, it did not follow the instructions. On the other hand, Gemini accepts one image input but rejects working on human images. These restrictions might limit our comprehensive evaluation for the image generation step, however, results on the most preferred models in the field; GPT-4o, SEED-LLaMa, and EMU-2, are sufficient to have an idea of how successful state-of-art models to visualize the provided text data.

“The writing can be improved to highlight the most insightful and interesting takeaways”

We will review and make changes to the final draft to highlight the most important outcomes of the experiments.

“Why should the community use this benchmark to evaluate models’ relation understanding instead of existing VQA benchmarks that can also test for relation understanding?”

Thank you for your excellent question which reveals the significance of the VOILA. The visual analogy task, VOILA, consists of four distinct but related stages, and identifying the relation is one of them. The critical part of the study is how models apply the defined relationship to new data which differs VOILA from existing VQA benchmarks. The ablation study on 6.1 highlights the pivotal role of the relationship application process, as the model provided with ground truth relationship values still achieves only 17% performance. Additionally, VOILA is more systematic, better formulated, and better structured than general VQA benchmarks. Thus, it provides additional critical insights (as we discussed in the draft) about MLLM models.

The process of mapping abstract relationships between concepts and applying these relational patterns is a key element in IQ tests and cognitive assessments, where it helps evaluate high-level reasoning and problem-solving. To assess the cognitive intelligence of the model we need more than VQA benchmarks that focus on image description or identifying relations. VOILA provides the required logic to evaluate the models’ inference capacity in high-order relationships and the knowledge transfer ability of these relationships.

What differentiates VOILA from other analogy benchmarks is implementing distraction rules which expect models not only to understand the relationship and pattern between images but also to discover the irrelevant changes among properties. While human performers successfully detect the pattern-breaking features in the visual analogy questions, our experiment in 5.4 shows how current models struggle with revealing unrelated properties.

“Did the authors include examples of each subtask in the prompts?”

No, the examples are not given in the prompt. Instead, we use the L2M method and give instructions to generate the answer for each step.

“Table 3 is missing Emu-2 image generation values”

Thank you for your detailed review. We will add the accuracy of the image generation step of EMU-2 to Table 3. Our results on both VOILA-WD and VOILA-ND show that EMU-2 lacks the ability to accurately draw the number of subjects and especially actions in the image. The image generation performance of EMU-2 on both VOILA-WD and VOILA-ND is provided in the Table below.

We thank the reviewer for the insightful comments on our paper. We are pleased to see that our work, VOILA, is recognized as interesting, with the benchmark collection method detailedly explained.

Please find our detailed responses to your specific inquiries below:

“One line of related works I think this paper need to mention is on the benchmarking of multiple image understanding, such as works of Muirbench and MIRB, especially MIRB, which has already included visual analogy as one subtask.”

Although the MIRB [4] benchmark includes visual analogy as one of the sub-tasks, the authors acknowledged that their visual analogy task is taken from the VASR [5] dataset which we already discussed in our Related Work section.

The commonality of MUIRBENCH [5] and VOILA is to investigate the relations on multi-image inputs. However, MUIRBENCH focuses on 12 diverse multi-image understanding categories like a diagram, geographic, scene, cartoon, etc., and 10 relation types like narrative, temporal, independent, cropped/zoomed, etc. On the other hand, VOILA concentrates on arithmetic and pattern understanding using numbers, subjects, and actions and rule-based relation identification and application on multiple images. The comprehensive scope of MUIRBENCH does not cover the visual analogy task we implemented in VOILA, but we will discuss the commonality of the multiple-image understanding and visual analogical reasoning in the final draft.

“How reliable is the diffusion model used in the data generation pipeline, and how many images are discarded?”

As described in section 3.1 Data Cleaning, 30 images per prompt were generated by implementing the Stable Diffusion XL model. To verify the quality of the images and their alignments with provided text prompts, they were manually filtered and 10 images for each prompt and a total of 7280 diverse images were used to create visual analogy questions. Since all the images were cleaned and validated by human participants during the dataset creation process, the images in analogy questions are reliable and do not require additional human quality checks.

“Why is human performance only tested on the "Applying relationship" step?”

After obtaining unsatisfactory results from MLLMs applying a direct answering (one-step) approach, we adopted L2M methods with multiple steps to reveal the shortcomings sub-tasks of the models. However, the human performance was conducted where participants were tasked with predicting the properties of the missing fourth image with one step, which resulted in satisfactory outcomes. We did not see the need for additional experiment to evaluate their performance on previous steps. Since they do not perform intermediate stages and directly answer the analogy questions with text format, their performance is displayed in the Applying Relationship step in Table 3.

“Is using properties like number, subject, and action enough to cover all possible visual combinations that can form analogies?”

VOILA was designed as a foundational framework, the limited count of properties was provided for the initial evaluation process. The utilized properties may not cover all possible visual combinations; however, their diversity is adequate to assess the state-of-the-art MLLMs' reasoning capability, as detailed experiment results show. Extending the variation of properties would likely create a more comprehensive benchmark, however, it would increase the complexity of the task which current MLLMs already suffer. We preferred at this stage not to overwhelm the models with the rising number of properties, since the current visual analogy configuration already challenges them, but leaving it as future scope after models evolve.

We believe our response addresses your concerns comprehensively and encourage you to reevaluate our submission. We look forward to further discussion.

[4] Zhao, B., Zong, Y., Zhang, L., & Hospedales, T. (2024). Benchmarking multi-image understanding in vision and language models: Perception, knowledge, reasoning, and multi-hop reasoning. arXiv. https://arxiv.org/abs/2406.12742

[5] Bitton, Yonatan et al. “VASR: Visual Analogies of Situation Recognition.” AAAI Conference on Artificial Intelligence (2022).

[6] Wang, F., Fu, X., Huang, J. Y., Li, Z., Liu, Q., Liu, X., Ma, M. D., Xu, N., Zhou, W., Zhang, K., Yan, T. L., Mo, W. J., Liu, H.-H., Lu, P., Li, C., Xiao, C., Chang, K.-W., Roth, D., Zhang, S., Poon, H., & Chen, M. (2024). MuirBench: A comprehensive benchmark for robust multi-image understanding. arXiv. https://arxiv.org/abs/2406.09411

“Does GPT-4's difficulty in identifying relationships in VOILA affect the evaluation process, which also uses GPT-4?”

Thank you for raising this important question for discussion. Although the task of identifying relationships is different from comparing ground truths with model responses, the authors share the same wonder about how well GPT-4o performs in the evaluation task. To answer the question and measure the gap between human and GPT-4 evaluations, we conducted an error analysis, as requested. Since every model obtains distinct characteristic response styles, we evaluated 50 visual analogy question responses from diverse models with various rule configurations including the distraction rule. After processing the intermediate stages, a total of 180 responses were evaluated, with 30 corresponding to the image generation step. For a comprehensive evaluation, we utilized confusion tables for each step including all attributes. The terminology and tables are given below. The “Question” in tables represents the question answer merged with three properties (number + subject + action).

• True Positive (TP): The number of cases where both the human and the GPT-4o agree that the response is correct.
• False Negative (FN): The number of cases where the human expresses the answer is correct, but the GPT-4o says it is incorrect.
• False Positive (FP): The number of cases where the human says the answer is incorrect, but the GPT-4o states it is correct.
• True Negative (TN): The number of cases where both the human and the GPT-4o agree that the response is incorrect.
• Accuracy (Agreement): The number of cases where both the human and the GPT-4o agree.
  • Accuracy (Agreement rate) = (TP + TN) / (TP + TN + FP + FN)

First, we calculated false negative and false positive cases between human and GPT-4o evaluations for each step, attribute, and question answer. Then we calculated the accuracy also called the agreement rate. The results show that out of 50 evaluated questions and their intermediate steps, the agreement rate between human evaluation and GPT-4o was 91% for describing images, 94% for identifying relations, 92% for applying relationships, and 91% for image generation. Although the agreement rate for attributes is the lowest at 74%, the accuracy of question answers, which involve merging properties, exceeds 91%. This demonstrates that GPT-4o has a maximum error rate of 10% per step, and its difficulty in identifying relationships does not affect the benchmark evaluation process. We will add this comprehensive error analysis to the final draft.

Step 1 - Describing Images

Step 2 - Identifying Relations

Step 3 - Applying Relationships

Step 4 - Generating Images

We believe our response and new experiments address your concerns comprehensively and encourage you to reevaluate our submission. We look forward to further discussion.

We thank the reviewer for the insightful suggestions on our paper. We are pleased that you found VOILA interesting, well-curated, and designed with manual filtering to ensure quality. We are also pleased that you recognize that VOILA highlights the critical weaknesses in state-of-the-art MLLMs. Below, we provide additional clarifications and experiment results as requested:

“Novelty of the VOILA and comparison with MaRs-VQA benchmark”

We believe VOILA and the MaRs-VQA [1] benchmark are sufficiently distinct and thus VOILA has novelty. We agree that both studies work on abstract visual reasoning; however, MaRs-VQA are sourced from MaRs-IB [2] matrix reasoning questions, while VOILA has been generated from scratch with novel rule-based configurations with the Analogy Building Algorithm and the Stable Diffusion model. In addition to the dataset creation pipeline, the content, method, and structure of both studies are different. While MaRs-IB consists of 1440 static questions created with synthetic abstract figures on 3 by-3 matrix structures with 10 VQA questions, VOILA which dynamically generates up to 6.4M questions, focuses on real image-based analogical reasoning utilizing 7280 manually cleaned images on A : B :: A’ :: B’ framework with a multi-step process. Additionally, the application of the MaRs-IB is established on multiple answer choices which causes 25% success in a random selection. However, as stated in the Introduction part of the paper, "According to Bloom’s taxonomy of educational objectives, creation, rather than evaluation, requires the highest cognitive skills in the learning process [3].” By following this idea, VOILA implements generating a solution approach rather than selection to measure the high-level reasoning skills of the models.

“The evaluation of Voila is not grounded in existing psychological assessments for instance MaRs-VQA benchmark”

We respectfully disagree with the requirement of the psychological assessment in the VOILA benchmark. MaRs-VQA [1] is sourced from MaRs-IB matrix reasoning questions [2] which aim to investigate the developmental differences in reasoning capacity from adolescents to adults. Since MaRs-IB is conducted on a psychological baseline, the authors use neurodevelopmental and neuropsychological evaluations to validate the study. However, VOILA focuses on evaluating MLLMs’ high-level abstract visual reasoning capabilities and for this purpose, it utilizes analogical reasoning tasks which are widely used in IQ tests and cognitive assessments.

“What would happen if the task of identifying differences in subject, types, and actions in a single prompt was broken down into three separate sub-prompts?”

This is an interesting experiment that we did not investigate. As requested, we conducted a new experiment with GPT-4o on the VOILA-WD dataset to discover the impact of using subprompts on identifying property relationships. For a fair comparison, we froze the first-step answers and requested the model to find whether the number/subject/action changed or remained the same from the first image to the second image. After merging the results from three subtasks, we achieved a similar accuracy of 42% with a single prompt experiment. The accuracy of properties is also similar with 94% for numbers, 79% for subjects, and 56% for action. The relationship identification performance of GPT-4o with single vs three sub-prompts is provided in the Table below. The results demonstrate that GPT-4o's ability to identify relationships is not affected by the number of properties asked in the prompt. We will include the results of this ablation study in the final draft.

[1] Cao, X., Lai, B., Ye, W., Ma, Y., Heintz, J., Chen, J., Cao, J., & Rehg, J. M. (2024). What is the visual cognition gap between humans and multimodal LLMs?. arXiv. https://arxiv.org/abs/2406.10424

[2] Gabriele Chierchia, Delia Fuhrmann, Lisa J Knoll, Blanca Piera Pi-Sunyer, Ashok L Sakhardande, and Sarah-Jayne Blakemore. The matrix reasoning item bank (mars-ib): novel, 12open-access abstract reasoning items for adolescents and adults. Royal Society open science, 6(10):190232, 2019.

[3] B. S. Bloom, M. B. Engelhart, E. J. Furst, W. H. Hill, and D. R. Krathwohl. Taxonomy of educational objectives. The classification of educational goals. Handbook 1: Cognitive domain. Longmans Green, New York, 1956.

Dear Authors,

Thank You for the detailed responses and the new experiments. It is good to see that GPT-4's difficulty in identifying relationships does not affect VOILA. Thank You for the comparison with MaRs-VQA benchmark. I am raising my score to 6 in light of the clarified concerns and helping me understand the soundness of the paper. I am not raising my score further as the idea or implementation is not radically new but a good follow up work to an established line of works . Thank You.

We are encouraged by your positive feedback. We are pleased that you recognize VOILA reveals the limitations of state-of-the-art MLLMs with comprehensive ablation studies and detailed examinations. Below, we provide additional clarifications on the accuracy and scalability of the VOILA dynamic benchmark and resource usage of GPT-4o as requested:

“How to ensure the correctness of dynamic benchmark and how to make it scalable?”

The correctness of the dynamic VOILA dataset is ensured by a rigorous manual cleaning process of the 7280 images generated with the Stable Diffusion XL model based on pre-defined properties. Since all cleaned images include correct property tags, practitioners can build visual analogy questions running the Visual Analogy Generation algorithm without any annotation requirements.

VOILA offers the practitioners flexibility to work on various properties and rule configurations from the predefined set. The practitioner can modify the dataset size and select the desired properties and rule settings to generate visual analogy questions for different scenarios. The Visual Analogy Generation algorithm builds questions with a balanced distribution of image use, properties, and rules. Instead of a fixed dataset limit, the algorithm dynamically creates over 6.4 million analogy questions which makes VOILA highly scalable.

“GPT-4o makes the evaluation process resource-intensive and costly”

We respectfully disagree with the resource-intensive evaluation process. To optimize our evaluation pipeline, we compared the open-source Llama-3-70B-Instruct model with GPT-4o. While Llama-3-70B-Instruct requires more than 12 hours on 2 A100 GPUs to evaluate 10K results of a single model for each step, using Batch API of GPT-4o reduces the time to approximately 45 minutes for entire three steps with parallel processing without any GPU requirements.

The cost of GPT-4o API is calculated based on input and output tokens. The cost and the result time of the evaluation pipeline vary depending on the length of the models’ response. In our experiments, the API uses roughly 450, 300, and 230, 60 input tokens and precisely 74, 23, 24, and 30 output tokens sequentially at each step. Evaluating 10K questions costs $13.75* for input tokens and *$9.05 to generate output. In total, the evaluation cost of the entire task, including the image generation step for a single model is approximately $22.80. Suppose the practitioners do not prefer to follow the multi-step process. In that case, they can evaluate the text-based results in the third step with a Python script by matching the provided ground truth answers with the models’ responses. Although GPT-4o requires payment, it obtains several benefits to optimize the evaluation pipeline, including a faster process, high resource efficiency, easy result analysis in JSON format, and good quality accuracy.

We believe these clarifications address all your concerns and questions. We look forward to further discussion if needed.