HumanEval-V: Evaluating Visual Understanding and Reasoning Abilities of Large Multimodal Models Through Coding Tasks

W1: Lack of Novelty

We respectfully disagree with the assertion that HumanEval-V lacks novelty. As discussed in our response to Reviewer ZHrF’s W2, HumanEval-V and MMCode have distinct purposes. MMCode primarily assesses programming abilities using coding problems scraped from competitive programming websites, where detailed textual descriptions are provided, and visual reasoning plays a minimal role. MMCode’s primary aim is to evaluate programming performance with additional visual inputs.

In contrast, HumanEval-V specifically focuses on evaluating visual understanding and reasoning abilities in LMMs, using coding tasks as a structured means to assess these higher-order reasoning skills rather than programming abilities alone. This focus fills an important gap in the evaluation landscape, offering insights into LMMs' visual reasoning capabilities that are not the primary concern of MMCode.

W2: Limited Diversity of the Benchmark

As mentioned in our response to Reviewer aSN5’s W1, HumanEval-V was meticulously expanded through adaptation and mutation to enhance diversity. All images in HumanEval-V are redrawn and distinct, ensuring that adapted and mutated tasks independently challenge LMMs.

Q1: Details on Dataset Creation and Adaptation Process

Our annotation process involved contextual modification (e.g., altering object relationships, introducing new constraints) and image redrawing (e.g., changing object positions, adding visual elements) to create distinct tasks. The adapted questions are fundamentally different from the originals, and all images are redrawn to ensure uniqueness. For additional details, please refer to Appendix C.2 and C.3.

Q2: Dataset Diversity and Impact of Mutation-Based Extensions

All 108 tasks feature unique images. To validate dataset diversity, we conducted analyses (discussed in the response to Reviewer aSN5 W1) showing that success on one version of a task does not imply success on its mutated version. Specifically, our pass@10 evaluations indicate that models solving an initial task have only a 40% probability of solving its mutated version within the same number of attempts. This low transfer rate demonstrates that adapted tasks are sufficiently distinct to challenge models independently.

Q1: Analysis of Model Performance on Different Image Types

The visual contexts in HumanEval-V go beyond simple types like graphs or maps, often involving compound objects and layered programmatic patterns, which add a level of abstraction. Consequently, performance analysis cannot be based solely on basic image types.

Based on our analysis of experimental results, we found that current LMMs exhibit two main limitations in HumanEval-V tasks:

1. Visual Complexity: LMMs tend to perform better on tasks with simpler visual patterns and fewer elements. For instance, as shown in Figure 4, InternVL-2 (26B) struggled with an image illustrating the "OR" operation on 7-segment displays, where visual complexity was high, with many overlapping elements.
2. Counterintuitive Contexts: LMMs often struggle with tasks that include counterintuitive or unusual configurations. In Figure 1, for example, an image showing numbers arranged in a non-clockwise order created confusion for some models, indicating a gap in counterintuitive reasoning.

Q2: Steps Taken to Prevent Data Leakage

To prevent data leakage, as noted in Line 192 of the paper, we ensured that GPT-4o could not solve any of the coding tasks without access to the images. This process confirmed that the visual context was critical for task completion, as well as ensuring that models could not rely on memorized patterns or prior knowledge alone. By focusing on image-driven tasks without detailed problem descriptions, we also prevented succeeding based purely on memorization or common coding patterns in existing datasets.

Q3: Structure of Human-Annotated Image Descriptions

Please refer to Appendix B.3 for a detailed illustration of the image description annotation process. In this section, we outline the structure of the annotations and provide examples to clarify how descriptions were created to convey relevant visual information without revealing solutions directly.

Thank you for your comments. We respond to each point below.

W1: Limited Dataset Size

As discussed in our response to Reviewer aSN5’s W1, HumanEval-V was designed to provide a lightweight yet high-quality benchmark, focusing on the specific task of evaluating visual reasoning in LMMs. While expanding dataset size is always beneficial, we prioritized rigorous curation and quality assurance to ensure each coding task met high standards for relevance, clarity, and difficulty. This trade-off allows HumanEval-V to efficiently and effectively expose visual reasoning limitations in current LMMs, even with a relatively small dataset.

To create diversity, HumanEval-V’s 108 coding tasks are drawn from 40 unique cases that have been carefully adapted and mutated. Each task is redrawn to offer distinct perspectives on visual context comprehension. Our analyses show that model success on a single task does not guarantee success on its mutated variants, underscoring the effectiveness of this approach for robust evaluation.

W2: Novelty Concerns

We respectfully disagree with the view that HumanEval-V lacks sufficient novelty. While both HumanEval-V and MMCode incorporate visual elements with coding tasks, they have fundamentally different focuses. MMCode is primarily centered on programming abilities, often relying on existing problems scraped from coding competition websites, where textual descriptions are explicit, and visuals play a minor or even redundant role. MMCode thus emphasizes programming performance rather than visual reasoning.

In contrast, HumanEval-V explicitly emphasizes visual understanding and reasoning as its central focus. Coding tasks serve as a structured and quantifiable way to assess these abilities. The benchmark challenges models to interpret visual contexts critically and requires models to extract essential information from images, making it significantly different from MMCode.

Similarly, benchmarks like ChartMimic and Plot2Code focus on de-rendering visual elements into code, which mainly tests perceptual abilities such as OCR, object recognition, and parsing visual layouts. HumanEval-V differs in that its visual contexts involve complex, programmatic patterns, targeting higher-order reasoning abilities beyond simple visual recognition.

W3: Applicability of the Task

We believe that HumanEval-V provides insights into essential and broadly applicable visual reasoning abilities, which are relevant across many downstream tasks. The design choice to use coding tasks serves to maintain simplicity while making it easier for the research community to use the benchmark for repeatable, rigorous evaluations.

Regarding the limited pool of cases from Stack Overflow, our focus was not on representing the diversity of real-world coding scenarios but on creating visual tasks that would require reasoning skills, not just perceptual skills. Stack Overflow posts include high-quality coding problems, but many involve complex context that would shift focus away from visual reasoning and towards more advanced programming tasks. To retain focus on visual understanding, we selected visually-driven coding tasks that could be solved with minimal textual descriptions and limited programming complexity.

This does not mean that HumanEval-V lacks broader applicability. The visual reasoning tasks reflect essential abilities that LMMs must develop for diverse applications, from technical domains to general-purpose multimodal use cases. The relative rarity of suitable posts on Stack Overflow speaks to the fact that these reasoning challenges are too fundamental and intuitive for human users to ask, rather than niche for model evaluation.

Thank you for your review. We address each of your comments in detail below.

W1: Limited Number of Coding Tasks

As we discussed in our response to Reviewer aSN5’s W1, HumanEval-V is purposefully designed to be a lightweight yet high-quality benchmark focused on assessing the visual reasoning abilities of LMMs. While expanding the dataset size could add breadth, we prioritized ensuring each task met rigorous standards for quality, consistency, and visual reasoning complexity. We dedicated extensive human effort to curating tasks that would be challenging, distinctive, and free of data leakage while retaining a manageable scope for efficient testing. Our experiments, supported by statistics, showing that these tasks introduce enough variation to provide valuable insights into the strengths and weaknesses of current LMMs.

W2: Benchmark Complexity and Effort

We respectfully disagree with the notion that HumanEval-V is too simplistic or lacking in workload. Our benchmark is focused on visual understanding and reasoning within LMMs, and we have invested substantial effort to ensure that HumanEval-V meets this focus in a rigorous, valuable manner for the research community.

Achieving this high standard involved overcoming many challenges:

1. Visual Context Relevance: Each task’s visual component is critical to solving the problem. Images are designed to contain all relevant information needed for task completion, ensuring that visual context is not extraneous but essential.
2. Self-Explanatory Tasks: Each coding task was developed to be primarily self-explanatory through its visual context, with minimal reliance on textual descriptions.
3. Entry-Level Coding Complexity: The coding tasks are aimed at an entry-level programming level, requiring only common Python libraries, to keep the focus on reasoning rather than programming difficulty.

As detailed in Appendix C, we share the specific challenges and solutions we encountered during the benchmark’s construction. This effort not only ensures HumanEval-V’s quality but also provides a useful reference for the research community on conducting rigorous multimodal evaluations that reveal critical weaknesses in LMMs.

Furthermore, we have conducted extensive evaluations and ablation studies across a diverse set of proprietary and open-weight models, analyzing correlations with other benchmarks. These analyses yielded findings that we believe will offer valuable insights and directions for future research in multimodal model development.

Thank you for your comments. Below, we address each of your concerns in detail.

W1: Benchmark Size and Real-World Coverage

HumanEval-V was designed to be lightweight and efficient, serving as an informative measure of LMMs’ visual reasoning abilities. We prioritized quality over quantity, ensuring each coding task met specific standards: tasks were curated and redrawn with no data leakage, at an entry-level coding difficulty, and with clear dependence on visual context. HumanEval-V includes a broad variety of visual elements like trees, graphs, maps, plots, and beyond. Each visual context in the benchmark was created to be self-explanatory and rich in information, embedding complex rules, conditions, and algorithmic patterns into single images. This high standard requires significant design effort, but offers great value for the evaluation of LMMs.

Moreover, HumanEval-V has demonstrated its effectiveness in benchmarking multimodal models by exposing key performance gaps between open-weight and proprietary models and yielding valuable insights into the limitations of current LMMs. The benchmark’s diversity in both visual and coding patterns has proven valuable in uncovering weaknesses in LMMs, validating its design as a focused, high-quality tool for multimodal evaluation.

We also recognize the importance of dataset size. HumanEval-V’s 108 coding tasks are derived from 40 carefully screened unique cases, with each case mutated and redrawn to create diversity and robustness. Significant effort was put into ensuring that each mutated task presented a distinct perspective on visual context comprehension. To assess the effectiveness of our mutation process, we analyzed pass@10 results across models like GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, and Gemini Pro. Results showed that if a model could solve a coding task within 10 attempts, the probability of solving its mutated version within 10 attempts was only 40%. This outcome confirms the effectiveness and diversity of the mutation process, expanding the benchmark’s ability to reveal gaps in multimodal model performance.

Regarding your suggestion to include tasks with variations in task types, such as color modification in charts, we would like to clarify that HumanEval-V is not a traditional coding benchmark. Its core objective is to evaluate visual reasoning and understanding, with coding used as a proxy to assess complex reasoning. The suggested task (changing a line color) primarily tests perception and familiarity with plotting tools, which falls outside the focus of our study.

W2: Combining Multimodal Reasoning and Code Generation

We respectfully disagree that HumanEval-V’s integration of image understanding and coding represents a niche focus. As discussed in our response to Reviewer oUCU’s comment W1, the benchmark emphasizes visual understanding and reasoning, using coding as a valuable medium to assess these abilities in a structured and quantifiable way. This combination allows HumanEval-V to address previously uncovered aspects of LMM evaluation, advancing the field in assessing the alignment between visual and reasoning competencies in a rigorous coding task setting. We believe this benchmark provides a meaningful and unique contribution to the multimodal research community.

Q1: Clarification on Qwen-VL’s Performance on Line #472

As observed in Table 4, Qwen2-VL exhibits some inconsistencies in its coding abilities compared to its LLM counterpart, Qwen2. On HumanEval+, Qwen2-VL achieves a 6.7% improvement, whereas, on MBPP+, its performance drops by 9.5%. These contrasting results highlight variability in multimodal model performance, which aligns with our findings on the challenges these models face across different tasks. We will incorporate additional context around Qwen-VL’s performance to ensure clarity and prevent any potential misinterpretation by readers.

We appreciate your detailed feedback. Below, we address each of your comments and provide clarifications to support our choices and findings.

W1: Rigid integration of code and image tasks

You mention that integrating code with image tasks seems rigid, and that it doesn't reflect the full potential of multimodal models in code generation. We would like to clarify that HumanEval-V is focused on evaluating the visual understanding and reasoning abilities of large multimodal models (LMMs), not their code generation capabilities. While many multimodal models excel in general knowledge and common-sense reasoning tasks, HumanEval-V presents coding tasks that demand higher-level reasoning and visual context interpretation, which are not typically addressed by current benchmarks. The integration of visual elements with coding tasks is intentional, as it evaluates the ability of models to reason over visual patterns that requires programmatic thinking. We believe this approach provides valuable insights that other benchmarks overlook.

W2: Limited differentiation in performance scores

While we acknowledge that many smaller-scale LMMs achieve near-zero scores, we strongly disagree with the characterization that HumanEval-V lacks performance differentiation. It is true that smaller models with limited capabilities often perform poorly, but this highlights the unique challenge that HumanEval-V offers in assessing visual reasoning abilities. For example, in Table 7, we show how Qwen2-VL (8.3B) achieves near-zero performance in HumanEval-V, whereas it performs well on other benchmarks like MMMU and MathVista. This stark contrast emphasizes the gaps in visual and reasoning capabilities that the current benchmarks cannot capture.

Additionally, we have intentionally designed HumanEval-V to be approachable, with coding tasks that can be solved by entry-level programmers (as shown in Table 1). These tasks require only short context lengths and relatively simple code (median ground truth solution: 14 lines), so the low performance of smaller models provides valuable insights for improvement.

Further, Table 5 and Table 6 demonstrate that performance improves significantly with more examples, which suggests ample room for model advancement. For instance, InternVL-2 (4.2B) reaches 14.8% pass@100, and GPT-4o reaches 68.5% pass@1000. These results underscore that HumanEval-V identifies critical weaknesses in current models, offering crucial directions for future work.

Thus, we do not consider the near-zero scores of smaller models as a weakness; rather, they reveal important areas for improvement and contribute valuable insights for the research community.

W3: Narrow evaluation scope

We understand your suggestion of expanding the evaluation to consider multimodal understanding and agentic workflows. However, the tasks in HumanEval-V are designed to be simple and lightweight, with the goal of assessing zero-shot performance on entry-level coding problems. This approach mirrors other widely-used benchmarks like HumanEval and MBPP, which evaluate LLMs' coding abilities without the added complexity of agentic behavior.

Incorporating agentic evaluation would introduce significant complexities, such as the need for sophisticated prompt engineering and multi-round inference, which would likely increase computation costs. Given the simplicity and zero-shot nature of the current tasks, we felt that focusing on visual reasoning and code generation abilities was more in line with the goals of the benchmark.

That being said, we recognize the importance of investigating fine-grained abilities, and our ablation studies (Section 4.2) provide a more granular analysis of LMMs' abilities in visual understanding and code generation. These studies aim to highlight the model’s strengths and weaknesses in specific aspects.

Q1: Image + explanation yielding poorer results

We respectfully disagree with the conclusion that "image + explanation often yields poorer results than using explanation alone." In fact, Table 3 shows that for 7 out of 12 models, performance improved when both images and textual descriptions were provided. This indicates that for many models, combining visual and textual inputs does lead to performance gains.

However, there are cases where the inclusion of images can hurt performance. This can occur when models fail to correctly interpret the visual elements, leading to hallucinations or incorrect assumptions about the image content. In these cases, the multimodal input may not provide any additional benefit, and might even hinder performance due to the model's incorrect interpretation of the visual context. Since the image descriptions are human-annotated, we can be confident that the textual information is accurate and grounded. And in such cases, relying solely on textual descriptions can be more reliable.