Evaluating Graphical Perception of Large Multimodal Models

Thanks for your valuable comments and questions.

1. Only four LMMs: In our pilot exploration, we explored llava1.5, llava1.6, Yi, etc., but we found their results were not promising compared to the selected four models. Also, to support fine-grained model analysis, we believe the current selection standard has its own merits.

2. Thanks. "Most" means 50%; we will add these details in our later revisions. You are correct that we may omit some parts of the annotated regions and still get the correct information. That’s why we only ask for most of the annotated regions.

3. Thanks; we will use softer tones in our later revision.

4. Thanks for the typo; we will fix it.

Questions:

1. As we said in the caption of Table 3 (L312-314): "For example, the size, top part, and length of a bar are all proportional to the values."

2. Yes, intersection over union >= 50%. We will add these details in our revision.

3. Great question. First, it does not contradict what we intended to show in this figure: InternVL2 does not cover the necessary regions, so it is incorrect. Second, being sensitive to non-important regions can be the reason why the model is wrong.

Thank you for reviewing our work and providing comments.

1. Diversity and limited insights. We kindly disagree with your statement. Diversity may include various dimensions, like data source, chart types, and data number. The data source is from various domains, as detailed in L104. We include 14 chart types and 3D charts, and our evaluation framework clearly can support more diverse chart types as well as various data numbers. Also, we believe more fine-grained insights can be obtained from our evaluation framework and benchmark due to the reasons provided in the introduction.

2. Thank you for the comment. We agree that works like Blink and MMVP have shown the poor perception issue, but they do not provide fine-grained insights or findings, as we explained in the introduction. Additionally, they mainly consider natural images, so it is unknown whether their findings are transferable to chart images.

3. Thank you. Due to limited space, we will leave it for future work.

Thanks for taking time to review our work and providing comments.

1. Dataset description: We would highlight that our major contribution is the evaluation framework, and the benchmark is a byproduct. For details, please refer to Figure 2: 14 chart types, 10 questions, and 1000 datasets, resulting in 140,000 inputs for each model.

2. Limited contribution: We've explained our differences in L35-40 of our introduction.

3. Lack of in-depth analysis: Thanks for the suggestion. We explained this observation in the summary of Section 4, L345-346. We would appreciate it if you have concrete suggestions.

4. The results in Table 3 are all without annotations. The number of figures without annotations is actually larger than that of figures with annotations.

Thanks again for your comments, let us know if you have any other questions.

Thanks for reviewing our work and providing valuable comments and questions.

1. GPT evaluation. In fact, GPT-4o is good at solving these tasks when the data is provided in text-only format, especially when the dataset size is small, and we manually check the accuracy of evaluation (99.5%) in Section 2.4.

2. Thanks for bringing this up. We didn't systematically test that, but during our hard-coded generation process, we barely observed any incorrect charts. We will include the details in the next version.

3. Figures 4 and 5 provide some case studies, including both responses and our explanations. We will try to include more case studies in the appendix in the future.

Questions:

1. Please refer to our response to Weakness 2.
2. Thanks for the question. Given the poor performance on our simplest charts, I believe the decent performances on even more complex charts are more likely due to data contamination or relatively simple questions.
3. That's a great question. We don't think charts generated with different tools will result in significant performance differences based on our early exploration (We qualitatively tried charts generated with TikZ, Matplotlib, and Vega).

Thanks again for these insightful suggestions and questions. We will address these concerns accordingly in our next revision.

Thanks for the comments.

1. Thanks for the suggestion. We will consider that in our future work. In terms of constructing such data, our evaluation framework can easily generate more questions and chart types, which can be used for training, as discussed in the conclusion section.

2. In the 2-dimensional charts shown in Section 4, all visual elements are used to represent values, as stated in the caption of Table 3 (L312-314): "For example, the size, top part, and length of a bar are all proportional to the values." Ideally, the models should use the most effective visual elements for specific tasks to improve accuracy; however, they are only able to perceive specific combinations of visual elements.

Questions:

1. That's a good question. We didn't quantitatively test that, but our observation is that the differences among MLLMs in different domains are not substantial. The major issue remains generalization to different chart types and combinations of visual elements.