Do Vision-Language Models Really Understand Visual Language?

We thank the reviewer for the time! We would kindly point out that there are some misunderstandings about our work, and we would like to respond point by point for elaboration.

Point 1: However, contributions of this paper are not particularly prominent. Is the core contribution the proposed test suite, the evaluation dataset, or the insights gained?

Response: Our core contribution is not only the test suite or datasets but the resulting insights which may drive the future design of datasets and models. We mentioned our contribution in the abstract, introduction, as well as contribution sections. For example in the conclusion section, we mention:

We evaluate three LVLMs on diagram understanding, and find that while these models can perfectly recognize and reason about entities depicted in the diagrams, they struggle with recognizing the depicted relations. Furthermore, we demonstrate that the models primarily rely on knowledge shortcuts when answering complex diagram reasoning questions. These results suggest that the apparent successes of LVLMs in diagram reasoning tasks create a misleading impression of their true diagram understanding capabilities.

To do that, we proposed a test suite, generating and annotating new datasets. These contributions are also helpful for the community and future explorations.

Point 2: In fact, some of these insights have already been revealed, such as “models consistently perform well on entity-related questions”, “models exhibit significant difficulty in identifying relationships”. I suggest that authors review the existing work and clearly highlight the differences between this paper and previous work, including the evaluation methods and insights gained.

Response: According to our knowledge, our work first obtains these insights for large models on knowledge-rich diagrams. As discussed with Reviewer 6WSg, we include more discussions about other diagrams without rich background knowledge such as charts and graphs in the related work (see red lines 912-920). We would appreciate it if the reviewer could provide more specific references to help us further improve the related work section.

Point 3: Additionally, beyond the overall evaluation results, I would like to see specific results across different domain and topics (as shown in Table 2). This could lead to new discoveries and spark more valuable discussions (involving different knowledge systems).

Response: Thanks for the suggestion. Fortunately, we have all the performances. The model response as well as test accuracies are in the supplementary file. The reason why we do not include them in the paper is that diagrams in different domains vary quite differently, and the number of diagrams in each domain is a bit small. Thus, the test accuracy in each domain could be biased, and while presenting them, we might need to be more careful. Thus, we only report the overall accuracies to make sure the results are reliable. We will publish all the data and encourage future work for further exploration.

Point 4: The Chain of Thought approach mentioned by the authors is relatively simple, as it only involves adding prompts like “think Step-by-Step.” Are there any more effective ways to use Step-by-Step prompts for this kind of chart? I suggest that the authors explore this in greater depth.

Response: Regarding the CoT setting, we follow the popular setting proposed by Wei et al., 2022 and Kojima et al., 2023. We agree that proposing a better CoT prompt is interesting but this is not our focus.

Point 5: However, for some simple relational charts, how can we be certain that these large models have not encountered similar charts during training? This point remains uncertain.

Response: We also generate synthetic data on our own for our evaluations. Thus, we can reasonably assume that these diagrams are not seen by large models.

Point 6: By the way, in this work, beyond the evaluations conducted by the authors, I would like to see a model proposed by the authors specifically for understanding basic relational charts.

Response: Our work mainly focuses on the evaluation and interpretation side. Besides, we focus on the general diagram instead of specific charts (see section 2.1 for the diagram definition). We agree that exploring the model improvement is interesting but this is not our focus. We would like to encourage future work to do that.

In summary, we appreciate the reviewer's effort in reviewing our work. But we would like to kindly ask the reviewer to re-evaluate our work based on our response.

We highly appreciate the reviewer for recognizing our test suite and admiring our findings for the community. Let us try to answer the proposed questions point by point below.

Point 1: The evaluation of the middle figure needs to be revised, as the correct answers are not included in the answer options…

Response: We select the real example from the dataset as the case study. In fact, we also tried the case with the correct option. The answer is still the same. We may include these results in the main paper or Appendix.

Point 2: …To exclude the influence of spatial positionings on the observed hallucinations, I propose to rerun the case study with swapped spatial positions. Is the "fish" consistently predicted if swapped with the other entities?

Response: Indeed, in Appendix E.2.1 (Figure 5), we explore if the position would affect the model’s performance and the answer is yes. We suppose that there is position bias in real diagrams as well. But for your information, we tried several times manually with entities shuffled and the answer is still ``fish’’.

Question 1: The main text does not state how the real-world diagram set was curated. Specifically, it would be interesting which filtering criteria were applied. Did you filter diagrams only by their domain? Did you apply a limit for the maximum number of nodes/relations/edge crossing?

Response: We determine the domains first and manually select all diagrams in that domain for annotation. We filtered out some low-quality (e.g., blurred or over-simple) diagrams. During the question annotation, we also removed several diagrams that are extremely hard to annotate. We mention this in the revised version (see red lines 232-235).

Thanks for the positive comments! We are grateful that our experiment design is admired by the reviewer. It’s our great pleasure to provide useful and reliable insights for the community. Let us address the proposed concerns point by point below.

Point 1: …I think it is important to test if explicitly including instructions to ignore prior knowledge and solely answer using the information in the diagram improves the performance…

Response: We appreciate this constructive and delicate suggestion! Indeed, it is intuitive that the model performance drops when there is a contradiction between input content and prior knowledge. But in all our experiments, we ask the model to focus on the provided diagrams (see lines 251-252 and red lines 517-518) to answer questions, which can alleviate the contradiction. We improve our presentation to make this clear in the revised version (line 518).

Furthermore, we would like to kindly argue that in our scenario, it is not reasonable to look at the diagram for the entities and give the answer based on prior knowledge (hypothesized model’s manner). Specifically, it’s worth noting that we design questions that can only be answered by relying on diagram images, where the only text input is the question (with answer options). Regarding the example in section 5.2, we ask "How many food chains are there?". If the model ignores the diagram, the answer can’t be given even with prior knowledge. If the model gives the answers based on the diagram, the answer is naturally from the diagram instead of the prior knowledge. The question is not about picking from two contradictory options. It is about answering questions by analyzing the input, i.e., the diagram. Thus, we regard the hypothesized model’s manner as unsuitable.

Point 2: This paper tested 3 LVLMs. Perhaps testing a few more would be helpful, e.g., Claude 3.5 Sonnet.

Response: In our experiments, we find that all three models have similar performance manners. Thus, exploring additional models would cost much more money but probably provide similar results. Besides, when we started our experiments, Claude, as well as other LVLMs (e.g., Molmo, LLaMA3.2-Vision), were not available (in many countries). Therefore, we encourage future work on broader explorations of models.

Point 3: Even though I appreciate the inclusion of a real-world dataset, it is exclusively focused on science. A broader scope would be better.

Response: Thanks for the suggestion. As mentioned by Reviewer 6WSg, we add discussions of more types of diagrams e.g., charts in related work (see red lines 912-920). Since we focus on real diagrams that contain rich background knowledge, limited by existing datasets (e.g., AI2D), we mainly focus on the scope of science. We would like to leave more comprehensive explorations on the diagram scope for future work.

Question 1: …I am uneasy with the use of "primarily". The difference between KR and KF is at most ~15%. Saying "primarily" seems wrong to me…

Response: Thanks for the suggestion! We improve the word usage (see red lines 537-539) and will keep polishing the paper further.

Thank you for the response. It seems that there is a misunderstanding of my comments. I did not advocate for "look[ing] at the diagram for the entities and giv[ing] the answer based on prior knowledge". What I am saying is quite the contrary--that the model should be even more explicitly instructed to rely solely on the diagram to answer the question, without considering its prior knowledge. Without this explicit instruction, when the question resembles a trick question, it is imo not a fair assessment. The authors seem to have only made minor changes, so I will keep my score. That said, I still believe it is marginally above the acceptance threshold.

We thank the reviewer for recognizing our extensive and holistic experiment design. We explore whether LVLMs understand visual language by answering two questions in the Introduction with a series of experiments. We include carefully designed synthetic diagrams and comprehensive real diagrams from multiple domains to provide reliable insights. We try to address the proposed concerns point by point below.

Point 1: The paper write-up could perhaps be revisited. It is difficult to read Table 6 the first time — the relative improvement could perhaps be presented more intuitively.

Response: We improved the presentation of all tables in our paper by adding color to differentiate the zero-shot performance and CoT performance. Besides, we changed the table headers and descriptions of Tables 6, 7, and 8 to make them easier to follow.

Point 2: The motivation behind using ‘knowledge as a shortcut' in sections 4 and 5 was not clearly stated…

Response: Our paper is organized by answering the two questions in the Introduction. Specifically, we use one intuition, five observations, and one finding to connect all experiments. The motivation behind knowledge shortcuts is stated after observation 3 as well as at the beginning of section 4. We emphasize these transition contents to make them easier to follow (red lines 362-363). The motivation is also pasted below for your information.

We substantiate that LVLMs cannot understand relations in synthetic diagrams, yet this finding reveals a contradiction. While the models do not inherently possess the ability to recognize relations, they are able to do so in real diagrams. This outcome contradicts our initial intuition (Intuition 1). Therefore, we further investigate these counterintuitive findings by examining the key difference between synthetic and real diagrams: the role of background knowledge.

Point 3: Since ‘knowledge’ is a quite generic word, it might be useful to define more precisely what kind of knowledge they are referring to early in the paper.

Response: We will further polish our word usage. “knowledge’’ here refers to the background knowledge (e.g., commonsense knowledge) that can be used to understand the diagram.

Point 4: There are some relevant works from Chart, Graph, and 3D scene understanding that will be worth mentioning in the related works section...

Response: Thanks for the references. We add the discussion of these works in our related work (see red lines 910-926).

Question 1: Could it be possible to make the synthetic dataset public so that the reviewers have a better sense of its content?

Response: We attached all materials in the supplementary, including the code, data, and model responses. The synthetic dataset used in our experiment is also contained. Besides, we provide examples (including diagrams and responses) in the Appendix (Figure 10-33) for each experiment.

Question 2: The authors classified the diagram complexity based on the number of entities. Was there a reason they did not consider the number of relationship to measure complexity?

Response: Thanks for this good question! For real diagrams, the number of entities often increases with the number of relations. Thus, for simplicity, we use the entity number to roughly measure the diagram complexity. We could also report the results by the number of relations if the reviewer prefers.

Thanks to the authors for their response. I appreciate the time they invested in it. I want to stay with my original rating since I still feel it may not be an adequate contribution to be considered as a full paper.