VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information

We sincerely appreciate your insightful comments and valuable feedback!

I wonder whether there is any selection bias in the evaluation setting.

We designed our dataset to remove the influence of selection bias (Line 173). Specifically, we include an equal number of questions for each answer option and shuffle the order of the options for multiple-choice questions. For True/False questions, we include positive and negative versions of the questions (e.g., "There is a triangle ABC in this figure" and "There is no triangle ABC in this figure") for all questions to eliminate the influence of model preference toward a particular answer (True or False).

I would like to see additional results from more advanced models, such as Gemini-2.5-Pro and Qwen2.5-VL-72B.

Thanks for your suggestion. We conducted an additional evaluation and reported the results for the recent models on the Real split in the following table. Gemini 2.5 Pro (preview-05-06) exhibits large improvements in the diameter, chemistry, and chart tasks, narrowing the gap with human performance. However, its performance on most tasks involving geometric shapes remains considerably lower than human performance. We will include more results on recent models in the final version.

what factors other than the LLM's capabilities (discussed in Table 5) might significantly influence performance

Thank you for the suggestion. Our experiments show that training data and the capabilities of LLMs largely influence the geometric perception capabilities of LVLMs. However, we observe that even fine-tuned models and large models still often exhibit poor performance, indicating that simply scaling the training data or parameter size may not lead to human-level performance (Line 214). This observation implies that further improvements may require fundamental updates to the architecture of LVLMs. For example, in visual encoders, the process of splitting images into tiles may negatively affect their geometric perception.

In addition, as shown in our response to Reviewer PGab, we observe that LVLMs exhibit poor geometric perception even for very simple geometric shapes consisting of only two or three lines. Another analysis, presented in our response to Reviewer v9QR, shows that the attention maps of LVLMs do not effectively focus on the target geometric elements. These additional findings further suggest that the architecture of current LVLMs has fundamental limitations both in visual encoders and LLMs.

InternVL2-26B performs worse than InternVL2-4B, even though it has a larger vision encoder

InternVL2-26B performs worse than InternVL2-8B on average in VisOnlyQA-Eval-Real. Although the performance difference is small and not statistically significant, there is indeed no improvement from the 8B to the 26B model. As a possible explanation, the two models use largely different architectures with different ViTs and LLMs, as shown in the following table. InternVL2-26B, despite being larger, may have an architecture that is less well-suited for the geometric perception tasks in the Real split.

When the paper claims that performance is close to random guessing on certain tasks, what confidence level is being used to support that claim?

In response to your suggestion, we conducted a paired bootstrap test (n = 1000) and computed the 95% confidence intervals of the difference between model performance and random performance (model performance - random performance). This is a popular statistical method for comparing model performance against baseline performance (e.g., Table 2 in this paper).

The results for InternVL2 76B and Gemini 1.5 Pro on the Real split are shown in the table below. We observe that, for the Triangle, Quadrilateral, Chemistry (s), and Intersection tasks, the confidence intervals for both models include negative values, indicating that the improvement over random performance is not statistically significant. We will include this result in the final version.

We sincerely appreciate your insightful comments and valuable feedback!

A deeper analysis of possible bottlenecks—like transformer attention maps or patch sizes—would strengthen the paper.

In Section 4.4, we discuss that LLMs can be a bottleneck for visual perception in LVLMs. We show that LVLMs with larger LLMs exhibit better visual perception, which is a novel and counterintuitive observation.

For further analysis, we use VLM-Visualizer to visualize attention maps of LLaVA-1.5-7B on the Angle and Length tasks. We observe that attention maps on images often focus on text labels (e.g., A, B, C) and show weaker attention on geometric shapes such as lines. It is possible that the current architecture or pretraining is overly optimized for specific visual information like text and less suited for understanding geometric shapes. We will include more analysis of intermediate behaviors of LVLMs on our dataset in the final version of the paper.

We also agree that further analysis of architectural parameters, such as patch sizes, would provide insights into model bottlenecks. While such ablation studies require pretraining LVLMs with different architectures from scratch, which is not feasible with our resources, we consider it a promising direction for future research.

One possible reason for the poor performance could be that vision encoders haven’t seen enough geometric data during training. It would help to run separate fine-tuning experiments on just the vision encoder or just the language model to better understand their individual roles.

We agree that the lack of training data is a possible reason for the poor performance, and this consideration motivated our experiments to fine-tune LVLMs on our dataset (Line 275). However, our experiment does not show consistent improvements even after fine-tuning, suggesting that the lack of training data is not the only reason for the weak geometric perception of LVLMs (Line 307).

(Updated on June 4th)

Although our experiments suggest that additional training does not fully solve this issue, we agree that fine-tuning just the vision encoder or just the language model would provide further insights. We conducted an additional experiment on InternVL2-8B in the Length and Area tasks, where our original fine-tuning reasonably improved performance. Our original setting in the paper fine-tunes LLMs of LVLMs, which is a popular approach and is the default setting of the InternVL code published by the authors.

The results, shown in the following table, indicate that fine-tuning LLMs leads to greater performance gains than fine-tuning ViTs, particularly on out-of-distribution (real) figures. This finding reinforces our claim that LLMs play a critical role in geometric perception, even though the task does not require complex reasoning or external knowledge. We will include more results in the final version of the paper.

While the paper does try to balance real and synthetic data, the fine-tuning relies mostly on synthetic examples. It’s not clear if this creates a bias between training and evaluation, or how well the improvements would transfer to real-world tasks.

We evaluate the fine-tuned models on both synthetic and real figures, which correspond to in-distribution and out-of-distribution data. We designed this experimental setup to avoid potential evaluation biases. We observe that fine-tuned LVLMs often perform poorly even on in-distribution data (synthetic figures), although they also exhibit a certain degree of generalization to out-of-distribution (real) figures in some tasks (Figure 5).

Given that the proposed dataset is challenging for most LVLMs, even after fine-tuning, I suggest that the authors break down the questions into several simpler ones to better identify the bottlenecks in LVLM performance.

What if the figures were simplified and the questions focused on very basic tasks, such as comparing two angles or the lengths of two lines?

Thank you for the suggestion. We create a new dataset consisting of synthetic geometric shapes with different complexities. Here, we use the number of lines in geometric shapes as a surrogate for the complexity. We generate new synthetic images and questions for the Triangle, Length, and Angle tasks. Each subset includes geometric shapes with 2, 3, 4, 5, and 6 lines, and each subset consists of 50 instances.

The results of InternVL2 76B and Gemini 1.5 Pro are shown in the following tables. We observe that the performance of LVLMs is not largely influenced by the complexity of geometric shapes. Notably, their performance remains very low in cases where the figures contain only two or three lines. This result corroborates our findings that current LVLMs have fundamental limitations and cannot correctly perceive basic geometric properties even in very simple geometric shapes.

• InternVL2 76B

• Gemini 1.5 Pro

Do LVLMs truly understand the concept of degrees?

In section 4.2, we conducted a manual analysis of chain-of-thought reasoning generated by LVLMs. According to the reasoning articulated in their responses, Question Understanding Error is rare, indicating that LVLMs correctly understand the concept of geometric properties such as degrees asked in our dataset. In addition, we can expect the fine-tuned models to correctly understand the tasks they are asked to solve, but their performance often remains poor (Figure 5). These results suggest that current LVLMs exhibit limited geometric perception even when they correctly understand the underlying concepts and task requirements.

We appreciate your detailed and constructive feedback!

I believe the authors should demonstrate a significant effort in designing the textual questions

What criteria were used to design the questions?

Our main effort lies in designing questions that evaluate geometric perception independently from other capabilities. Evaluation of the geometric perception capability of LVLMs is a well-motivated and fundamental problem (Line 27), but existing datasets are not suitable for this purpose (Lines 31–40, Table 1). To address this challenge, our design criterion is to annotate questions that directly ask about various geometric properties in images (Line 123). The annotated questions focus on the fundamental ability of geometric perception, while being disentangled from complex reasoning or external knowledge (Line 124).

How is the difficulty level of the questions controlled? How to control the difficulty level?

We designed our dataset to include geometric perception tasks across a range of complexity levels. We consider the task complexity to depend on both geometric shapes and textual questions, and tasks in our dataset can be categorized into three complexity levels as follows:

• Simple -- Question: recognizing geometric information, Image: geometric shapes
  • Geometry-Triangle, Quadrilateral, Angle, Diameter
• Medium -- Question: comparing geometric information of multiple elements, Image: geometric shapes
  • Geometry-Length, Areas
• Complex -- Question: comparing geometric information of multiple elements, Image: complex scientific images
  • Chemistry, Charts, 3D

However, based on our experiments, we observe that the difficulty for current LVLMs does not necessarily correlate with the complexity of the tasks or images. For example, LVLMs exhibit near-random performance on the simple task of recognizing triangles, while recent models achieve better performance on more complex tasks involving chemistry figures.

What is the quality of the designed questions? Do they always make sense? I do not see any description of a quality verification process or filtering.

The quality of our questions is ensured through three aspects. First, the questions are generated by an annotator and verified by another annotator. Second, after the dataset is finalized, we evaluated its quality through human performance (Table 4). Table 4 shows that human performance is nearly perfect, achieving 93.5% accuracy. Third, we conducted error analysis in Section 4.2 by manually analyzing whether LVLMs correctly interpret the questions (Question Understanding Error), and we observed that even relatively weak models correctly understand the questions in over 90% of the cases. These results indicate that the questions in VisOnlyQA are clear for humans and of high quality.

In response to your concern, we also conducted an additional manual analysis to directly assess the quality of the questions. We manually evaluated 100 randomly selected cases in the Real split (10 cases from each task) and confirmed that all questions are correct and make sense. We will include this result in the final version of the paper.