Thank you for thoughtful comments and the time you spent for the detailed review. We hope to address your concerns below:

More Baselines

We acknowledge your suggestion to include more baseline models. We agree this would strengthen our analysis and have already begun running these additional experiments. Our GPUs are currently processing. Given that our framework requires six distinct tasks per model and considering our computational constraints during the ICML rebuttal period, these experiments will take some time to complete. We will update you with the results during the discussion period as they become available. Thank you for valuable recommendation.

Real world domains:

Regarding our emphasizing of the significance of varying visual representations of identical concepts across different real-world domains, we would like to be more clear: The ability to understand concepts is a fundamental capability expected from an "intelligent" agent. And this makes investigating the capability of our models in this regard significant. The simplest and dataset-makeable instance of conceptualization that we could think of was graphs. Our work deliberately focuses on graphs as a clear, well-defined example of such visual representation challenges. Upon testing the existing models on these basic graph tasks, especially isomorphism, we saw them significantly struggling. Just to be more clear, we do not in any shape claim that our dataset directly enhances model conceptualization in other scientific domains like chemistry. We believe that these, like simple graphs, are other instances of conceptualization, but more complex. We hope this work will inspire future research to introduce diverse real-world representation datasets for conceptualization.

Regarding similarities with [1]:

We thank the reviewer for bringing this work to our attention. We missed this reference as our creation of VGA took several months, and at the time we began, [1] was not yet published. We will certainly add a comprehensive discussion of it to our final paper.

Our work is fundamentally distinct from [1] in several important ways:

1. Representation approach: [1] inputs textual data of graphs node and edge lists in addition to images. In contrast, our work focuses exclusively on visual conceptualization without additional text information. More importantly, the edge list that they are giving as text information, is layout independent, and this is in striking contrast with the whole idea of our paper.

2. Purpose of layout variation: We deliberately employ different layouts in training vs testing (and no two graphs being the same) to assess models' conceptualization abilities and to prevent pattern memorization between training and testing. Conversely, [1] uses different layouts as an augmentation technique to expand training data, while also providing layout-invariant textual representations. This is strongly the opposite of what we intend to do.

3. Human interpretability: The essences of the plotted data and the way they are plotted are far apart. We carefully selected layouts and representations to ensure they remain fully understandable by human subjects, as confirmed by our human evaluation studies. This was critical since our approach relies solely on visual information. By contrast, [1] employs layouts and graphs that are often visually complex and challenging even for humans to interpret without supplementary textual information. For reference, examine this example of graph 100 in different layouts from [1]:

• Circo layout

• Dot layout

• FDP layout

• Neato layout

• SFDP layout

• ...

Reagding [2]: we have already cited it in L132. We will include [1] and [3] in our final version.

Question Regarding layout selection, we carefully chose the current layouts from standard graph representation methods that remain human-understandable. For instance, we deliberately avoided circular layouts since they become visually challenging even for humans as the number of nodes increases. We are currently working on a carefully hand-crafted test set with special graph representations to publish on the dataset website in the future. However, since these are manually created, it takes time to develop a collection that is both rich enough and of high quality.

We thank you for your thoughtful feedback and positive evaluation of our paper. We're happy that you found our benchmark sound and valuable for evaluating visual models' ability to understand concepts across different visual representations. Below we try to address your concerns.

Additional datasets to discuss

Regarding the datasets you mentioned (Geometry problems, Table QA, and Science Figure QA), we appreciate the recommendation. We have discussed and cited FigureQA in our related work section, ~L160 (Kahou et al., 2017). We will expand our discussion to include the other suggested datasets and clarify the key differences between our approach and these works. Our benchmark uniquely focuses on isolating the specific challenge of conceptualization across visual representations using graph-theoretic concepts as the test case.

Graphs described in natural language

Your question about how GPT-o1 would perform when given graphs described in natural language is insightful. While this is an interesting direction, our focus was specifically on the visual format. Graphs visually represent complex data and relationships in ways that are intuitive for humans to understand without textual descriptions. The visual aspect is critical to our research question about conceptualization across different visual layouts. There is indeed existing work on textual graph representation for language models, such as Fatemi et al.'s "Talk Like a Graph: Encoding Graphs for Large Language Models," ICLR2024, which explores different textual encoding strategies for graph reasoning tasks. However, our work deliberately targets the visual domain to evaluate a different capability - the ability to recognize the same underlying concept despite variations in visual form, which is fundamental to human visual reasoning. We hope to revise the paper to include this refrenece and clarify the focus.

We sincerely thank you for your thoughtful and detailed feedback. We appreciate the positive reception to the core idea of VGA. Below, we address each of the points raised:

3D format

You raise an interesting point about testing graphs in 3D format. Our current focus was on testing visual conceptualization across different 2D layouts (Kamada-Kawai, planar, random) as these represent common ways graphs are visualized in practice. Testing 3D representations would be a valuable extension; however, it would present significant technical challenges. Current top multimodal LLMs like ChatGPT and Claude only support 2D image inputs, with no native capability to process or reason about true 3D representations. This limitation would make direct testing of 3D graph comprehension infeasible.

More Baselines

We acknowledge your suggestion to include more baseline models. We agree this would strengthen our analysis and have already begun running these additional experiments. Our GPUs are currently processing. Given that our framework requires six distinct tasks per model and considering our computational constraints during the ICML rebuttal period, these experiments will take some time to complete. We will update you with the results during the discussion period as they become available. Thank you for valuable recommendation.

Table 2 Clarification

Thank you for pointing this out, You're absolutely right that this needs clarification. In our study, all vision models (ViT, Swin, ConvNeXt) were fine-tuned on VGA's training sets, while the multimodal LLMs were evaluated in a zero-shot setting without fine-tuning. We will make this distinction explicit in the revised table, caption, and methodology section.

Sample Size for LLM testing

We appreciate your concern about the sample size for LLM testing. Our 20-sample approach was sufficient to detect random-level performance in most cases; we expanded to 100 samples only where GPT-o1 showed promise, to obtain better precision. We adopted the "FAIL" designation from Tay et al. (2021) when performance was not better than random guessing, as precise values near the random baseline offer minimal scientific insight. It's worth noting that querying these models is not free—accessing models like GPT-4o, Claude, and Gemini involves substantial API costs. Testing across different layouts means each model required ~160 experiments, and we’ve conducted nearly 900-1000 evaluations including unreported models (Claude Opus, Gemini 1) and we have to update the existing models to Claude 3.7 and O3-mini. Expanding to 50+ samples would significantly increase costs with low returns in clearly failing conditions. However, if the reviewer believes that larger sample sizes are essential for the scientific contribution of this paper, we would absolutely prioritize and carry out additional testing accordingly.

Effect on model performance on natural images

This is an insightful question. Since we start with pretrained models and finetune them on VGA tasks—replacing the classification layer with a binary one—it inevitably alters the underlying weights. As such, it's unclear how to meaningfully assess "performance on natural images" post-finetuning. In practice, a multitask setup would likely be needed to preserve general visual capabilities. If the reviewer has thoughts on how to evaluate this more effectively, we’d be very interested to hear them.

We greatly appreciate your feedback, which will help us strengthen the paper. We believe addressing these points will improve the work and hope you'll reconsider your assessment based on our planned revisions.

We thank you for your positive assessment of our work and for helpful comments. Below, we address your questions and suggestions.

Distinction from Related Work The reviewer asked how VGA differs from the cited papers on visual reasoning. While all these works explore aspects of visual abstraction and reasoning, VGA offers several important distinctions:

• VGA uniquely tests whether models can recognize the same graph concept across different visual layouts. By training on one layout (e.g., Kamada-Kawai) and testing on another (e.g., planar), we isolate the ability to abstract away from visual form to understand invariant properties.

• Graph-theoretical foundation: Unlike more general visual reasoning datasets, VGA's tasks are grounded in well-defined graph theory problems with deterministic answers. This provides clear evaluation criteria without subjective interpretation.

• With 27,000-150,000 training samples per task, VGA is substantially large, while maintaining careful separation between train and test sets with no graph repetition.

Other comments or suggestions

We appreciate the reviewer's hypothesis about the Middle-Score and Easier-Worse anomalies. We're interested in exploring controlled experiments that could isolate memorization effects from genuine conceptualization, and agree that evaluating the memorization effect is a promising direction for future work. We used different layouts for training and testing sets to avoid memorization when trained on our dataset.

To test on MLLMs, one could design experiments with systematically varied graph structures that share superficial patterns but require different abstract reasoning. We would appreciate it very much if the reviewer has experiment suggestions in this direction for future work.